Ask 1 from fpc-binkp consumer thread: non-storage libraries
(fpc-ftn-transport, fpc-binkp, future fpc-comet-proto / fpc-emsi,
SQL-backed messaging like Fastway) only need TFTNAddress, not the
full 1041-line mb.types. Extract to src/mb.address.pas (~90 lines,
only SysUtils) so they can cp a single file into their project.
mb.types continues to uses mb.address so existing callers see the
type transitively -- BUT FPC does not propagate record-field access
through re-export, so consumers that touch TFTNAddress.Zone/Net/
Node/Point directly must add mb.address to their own uses clause.
All 7 in-tree .uni adapters, 2 examples, 5 test harnesses updated.
No behavioural change. Full suite passes, multi-target build
green (x86_64-linux, i386-{linux,freebsd,win32,os2,go32v2}).
Two related doc passes:
1. UpdateMessage docs. docs/API.md "Updating & deleting" section
previously said "Optional operations. Not every backend
implements them -- default returns False." Since v0.6.0 that's
wrong: UpdateMessage has header-metadata-only semantics on
every backend. Rewrote the section to describe the semantics,
typical use cases (flipping attr bits, reply-chain pointers),
and the boundary (body / CtrlInfo / SubFields / inline kludges
untouched; need Delete + WriteMessage for body changes). Added
a note by the .Native escape-hatch example that most previously-
native-only operations now have uni-API equivalents.
2. ma.* -> mb.* namespace sweep. The 0.5.0 rename renamed every
source unit but the docs kept the old names in code-block
`uses` clauses and architecture diagrams. Copy-paste consumers
hitting those would get compile errors. Replaced word-boundary
ma.<api|types|events|lock|paths|kludge|fmt> references in
README.md, docs/API.md, docs/architecture.md, docs/ftsc-
compliance.md, and docs/format-notes/dependencies.md with the
mb.* form. Left docs/PROPOSAL.md alone -- that is the pre-
rename design doc and reads as historical record.
All 75 tests pass; full six-target build clean. Retagging v0.6.0
in place since no downstream has pinned yet.
NR caught a real durability gap during migration prep: the
sequence
base.WriteMessage(msg);
source.MarkSent(srcMsg);
looks atomic, but the OS write buffer hasn't necessarily reached
the platter when MarkSent runs. Crash in that window = silent
message drop.
Add `Sync` virtual on TMessageBase (default no-op for read-only
and in-memory backends). Six writable backends override and
flush every open stream they own via SysUtils.FileFlush
(fpfsync on Unix, FlushFileBuffers on Windows):
JAM .JHR / .JDT / .JDX / .JLR
Squish .SQD / .SQI / .SQL
Hudson msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.BBS
PCBoard MSGS file + index
EzyCom header + text
GoldBase msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.DAT
MSG inherits no-op (per-write open/close — buffer flushes on
close, but dir entry isn't fsynced; future enhancement).
Wildcat inherits no-op (legacy `file` IO, not TFileStream).
Helper for backend authors: TMessageBase.FlushStream(S: TStream)
class method that handles the TFileStream cast + nil-safety.
`Sync` raises after Close (data is finalized; nothing to flush).
Test: TestSyncWriteable in test_consumer_round1 -- writes a
message via JAM and Squish, calls Sync (no raise), Close, calls
Sync again (raise expected).
docs/API.md: new "Sync (durability)" section explaining the
commit-after-fsync pattern with the canonical example.
Symmetric to fpc-ftn-transport TPktWriter.Sync (commit ee8c6ad)
that NR's review prompted on the transport side.
Suite: 48/48 (added TestSyncWriteable to test_consumer_round1).
Removes all PKT code from fpc-msgbase. The wire format and its
container concerns now live in the sibling fpc-ftn-transport
library (units tt.pkt.format, tt.pkt.reader, tt.pkt.writer,
tt.pkt.batch). Pair this commit with fpc-ftn-transport's
0.2.0 (commit 6bb71a6).
Why: the previous "reader here, writer there" split (briefly
landed in 0.3.5) baked in a coupling that didn't survive a
fresh look. The writer reached into fpc-msgbase for types,
the wire format lived in the wrong house, and consumers reading
fpc-msgbase saw "PKT support" that was actually only half-
support. Cleanest split: PKT is a wire format, both directions
belong with the wire-format-aware library; fpc-msgbase becomes
purely real message bases (Hudson / JAM / Squish / MSG /
PCBoard / EzyCom / GoldBase / Wildcat).
Also a cleaner separation-of-concerns story: a BBS that just
reads JAM/Squish never needs fpc-ftn-transport. A pure store-
and-forward node doing only ArcMail unbundle never depends on
storage formats. Each library = one concern.
Removed:
src/formats/ma.fmt.pkt.pas -> tt.pkt.format
src/formats/ma.fmt.pkt.uni.pas -> tt.pkt.reader
(TPktMessageBase -> TPktReader)
src/ma.batch.pas -> tt.pkt.batch
(TPacketBatch class name unchanged)
tests/test_batch.pas -> tests/test_pkt_writer.pas
(consolidated PKT tests)
examples/example_tosser.pas -> moves with the batch helper
Reduced in src/ma.types.pas:
- PacketRecord
- FlavourType / FlavourTypeSet / DateTimeArray
- FlagsToFido / FidoToFlags
- VersionNum (PKT-product-code stamping)
All moved to tt.pkt.format.
Kept in src/ma.types.pas:
- mbfPkt enum value (so tt.pkt.reader can register the backend
with the unified-API factory; consumers still use the
standard MessageBaseOpen(mbfPkt, ...) shape)
Migration for vendoring consumers:
before: after:
uses ma.fmt.pkt; uses tt.pkt.format;
uses ma.fmt.pkt.uni; uses tt.pkt.reader;
uses ma.batch; uses tt.pkt.batch;
(no writer surface) uses tt.pkt.writer;
TPktMessageBase TPktReader
TPktFile, TPktMessage, (unchanged class names)
TPktHeaderInfo, etc.
TPacketBatch (unchanged)
Docs sweep:
- README: PKT row called out as "moved to fpc-ftn-transport";
TPacketBatch removed from features.
- docs/architecture.md: layer diagram drops PKT + ma.batch;
new sibling-library box added for fpc-ftn-transport.
- docs/attributes-registry.md: PKT column dropped from per-
format support matrix; pointer to fpc-ftn-transport.
- docs/API.md: PKT cheat-sheet entry redirects to
fpc-ftn-transport; TPacketBatch section reduced to a
"moved" pointer with the new uses-clause shape.
- docs/ftsc-compliance.md: Type-2 / 2+ / 2.2 / AuxNet rows
annotated as living in tt.pkt.format.
Suite: 47/47 across 9 programs (was 9 with test_batch; now 9
with the PKT bits dropped from test_consumer_round1 and
test_hwm). All other tests untouched.
Decision 2026-04-18: container-level packet concerns (atomic
.tmp/rename, multi-pkt rotation, BSO .flo/.bsy, ArcMail bundles,
packet-header construction) belong in a new sibling library
fpc-ftn-transport, not in fpc-msgbase. ma.fmt.pkt(.uni) stays
here as the per-message reader/iterator surface.
This commit updates the PKT entry in docs/API.md's format
cheat-sheet so consumers reading it today know where the write-
side container API will live, instead of expecting it under
fpc-msgbase eventually. fpc-ftn-transport itself doesn't exist
yet -- this is a scope-boundary signal.
Driven by NetReader / Fimail consumer-round-1 feedback: the
storage-vs-transport split that already separates fpc-msgbase
from comet generalizes well; PKT is 70% transport with a
message-iteration angle, and the angle is the only reason it
fit here at all.
No code change.
Five consumer-feedback items, one milestone:
(1) Shared FTSC kludge plumbing in src/ma.kludge.pas
ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix,
BuildKludgeSuffix. Single source of truth for kludge naming,
INTL/FMPT/TOPT recognition, and the kludge.<lowername>
forward-compat passthrough. Eliminates the four near-identical
parsers MSG/PKT/Squish were carrying; JAM's FTSKLUDGE subfield
walking also routes through ParseKludgeLine so its unknown
kludges land in the same `kludge.<name>` slot as the others.
Bug fix folded in: the parser previously split kludge name from
value at the first ':' it found, which broke INTL (the value
contains an FTN address with ':' in it). Now picks the earlier
of space and colon, which handles both colon-form ("MSGID: foo")
and space-form ("INTL <to> <from>") kludges correctly.
(2) INTL / FMPT / TOPT slots in attributes registry
FSC-4008 cross-zone routing kludges every netmail tosser carries.
Added to JAM/Squish/MSG/PKT capability lists, parsed natively,
emitted on Write. Round-trip covered by tests.
(3) Unified `kludge.*` namespace for unknown FTSC kludges
Squish's `squish.kludge.<name>`, MSG's `msg.kludge.<name>`, and
PKT's `pkt.kludge.<name>` all collapse to plain `kludge.<name>`.
Consumers find passthrough kludges without switching on format.
JAM's numeric `jam.subfield.<id>` stays — those are JAM-specific
binary subfields, not FTSC-form kludges.
(4) `area` auto-populated from base.AreaTag on Read
When the caller passes AAreaTag to MessageBaseOpen (or sets
the AreaTag property post-construction), every successful
ReadMessage fills msg.Attributes['area'] unless the adapter
already populated it from on-disk data (e.g. PKT AREA kludge).
Saves echomail consumers from copying AreaTag into every
message attribute manually.
(5) TMsgAttributes multi-line helpers
GetList / SetList / AppendListItem on TMsgAttributes for the
multi-instance attributes (seen-by, path, via, trace) that
store with #13 between entries. Consumers don't have to roll
their own split/join.
Plus two PKT polish items from the same feedback round:
(6) ma.fmt.pkt.uni.DoWriteMessage now raises EMessageBase
explicitly with a pointer to the Native API instead of
silently returning False.
(7) TPktFile.CreateFromStream / CreateNewToStream constructors
accept any TStream (with optional ownership), so unit tests
that round-trip via TMemoryStream don't have to tempfile-dance.
FStream is now TStream; FOwnsStream gates Free in destructor.
TStringDynArray moved from ma.api.pas to ma.types.pas so both
the capabilities API and the new attribute helpers can share it.
Docs sweep:
- docs/attributes-registry.md: intl/fmpt/topt added; unknown-kludge
convention documented; multi-line helper section added.
- docs/architecture.md: ma.kludge layer surfaced; .uni adapter
registration gotcha called out loudly with the recommended
uses clause; area auto-pop documented.
- docs/API.md: TUniMessage section rewritten for Body+Attributes
model (was still pre-0.2); HWM API documented; PKT cheat-sheet
notes Native + CreateFromStream; tests/programs list updated.
- README.md: Building section flags the .uni gotcha first
thing; ma.kludge added to features.
tests/test_consumer_round1.pas: 7 new tests covering INTL/FMPT/
TOPT round-trip on JAM/Squish/MSG, area auto-pop, GetList/SetList/
AppendListItem, PKT raise, and TPktFile in-memory stream
round-trip.
Suite: 47/47 across 10 programs (test_consumer_round1 adds 7).
EzyCom's per-user state lives in the BBS user records, not in
the message base. There is no msg-base-side LASTREAD file to
plumb (cf. Hudson's LASTREAD.BBS or JAM's .JLR). Per the design
principle adopted earlier ("if a format wasn't designed with
HWM, it doesn't get one — that's on the BBS"), EzyCom stays at
-1 instead of getting a faked sidecar.
Final HWM coverage map:
JAM ✓ .JLR (CRC32-keyed)
Squish ✓ .SQL (CRC32-keyed)
Hudson ✓ LASTREAD.BBS (per user-id, per board)
GoldBase ✓ LASTREAD.DAT (per user-id, per board)
EzyCom -1 BBS-side state, no msg-base file
Wildcat -1 SDK has no per-user HWM primitive
PCBoard -1 USERS file lastread, deferred
MSG -1 spec has no HWM concept
PKT -1 spec has no HWM concept
4 native of 9 formats — covers JAM (most common), Squish (second
most), and the QuickBBS family. NetReader and similar consumers
fall back to their own state for the remaining 5 (e.g. dupedb
keyed by area).
README.md feature bullet updated to reflect the four native
formats.
Suite still 40/40 across 9 programs; no code change in this
commit.
Adds per-(user, board) HWM for the QuickBBS-family multi-board
formats. The same physical Hudson/GoldBase base file set holds
ALL boards (1..200 for Hudson, 1..500 for GoldBase) in one
LASTREAD.BBS / LASTREAD.DAT file, indexed by user number with
one word slot per board. Caller has to provide both pieces of
context before HWM operations make sense:
- base.MapUser('NetReader', 60001) - pick a numeric user ID
- base.Board := 5 - which board this scan is for
src/ma.api.pas:
- New TMessageBase.Board property (longint, default 0).
- Single-area formats (JAM, Squish) ignore it.
- Multi-board formats return -1 from GetHWM when Board <= 0.
src/formats/ma.fmt.hudson.pas:
- New HudsonLastRead record matching QuickBBS LASTREAD.BBS layout.
- TJamBase pattern: FLrStream lazy + EnsureLrStream +
GetLastRead(user, board) + SetLastRead(user, board, msgnum).
- SetLastRead extends file with zeros to reach the user slot,
matching QuickBBS convention.
- Uses fpOpen on Unix (same FPC auto-flock workaround as Squish).
src/formats/ma.fmt.goldbase.pas:
- Same shape, GoldBaseLastRead with GOLDBASE_MAX_BOARDS (500)
slots, file is LASTREAD.DAT.
Both .uni adapters wire DoSupportsHWM/DoGetHWMById/DoSetHWMById
to the new native methods, gating on Board > 0.
tests/test_hwm.pas: 3 new tests covering Hudson + GoldBase:
- TestHudsonRequiresMapUserAndBoard verifies -1 returns when
MapUser missing, Board missing, or both.
- TestHudsonSetGetPersistence covers two users on two boards
with cross-session persistence.
- TestGoldBaseSetGet covers a high board number (250) to
exercise the wider GOLDBASE_MAX_BOARDS range.
Updated docs/architecture.md HWM coverage map: Hudson and
GoldBase moved from deferred to native. EzyCom still deferred
(per-area layout differs); Wildcat/PCBoard still -1.
Suite: 40/40 across 9 programs (test_hwm now 11/11).
NetReader and similar consumers can now register tossers as
high-numbered users (60000+) and walk per-board HWM the way
Allfix has historically done. Tossers coexist with human BBS
users in the same LASTREAD file (different user slots).
Documents the HWM API in architecture.md and surfaces it in
README.md's feature list. Includes the auto-bump pattern, the
multi-tenant convention (each tosser registers as a named user
in the same lastread file), and the per-format coverage map.
Coverage decisions for 0.3.x:
JAM -- native (.JLR) [shipped 0.3.0]
Squish -- native (.SQL) [shipped 0.3.1]
MSG / PKT -- spec has no HWM, returns -1 [structural]
PCBoard -- USERS file too entangled, deferred
Wildcat -- WC SDK exposes only per-message MarkMsgRead,
no per-user HWM primitive; defer until either
the SDK gains the call or we reverse-engineer
the user-conference state file
Hudson -- LASTREAD.BBS is per-(user, board) and the
GoldBase base instance doesn't carry board context;
EzyCom needs API design before impl
For deferred formats, GetHWM honestly returns -1 and the caller
falls back to its own state (e.g. NR's dupedb keyed by area).
This matches the "no fakery" principle: don't pretend a format
supports HWM when it doesn't, and don't silently sidecar in a
location consumers can't discover.
The 0.3.0 / 0.3.1 trio gives NetReader native HWM coverage for
the two formats that account for the overwhelming majority of
real-world FidoNet areas (JAM, Squish). Everything else falls
back to dupedb.
No code changes in this commit -- docs only.
New docs/attributes-registry.md publishes the canonical attribute
key catalog in four tiers:
1. Universal headers — msg.num, from, to, subject, date.*, addr.*,
area, board, cost. Every Fido format carries them.
2. Canonical attribute bits — attr.private, attr.crash, etc.,
mapped to/from the FTS-1 attribute word.
3. FTSC kludges — msgid, replyid, pid, tid, flags, chrs, tzutc,
seen-by, path, via. Multi-line keys use #13 between lines.
4. Format-specific — jam.*, squish.*, hudson.*, goldbase.*, ezy.*,
pcb.*, wildcat.*, pkt.*, msg.*. Each backend's namespace.
Plus a per-format support matrix showing which keys each backend
carries. Authoritative source remains each backend's
ClassSupportedAttributes -- the matrix can drift; SupportsAttribute()
is the runtime-correct query.
docs/architecture.md TUniMessage section rewritten:
- Documents the strict two-area model (Body + Attributes only).
- Body holds only the message text, never kludges or headers.
- Library never composes presentation -- consumers walk Attributes
and assemble their own display.
- Adds the capabilities API section pointing at the registry.
- Removes the stale "kludge lines intact and CR-separated" promise
the previous adapter implementations didn't honor.
docs/PROPOSAL.md flags the original Extras-bag section as
SUPERSEDED 2026-04-17, points to the registry + architecture docs
as the live design. Original text retained as historical context
since it captures the conversation that drove the redesign.
README.md:
- Features list now leads with the lossless two-area model and the
capabilities API.
- Adds a Status note flagging 0.2 as a breaking change vs 0.1 with
a one-paragraph migration sketch (msg.WhoFrom -> Attributes.Get
('from'), etc.).
- Documentation index links to the new registry doc.
Project renamed from message_api → fpc-msgbase. Folder, README title,
docs, build.sh, fpc.cfg, and test banners all updated for consistency
with the planned remote at kjgr.io:2222/kenjreno/fpc-msgbase.git.
Also scrubbed claims that backends were "ported from Allfix" or
"match Allfix's msgutil/domsg" — none of this code was ported from
Allfix; it was implemented from FTSC documents and the original
format authors' published specs (jam.txt, squish.doc, pcboard.doc,
EzyCom reference, WildCat 4 SDK headers). Author credits live in
docs/ftsc-compliance.md.
Real interop facts that mention Allfix-the-product stay documented:
the PCB Extra2 sent-bit ($40) Allfix sets when tossing, and the
FTSC-registered product code $EB. These describe external software
behavior we interoperate with, not provenance.
docs/format-notes/hudson.md removed — stale planning doc that
predates the working ma.fmt.hudson backend.
Six-week reshape plan: lossless TMsgRecord with Extras bag, ITsmIO
abstraction, TOutboundBatch, single-callback events, explicit locking,
typed exception tree. Captures the cross-project conversation with
NetReader and the byte-agreement cross-verifier as the first
actionable step. Pre-rename baseline.
- ma.lock.Release now unlinks the .lck sentinel after releasing
the OS lock (SysUtils.DeleteFile to avoid Windows API collision)
- Unit headers trimmed to standard dev-comment form (purpose only)
- docs/API.md: complete API reference with runnable examples,
event types, locking semantics, tosser wiring, native-backend
drop-down pattern, per-format cheat-sheet
- README links to the new doc
The TWildcatBase.OpenConference was creating a TMsgDatabase without
populating the SDK globals, causing access violations on MwConfig.
Now calls the full Register sequence: InitWCglobal, LoadMakeWild,
BTInitIsam before opening; BTExitIsam + DisposeWCglobal on close.
test_wildcat reads conferences 0..6 from vendored testdata (copied
to /tmp/ma_wildcat so the source tree stays read-only).
