Five consumer-feedback items, one milestone:
(1) Shared FTSC kludge plumbing in src/ma.kludge.pas
ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix,
BuildKludgeSuffix. Single source of truth for kludge naming,
INTL/FMPT/TOPT recognition, and the kludge.<lowername>
forward-compat passthrough. Eliminates the four near-identical
parsers MSG/PKT/Squish were carrying; JAM's FTSKLUDGE subfield
walking also routes through ParseKludgeLine so its unknown
kludges land in the same `kludge.<name>` slot as the others.
Bug fix folded in: the parser previously split kludge name from
value at the first ':' it found, which broke INTL (the value
contains an FTN address with ':' in it). Now picks the earlier
of space and colon, which handles both colon-form ("MSGID: foo")
and space-form ("INTL <to> <from>") kludges correctly.
(2) INTL / FMPT / TOPT slots in attributes registry
FSC-4008 cross-zone routing kludges every netmail tosser carries.
Added to JAM/Squish/MSG/PKT capability lists, parsed natively,
emitted on Write. Round-trip covered by tests.
(3) Unified `kludge.*` namespace for unknown FTSC kludges
Squish's `squish.kludge.<name>`, MSG's `msg.kludge.<name>`, and
PKT's `pkt.kludge.<name>` all collapse to plain `kludge.<name>`.
Consumers find passthrough kludges without switching on format.
JAM's numeric `jam.subfield.<id>` stays — those are JAM-specific
binary subfields, not FTSC-form kludges.
(4) `area` auto-populated from base.AreaTag on Read
When the caller passes AAreaTag to MessageBaseOpen (or sets
the AreaTag property post-construction), every successful
ReadMessage fills msg.Attributes['area'] unless the adapter
already populated it from on-disk data (e.g. PKT AREA kludge).
Saves echomail consumers from copying AreaTag into every
message attribute manually.
(5) TMsgAttributes multi-line helpers
GetList / SetList / AppendListItem on TMsgAttributes for the
multi-instance attributes (seen-by, path, via, trace) that
store with #13 between entries. Consumers don't have to roll
their own split/join.
Plus two PKT polish items from the same feedback round:
(6) ma.fmt.pkt.uni.DoWriteMessage now raises EMessageBase
explicitly with a pointer to the Native API instead of
silently returning False.
(7) TPktFile.CreateFromStream / CreateNewToStream constructors
accept any TStream (with optional ownership), so unit tests
that round-trip via TMemoryStream don't have to tempfile-dance.
FStream is now TStream; FOwnsStream gates Free in destructor.
TStringDynArray moved from ma.api.pas to ma.types.pas so both
the capabilities API and the new attribute helpers can share it.
Docs sweep:
- docs/attributes-registry.md: intl/fmpt/topt added; unknown-kludge
convention documented; multi-line helper section added.
- docs/architecture.md: ma.kludge layer surfaced; .uni adapter
registration gotcha called out loudly with the recommended
uses clause; area auto-pop documented.
- docs/API.md: TUniMessage section rewritten for Body+Attributes
model (was still pre-0.2); HWM API documented; PKT cheat-sheet
notes Native + CreateFromStream; tests/programs list updated.
- README.md: Building section flags the .uni gotcha first
thing; ma.kludge added to features.
tests/test_consumer_round1.pas: 7 new tests covering INTL/FMPT/
TOPT round-trip on JAM/Squish/MSG, area auto-pop, GetList/SetList/
AppendListItem, PKT raise, and TPktFile in-memory stream
round-trip.
Suite: 47/47 across 10 programs (test_consumer_round1 adds 7).
EzyCom's per-user state lives in the BBS user records, not in
the message base. There is no msg-base-side LASTREAD file to
plumb (cf. Hudson's LASTREAD.BBS or JAM's .JLR). Per the design
principle adopted earlier ("if a format wasn't designed with
HWM, it doesn't get one — that's on the BBS"), EzyCom stays at
-1 instead of getting a faked sidecar.
Final HWM coverage map:
JAM ✓ .JLR (CRC32-keyed)
Squish ✓ .SQL (CRC32-keyed)
Hudson ✓ LASTREAD.BBS (per user-id, per board)
GoldBase ✓ LASTREAD.DAT (per user-id, per board)
EzyCom -1 BBS-side state, no msg-base file
Wildcat -1 SDK has no per-user HWM primitive
PCBoard -1 USERS file lastread, deferred
MSG -1 spec has no HWM concept
PKT -1 spec has no HWM concept
4 native of 9 formats — covers JAM (most common), Squish (second
most), and the QuickBBS family. NetReader and similar consumers
fall back to their own state for the remaining 5 (e.g. dupedb
keyed by area).
README.md feature bullet updated to reflect the four native
formats.
Suite still 40/40 across 9 programs; no code change in this
commit.
Adds per-(user, board) HWM for the QuickBBS-family multi-board
formats. The same physical Hudson/GoldBase base file set holds
ALL boards (1..200 for Hudson, 1..500 for GoldBase) in one
LASTREAD.BBS / LASTREAD.DAT file, indexed by user number with
one word slot per board. Caller has to provide both pieces of
context before HWM operations make sense:
- base.MapUser('NetReader', 60001) - pick a numeric user ID
- base.Board := 5 - which board this scan is for
src/ma.api.pas:
- New TMessageBase.Board property (longint, default 0).
- Single-area formats (JAM, Squish) ignore it.
- Multi-board formats return -1 from GetHWM when Board <= 0.
src/formats/ma.fmt.hudson.pas:
- New HudsonLastRead record matching QuickBBS LASTREAD.BBS layout.
- TJamBase pattern: FLrStream lazy + EnsureLrStream +
GetLastRead(user, board) + SetLastRead(user, board, msgnum).
- SetLastRead extends file with zeros to reach the user slot,
matching QuickBBS convention.
- Uses fpOpen on Unix (same FPC auto-flock workaround as Squish).
src/formats/ma.fmt.goldbase.pas:
- Same shape, GoldBaseLastRead with GOLDBASE_MAX_BOARDS (500)
slots, file is LASTREAD.DAT.
Both .uni adapters wire DoSupportsHWM/DoGetHWMById/DoSetHWMById
to the new native methods, gating on Board > 0.
tests/test_hwm.pas: 3 new tests covering Hudson + GoldBase:
- TestHudsonRequiresMapUserAndBoard verifies -1 returns when
MapUser missing, Board missing, or both.
- TestHudsonSetGetPersistence covers two users on two boards
with cross-session persistence.
- TestGoldBaseSetGet covers a high board number (250) to
exercise the wider GOLDBASE_MAX_BOARDS range.
Updated docs/architecture.md HWM coverage map: Hudson and
GoldBase moved from deferred to native. EzyCom still deferred
(per-area layout differs); Wildcat/PCBoard still -1.
Suite: 40/40 across 9 programs (test_hwm now 11/11).
NetReader and similar consumers can now register tossers as
high-numbered users (60000+) and walk per-board HWM the way
Allfix has historically done. Tossers coexist with human BBS
users in the same LASTREAD file (different user slots).
Documents the HWM API in architecture.md and surfaces it in
README.md's feature list. Includes the auto-bump pattern, the
multi-tenant convention (each tosser registers as a named user
in the same lastread file), and the per-format coverage map.
Coverage decisions for 0.3.x:
JAM -- native (.JLR) [shipped 0.3.0]
Squish -- native (.SQL) [shipped 0.3.1]
MSG / PKT -- spec has no HWM, returns -1 [structural]
PCBoard -- USERS file too entangled, deferred
Wildcat -- WC SDK exposes only per-message MarkMsgRead,
no per-user HWM primitive; defer until either
the SDK gains the call or we reverse-engineer
the user-conference state file
Hudson -- LASTREAD.BBS is per-(user, board) and the
GoldBase base instance doesn't carry board context;
EzyCom needs API design before impl
For deferred formats, GetHWM honestly returns -1 and the caller
falls back to its own state (e.g. NR's dupedb keyed by area).
This matches the "no fakery" principle: don't pretend a format
supports HWM when it doesn't, and don't silently sidecar in a
location consumers can't discover.
The 0.3.0 / 0.3.1 trio gives NetReader native HWM coverage for
the two formats that account for the overwhelming majority of
real-world FidoNet areas (JAM, Squish). Everything else falls
back to dupedb.
No code changes in this commit -- docs only.
New docs/attributes-registry.md publishes the canonical attribute
key catalog in four tiers:
1. Universal headers — msg.num, from, to, subject, date.*, addr.*,
area, board, cost. Every Fido format carries them.
2. Canonical attribute bits — attr.private, attr.crash, etc.,
mapped to/from the FTS-1 attribute word.
3. FTSC kludges — msgid, replyid, pid, tid, flags, chrs, tzutc,
seen-by, path, via. Multi-line keys use #13 between lines.
4. Format-specific — jam.*, squish.*, hudson.*, goldbase.*, ezy.*,
pcb.*, wildcat.*, pkt.*, msg.*. Each backend's namespace.
Plus a per-format support matrix showing which keys each backend
carries. Authoritative source remains each backend's
ClassSupportedAttributes -- the matrix can drift; SupportsAttribute()
is the runtime-correct query.
docs/architecture.md TUniMessage section rewritten:
- Documents the strict two-area model (Body + Attributes only).
- Body holds only the message text, never kludges or headers.
- Library never composes presentation -- consumers walk Attributes
and assemble their own display.
- Adds the capabilities API section pointing at the registry.
- Removes the stale "kludge lines intact and CR-separated" promise
the previous adapter implementations didn't honor.
docs/PROPOSAL.md flags the original Extras-bag section as
SUPERSEDED 2026-04-17, points to the registry + architecture docs
as the live design. Original text retained as historical context
since it captures the conversation that drove the redesign.
README.md:
- Features list now leads with the lossless two-area model and the
capabilities API.
- Adds a Status note flagging 0.2 as a breaking change vs 0.1 with
a one-paragraph migration sketch (msg.WhoFrom -> Attributes.Get
('from'), etc.).
- Documentation index links to the new registry doc.
Project renamed from message_api → fpc-msgbase. Folder, README title,
docs, build.sh, fpc.cfg, and test banners all updated for consistency
with the planned remote at kjgr.io:2222/kenjreno/fpc-msgbase.git.
Also scrubbed claims that backends were "ported from Allfix" or
"match Allfix's msgutil/domsg" — none of this code was ported from
Allfix; it was implemented from FTSC documents and the original
format authors' published specs (jam.txt, squish.doc, pcboard.doc,
EzyCom reference, WildCat 4 SDK headers). Author credits live in
docs/ftsc-compliance.md.
Real interop facts that mention Allfix-the-product stay documented:
the PCB Extra2 sent-bit ($40) Allfix sets when tossing, and the
FTSC-registered product code $EB. These describe external software
behavior we interoperate with, not provenance.
docs/format-notes/hudson.md removed — stale planning doc that
predates the working ma.fmt.hudson backend.
Six-week reshape plan: lossless TMsgRecord with Extras bag, ITsmIO
abstraction, TOutboundBatch, single-callback events, explicit locking,
typed exception tree. Captures the cross-project conversation with
NetReader and the byte-agreement cross-verifier as the first
actionable step. Pre-rename baseline.
- ma.lock.Release now unlinks the .lck sentinel after releasing
the OS lock (SysUtils.DeleteFile to avoid Windows API collision)
- Unit headers trimmed to standard dev-comment form (purpose only)
- docs/API.md: complete API reference with runnable examples,
event types, locking semantics, tosser wiring, native-backend
drop-down pattern, per-format cheat-sheet
- README links to the new doc
The TWildcatBase.OpenConference was creating a TMsgDatabase without
populating the SDK globals, causing access violations on MwConfig.
Now calls the full Register sequence: InitWCglobal, LoadMakeWild,
BTInitIsam before opening; BTExitIsam + DisposeWCglobal on close.
test_wildcat reads conferences 0..6 from vendored testdata (copied
to /tmp/ma_wildcat so the source tree stays read-only).
