Rolls up two consumer-surfaced interop fixes that landed earlier
today (cd2cc90 FLAGS space-separator, ee0f1ca AREA: envelope
recognition) into a tagged release so NR can stabilise its pin.
Both caught by NetReader's hpt-vs-nr byte-diff harness. See
CHANGELOG.md for the full writeup.
No API change. Pure behaviour fix.
AREA: is the echomail transport envelope -- FTS-0004 puts it at the
head of the packet-message body, before any ^A kludges. It has the
same no-^A-prefix shape as the FTS-4 trailing control lines (SEEN-BY:
/ PATH:) the parser already handled.
Without this, SplitKludgeBlob would leave AREA:<tag> sitting in the
"body" output and msgbase writers would store the echomail
envelope-tag as the first 12+ characters of the on-disk body. HPT
and every other tosser strip AREA: before write since the area tag
is implicit in the destination container (JAM directory, Squish
basename, SDM directory).
Fix: add AREA: to isTrailControl, add 'area' to the dispatch case.
The attribute bag now carries area under the 'area' key (same slot
0.5.0 already populates from `MessageBaseOpen`'s area-tag argument
on reads), consumers that need the tag read it there.
Broke test_uue_hpt_compare when NR's `post -u` path built a body
blob starting with "AREA:<tag>" and handed it to WriteMessage via
SplitKludgeBlob; the envelope leaked through into the JAM body,
inflating decoded UUE by the envelope length per section. With
the parse fix every consumer now gets a clean body + a populated
'area' attribute.
FTS-4001 §4 specifies `^AFLAGS flag_string`, space-separated, no
colon. fmail's jamfun.c:321 ("FLAGS ") and ftr.c:113
("\1FLAGS DIR\r") both emit the same form; HPT's ctrl-buffer
parser accepts it verbatim without a colon.
BuildKludgePrefix was previously emitting `^AFLAGS: DIR` with a
colon, which caused consumer parsers (NR's ParseKludges, HPT's
getFlags, fmail's getKludge) to miss the FLAGS kludge entirely.
In NR that manifested as FLAGS DIR / IMM routing being silently
ignored -- the message would fall into normal routing instead of
direct/immediate delivery, and the outbound pkt's destination
node ended up unset (from the route resolver rather than the
message's actual destination).
One-line fix: drop the colon, keep the rest of the kludge
emission unchanged. MSGID: / REPLY: / PID: / CHRS: / TZUTC:
keep their colons per their respective specs (FTS-0009, FRL-1028,
FSC-0054, FSC-0093).
Process gap fix. Up to today the library had no version constant
and no git tags -- downstream consumers were pinning commit
hashes, which is fragile (any force-push or rebase invalidates
the pin silently, and the hash doesn't convey "breaking" vs
"patch"). Two pieces land together:
src/mb.version.pas MB_VERSION = '0.5.1' + split major/minor/patch
constants. Included from build.sh so every
target gets it.
CHANGELOG.md Index of prior milestones (0.3.0 through 0.5.0)
with the commit each one maps to, and the
release notes for 0.5.1 (today's security-
hardening round).
Retroactive tags follow in the next invocation. Going forward:
every release bumps MB_VERSION and places a matching v<ver> tag
in the same commit.
README "Status" section now points consumers at CHANGELOG.md and
the v0.5.1 tag.
NR asked (Message 11 in the joint inbox) for a way to round-trip
TSquishMessage.Replies[1..MAX_REPLY] through the attribute bag so
nr.linker can retire its nr.msgbase.squish.pas and write reply-
chain metadata via mb.api like it already does for JAM and SDM.
Mirror JAM's naming for uniformity across formats:
squish.replyto -- parent (scalar; existed already)
squish.reply1st -- first child (scalar, = Replies[1])
squish.replynext -- remaining chain (list, = Replies[2..MAX_REPLY])
JAM's `replynext` is a single longint because JAM walks a linked
list sibling-to-sibling. Squish stores all direct children on the
parent, so `replynext` here is a LIST attribute (via TMsgAttributes
GetList/SetList). Same key names, shape reflects the on-disk
truth -- consumers that only care about the primary reply hit the
scalar on both formats; consumers that need the full chain
(nr.linker) call GetList on Squish and walk sibling records on JAM.
SquishFromUni now rebuilds Replies[] from these keys instead of
unconditionally zeroing the array, closing the write-side drop
that blocked NR's migration.
ClassSupportedAttributes advertises the new keys alongside the
existing `squish.umsgid`.
Test: test_consumer_round1.TestSquishReplyChain -- writes a
message with reply1st=101 and replynext=[102,103,104], closes,
reopens, reads, and asserts the full chain survives.
Audit pass over the vendored Wildcat! 4 SDK (~11 K LoC, circa 2003).
Most findings are either defended at the wrapper layer already
(mb.fmt.wildcat clamps MsgBytes) or live in DOS-era 16-bit segment
math that is unreachable under 32/64-bit FPC. One credible DoS did
warrant a direct patch:
vrec.pas (IsamDeleteVariableRec): the repeat-loop walks a
NextRefNr chain with no cycle guard. A corrupted .MDF that points
NextRefNr back at an earlier record caused the deleter to loop
forever, deleting the same records repeatedly. Added a 10 M hop
cap matching the pattern used in the Squish ReIndex fix.
SECURITY.md: documents the threat model for the vendored code,
what is defended (MsgBytes clamp, O_NOFOLLOW on lock file, new
cycle cap), and what is not (arbitrary-offset Seek via
UserConfData, tree-walks below the variable-record layer,
FileOpen without O_NOFOLLOW on data files). Consumers get
deployment guidance: run Wildcat under an isolated account, mount
externally-sourced bases read-only in a sandbox.
New tests/adversarial/ suite covers each driver plus mb.kludge with
crafted-input scenarios: empty files, truncated headers, garbage
payloads, oversized length fields, infinite-loop bait. The
invariant under test is graceful degradation: no crash, no hang,
no OOM. Every allocation caps, every loop terminates, every
unreadable record returns False cleanly.
Coverage:
test_fuzz_jam 7 cases (.JHR/.JDT/.JDX/.JLR corruption)
test_fuzz_squish 5 cases (clen underflow, 2 GB clen, garbage idx)
test_fuzz_hudson 3 cases (bundle-file corruption)
test_fuzz_goldbase 2 cases
test_fuzz_pcboard 2 cases
test_fuzz_msg 4 cases (50 MB no-NUL body, strange names)
test_fuzz_kludge 3 cases (100 K CRs, 1 M CRs, legit round-trip)
run_tests.sh builds and runs them after the happy-path suite.
All 26 fuzz cases pass; all 47 existing tests still pass.
Audit pass targeting attacker-controlled binary inputs (.JHR, .SQD,
*.MSG, Wildcat ISAM) and shared-directory sentinels. Caps bound
allocations driven by in-file length fields; a few forward-only
invariants bound chain walks; O_NOFOLLOW plugs a lock-file
symlink-swap window.
JAM (mb.fmt.jam): cap SubfieldLen (ReadSubFields) and TxtLen
(ReadBody) at 64 MiB. Reject negative TxtOffset before seek.
Squish (mb.fmt.squish): reject clen/msg_length outside the frame,
and clen > msg_length - SizeOf(SqMsgHdr) (prevents bodyLen
underflow into a negative SetLength argument). ReIndex now
rejects non-forward next_frame and caps total hops.
Wildcat (mb.fmt.wildcat): clamp Hdr.MsgBytes to SizeOf(TMsgText)
in ReadBody so Move() cannot read past the fixed-size buffer.
*.MSG (mb.fmt.msg): cap ReadBody growth at 16 MiB; pre-size the
AnsiString so concatenation is O(n), not O(n^2).
mb.kludge: pre-count CRs so SplitKludgeBlob is O(n) instead of
O(n^2), and cap the parsed line count at 10 K.
mb.lock: POSIX fpOpen now passes O_NOFOLLOW so a symlink that an
attacker drops in place of the sentinel between FileExists and
fpOpen does not redirect us. Advisory flock semantics unchanged.
mb.paths: new IsSafePathComponent / EnforceSafePathComponent
helpers. Reject empty, '.', '..', absolute, drive-prefixed, or
separator-bearing tails; used by callers that accept area tags or
filenames from outside data.
NR caught a real durability gap during migration prep: the
sequence
base.WriteMessage(msg);
source.MarkSent(srcMsg);
looks atomic, but the OS write buffer hasn't necessarily reached
the platter when MarkSent runs. Crash in that window = silent
message drop.
Add `Sync` virtual on TMessageBase (default no-op for read-only
and in-memory backends). Six writable backends override and
flush every open stream they own via SysUtils.FileFlush
(fpfsync on Unix, FlushFileBuffers on Windows):
JAM .JHR / .JDT / .JDX / .JLR
Squish .SQD / .SQI / .SQL
Hudson msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.BBS
PCBoard MSGS file + index
EzyCom header + text
GoldBase msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.DAT
MSG inherits no-op (per-write open/close — buffer flushes on
close, but dir entry isn't fsynced; future enhancement).
Wildcat inherits no-op (legacy `file` IO, not TFileStream).
Helper for backend authors: TMessageBase.FlushStream(S: TStream)
class method that handles the TFileStream cast + nil-safety.
`Sync` raises after Close (data is finalized; nothing to flush).
Test: TestSyncWriteable in test_consumer_round1 -- writes a
message via JAM and Squish, calls Sync (no raise), Close, calls
Sync again (raise expected).
docs/API.md: new "Sync (durability)" section explaining the
commit-after-fsync pattern with the canonical example.
Symmetric to fpc-ftn-transport TPktWriter.Sync (commit ee8c6ad)
that NR's review prompted on the transport side.
Suite: 48/48 (added TestSyncWriteable to test_consumer_round1).
Bug-hunt round (test_hwm Hudson per-(user,board) round-trip
appeared to hang at high UserId values). Root cause: SetLastRead
extended the .LRD file by writing zeros one byte at a time:
for i := 1 to gap do FLrStream.Write(zero, 1);
For UserId=60001, gap is ~24M bytes (Hudson) or ~60M bytes
(GoldBase) — millions of syscalls. Replace with
`FLrStream.Size := needSize`, which on Unix calls ftruncate
(sparse file, unwritten slots read as zero) and on Windows
calls SetEndOfFile. One syscall instead of millions.
Bonus: build.sh UNITS array still listed the pre-rename ma.*
paths after 0.5.0 — fresh clone + ./build.sh would have failed
immediately. Updated to mb.* paths and added the missing
mb.kludge.pas entry (was always missing — never built into a
.ppu cross-target).
Suite: 47/47 (test_hwm now finishes in milliseconds rather
than minutes).
Across-the-board rename so the unit prefix matches the repo
name (mb = msgbase). Brings naming into line with
fpc-ftn-transport's tt.* prefix and avoids the historical
"ma" abbreviation that meant nothing to new readers.
Files renamed via git mv:
src/ma.{api,events,kludge,lock,paths,types}.pas
-> src/mb.{...}.pas
src/formats/ma.fmt.{jam,squish,hudson,msg,pcboard,ezycom,
goldbase,wildcat,wcutil}{,.uni}.pas
-> src/formats/mb.fmt.*.pas
All `unit ma.X` declarations and `uses ma.X` clauses rewritten
to `mb.X` across src/, examples/, tests/.
Suite: 47/47 (read 7, hwm 11, lock 4, pack 4, write 5,
wildcat 5, consumer_round1 5, batch's gone w/ PKT relocation,
plus testutil).
Consumer impact: anyone with `uses ma.api;` etc. needs to
update to `uses mb.api;`. No semantic changes; a search/replace
on the consumer's source tree is the only migration step.
NR's notes (~/.MSGAPI_MSGS.md round 3) align this against
their already-pinned 8130b40; the next NR pin bump rolls in
both this rename and any further work in one step.
Pack in fpc-msgbase means "compact the message base" — wrong
semantic for fpc-ftn-transport's TPktWriter, which is doing a
transactional commit (atomic .tmp -> .pkt rename), not compaction.
Add distinct event types so writers across the ecosystem can
signal commit/rollback without overloading Pack semantics.
Additive: existing Pack* types unchanged.
Removes all PKT code from fpc-msgbase. The wire format and its
container concerns now live in the sibling fpc-ftn-transport
library (units tt.pkt.format, tt.pkt.reader, tt.pkt.writer,
tt.pkt.batch). Pair this commit with fpc-ftn-transport's
0.2.0 (commit 6bb71a6).
Why: the previous "reader here, writer there" split (briefly
landed in 0.3.5) baked in a coupling that didn't survive a
fresh look. The writer reached into fpc-msgbase for types,
the wire format lived in the wrong house, and consumers reading
fpc-msgbase saw "PKT support" that was actually only half-
support. Cleanest split: PKT is a wire format, both directions
belong with the wire-format-aware library; fpc-msgbase becomes
purely real message bases (Hudson / JAM / Squish / MSG /
PCBoard / EzyCom / GoldBase / Wildcat).
Also a cleaner separation-of-concerns story: a BBS that just
reads JAM/Squish never needs fpc-ftn-transport. A pure store-
and-forward node doing only ArcMail unbundle never depends on
storage formats. Each library = one concern.
Removed:
src/formats/ma.fmt.pkt.pas -> tt.pkt.format
src/formats/ma.fmt.pkt.uni.pas -> tt.pkt.reader
(TPktMessageBase -> TPktReader)
src/ma.batch.pas -> tt.pkt.batch
(TPacketBatch class name unchanged)
tests/test_batch.pas -> tests/test_pkt_writer.pas
(consolidated PKT tests)
examples/example_tosser.pas -> moves with the batch helper
Reduced in src/ma.types.pas:
- PacketRecord
- FlavourType / FlavourTypeSet / DateTimeArray
- FlagsToFido / FidoToFlags
- VersionNum (PKT-product-code stamping)
All moved to tt.pkt.format.
Kept in src/ma.types.pas:
- mbfPkt enum value (so tt.pkt.reader can register the backend
with the unified-API factory; consumers still use the
standard MessageBaseOpen(mbfPkt, ...) shape)
Migration for vendoring consumers:
before: after:
uses ma.fmt.pkt; uses tt.pkt.format;
uses ma.fmt.pkt.uni; uses tt.pkt.reader;
uses ma.batch; uses tt.pkt.batch;
(no writer surface) uses tt.pkt.writer;
TPktMessageBase TPktReader
TPktFile, TPktMessage, (unchanged class names)
TPktHeaderInfo, etc.
TPacketBatch (unchanged)
Docs sweep:
- README: PKT row called out as "moved to fpc-ftn-transport";
TPacketBatch removed from features.
- docs/architecture.md: layer diagram drops PKT + ma.batch;
new sibling-library box added for fpc-ftn-transport.
- docs/attributes-registry.md: PKT column dropped from per-
format support matrix; pointer to fpc-ftn-transport.
- docs/API.md: PKT cheat-sheet entry redirects to
fpc-ftn-transport; TPacketBatch section reduced to a
"moved" pointer with the new uses-clause shape.
- docs/ftsc-compliance.md: Type-2 / 2+ / 2.2 / AuxNet rows
annotated as living in tt.pkt.format.
Suite: 47/47 across 9 programs (was 9 with test_batch; now 9
with the PKT bits dropped from test_consumer_round1 and
test_hwm). All other tests untouched.
Decision 2026-04-18: container-level packet concerns (atomic
.tmp/rename, multi-pkt rotation, BSO .flo/.bsy, ArcMail bundles,
packet-header construction) belong in a new sibling library
fpc-ftn-transport, not in fpc-msgbase. ma.fmt.pkt(.uni) stays
here as the per-message reader/iterator surface.
This commit updates the PKT entry in docs/API.md's format
cheat-sheet so consumers reading it today know where the write-
side container API will live, instead of expecting it under
fpc-msgbase eventually. fpc-ftn-transport itself doesn't exist
yet -- this is a scope-boundary signal.
Driven by NetReader / Fimail consumer-round-1 feedback: the
storage-vs-transport split that already separates fpc-msgbase
from comet generalizes well; PKT is 70% transport with a
message-iteration angle, and the angle is the only reason it
fit here at all.
No code change.
Five consumer-feedback items, one milestone:
(1) Shared FTSC kludge plumbing in src/ma.kludge.pas
ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix,
BuildKludgeSuffix. Single source of truth for kludge naming,
INTL/FMPT/TOPT recognition, and the kludge.<lowername>
forward-compat passthrough. Eliminates the four near-identical
parsers MSG/PKT/Squish were carrying; JAM's FTSKLUDGE subfield
walking also routes through ParseKludgeLine so its unknown
kludges land in the same `kludge.<name>` slot as the others.
Bug fix folded in: the parser previously split kludge name from
value at the first ':' it found, which broke INTL (the value
contains an FTN address with ':' in it). Now picks the earlier
of space and colon, which handles both colon-form ("MSGID: foo")
and space-form ("INTL <to> <from>") kludges correctly.
(2) INTL / FMPT / TOPT slots in attributes registry
FSC-4008 cross-zone routing kludges every netmail tosser carries.
Added to JAM/Squish/MSG/PKT capability lists, parsed natively,
emitted on Write. Round-trip covered by tests.
(3) Unified `kludge.*` namespace for unknown FTSC kludges
Squish's `squish.kludge.<name>`, MSG's `msg.kludge.<name>`, and
PKT's `pkt.kludge.<name>` all collapse to plain `kludge.<name>`.
Consumers find passthrough kludges without switching on format.
JAM's numeric `jam.subfield.<id>` stays — those are JAM-specific
binary subfields, not FTSC-form kludges.
(4) `area` auto-populated from base.AreaTag on Read
When the caller passes AAreaTag to MessageBaseOpen (or sets
the AreaTag property post-construction), every successful
ReadMessage fills msg.Attributes['area'] unless the adapter
already populated it from on-disk data (e.g. PKT AREA kludge).
Saves echomail consumers from copying AreaTag into every
message attribute manually.
(5) TMsgAttributes multi-line helpers
GetList / SetList / AppendListItem on TMsgAttributes for the
multi-instance attributes (seen-by, path, via, trace) that
store with #13 between entries. Consumers don't have to roll
their own split/join.
Plus two PKT polish items from the same feedback round:
(6) ma.fmt.pkt.uni.DoWriteMessage now raises EMessageBase
explicitly with a pointer to the Native API instead of
silently returning False.
(7) TPktFile.CreateFromStream / CreateNewToStream constructors
accept any TStream (with optional ownership), so unit tests
that round-trip via TMemoryStream don't have to tempfile-dance.
FStream is now TStream; FOwnsStream gates Free in destructor.
TStringDynArray moved from ma.api.pas to ma.types.pas so both
the capabilities API and the new attribute helpers can share it.
Docs sweep:
- docs/attributes-registry.md: intl/fmpt/topt added; unknown-kludge
convention documented; multi-line helper section added.
- docs/architecture.md: ma.kludge layer surfaced; .uni adapter
registration gotcha called out loudly with the recommended
uses clause; area auto-pop documented.
- docs/API.md: TUniMessage section rewritten for Body+Attributes
model (was still pre-0.2); HWM API documented; PKT cheat-sheet
notes Native + CreateFromStream; tests/programs list updated.
- README.md: Building section flags the .uni gotcha first
thing; ma.kludge added to features.
tests/test_consumer_round1.pas: 7 new tests covering INTL/FMPT/
TOPT round-trip on JAM/Squish/MSG, area auto-pop, GetList/SetList/
AppendListItem, PKT raise, and TPktFile in-memory stream
round-trip.
Suite: 47/47 across 10 programs (test_consumer_round1 adds 7).
EzyCom's per-user state lives in the BBS user records, not in
the message base. There is no msg-base-side LASTREAD file to
plumb (cf. Hudson's LASTREAD.BBS or JAM's .JLR). Per the design
principle adopted earlier ("if a format wasn't designed with
HWM, it doesn't get one — that's on the BBS"), EzyCom stays at
-1 instead of getting a faked sidecar.
Final HWM coverage map:
JAM ✓ .JLR (CRC32-keyed)
Squish ✓ .SQL (CRC32-keyed)
Hudson ✓ LASTREAD.BBS (per user-id, per board)
GoldBase ✓ LASTREAD.DAT (per user-id, per board)
EzyCom -1 BBS-side state, no msg-base file
Wildcat -1 SDK has no per-user HWM primitive
PCBoard -1 USERS file lastread, deferred
MSG -1 spec has no HWM concept
PKT -1 spec has no HWM concept
4 native of 9 formats — covers JAM (most common), Squish (second
most), and the QuickBBS family. NetReader and similar consumers
fall back to their own state for the remaining 5 (e.g. dupedb
keyed by area).
README.md feature bullet updated to reflect the four native
formats.
Suite still 40/40 across 9 programs; no code change in this
commit.
Adds per-(user, board) HWM for the QuickBBS-family multi-board
formats. The same physical Hudson/GoldBase base file set holds
ALL boards (1..200 for Hudson, 1..500 for GoldBase) in one
LASTREAD.BBS / LASTREAD.DAT file, indexed by user number with
one word slot per board. Caller has to provide both pieces of
context before HWM operations make sense:
- base.MapUser('NetReader', 60001) - pick a numeric user ID
- base.Board := 5 - which board this scan is for
src/ma.api.pas:
- New TMessageBase.Board property (longint, default 0).
- Single-area formats (JAM, Squish) ignore it.
- Multi-board formats return -1 from GetHWM when Board <= 0.
src/formats/ma.fmt.hudson.pas:
- New HudsonLastRead record matching QuickBBS LASTREAD.BBS layout.
- TJamBase pattern: FLrStream lazy + EnsureLrStream +
GetLastRead(user, board) + SetLastRead(user, board, msgnum).
- SetLastRead extends file with zeros to reach the user slot,
matching QuickBBS convention.
- Uses fpOpen on Unix (same FPC auto-flock workaround as Squish).
src/formats/ma.fmt.goldbase.pas:
- Same shape, GoldBaseLastRead with GOLDBASE_MAX_BOARDS (500)
slots, file is LASTREAD.DAT.
Both .uni adapters wire DoSupportsHWM/DoGetHWMById/DoSetHWMById
to the new native methods, gating on Board > 0.
tests/test_hwm.pas: 3 new tests covering Hudson + GoldBase:
- TestHudsonRequiresMapUserAndBoard verifies -1 returns when
MapUser missing, Board missing, or both.
- TestHudsonSetGetPersistence covers two users on two boards
with cross-session persistence.
- TestGoldBaseSetGet covers a high board number (250) to
exercise the wider GOLDBASE_MAX_BOARDS range.
Updated docs/architecture.md HWM coverage map: Hudson and
GoldBase moved from deferred to native. EzyCom still deferred
(per-area layout differs); Wildcat/PCBoard still -1.
Suite: 40/40 across 9 programs (test_hwm now 11/11).
NetReader and similar consumers can now register tossers as
high-numbered users (60000+) and walk per-board HWM the way
Allfix has historically done. Tossers coexist with human BBS
users in the same LASTREAD file (different user slots).
Documents the HWM API in architecture.md and surfaces it in
README.md's feature list. Includes the auto-bump pattern, the
multi-tenant convention (each tosser registers as a named user
in the same lastread file), and the per-format coverage map.
Coverage decisions for 0.3.x:
JAM -- native (.JLR) [shipped 0.3.0]
Squish -- native (.SQL) [shipped 0.3.1]
MSG / PKT -- spec has no HWM, returns -1 [structural]
PCBoard -- USERS file too entangled, deferred
Wildcat -- WC SDK exposes only per-message MarkMsgRead,
no per-user HWM primitive; defer until either
the SDK gains the call or we reverse-engineer
the user-conference state file
Hudson -- LASTREAD.BBS is per-(user, board) and the
GoldBase base instance doesn't carry board context;
EzyCom needs API design before impl
For deferred formats, GetHWM honestly returns -1 and the caller
falls back to its own state (e.g. NR's dupedb keyed by area).
This matches the "no fakery" principle: don't pretend a format
supports HWM when it doesn't, and don't silently sidecar in a
location consumers can't discover.
The 0.3.0 / 0.3.1 trio gives NetReader native HWM coverage for
the two formats that account for the overwhelming majority of
real-world FidoNet areas (JAM, Squish). Everything else falls
back to dupedb.
No code changes in this commit -- docs only.
Adds Squish HWM via the .SQL lastread file (CRC32-keyed by
lowercased username, identical layout to JAM's .JLR per
Squish.doc).
src/formats/ma.fmt.squish.pas:
- New SqLastRead record type matching the Squish.doc spec.
- TSquishBase gains FSqlStream (lazy) + EnsureSqlStream +
GetLastRead/GetHighRead/SetLastRead methods, mirroring JAM's
.JLR pattern.
- Close releases the lastread stream alongside .SQD/.SQI.
src/formats/ma.fmt.squish.uni.pas:
- TSquishMessageBase wires DoGetHWMByName / DoSetHWMByName,
reusing TJamBase.CalcUserCRC for the (shared) CRC32 algorithm.
- DoSetHWMByName preserves HighReadMsg monotonicity.
src/ma.lock.pas:
- TMessageLock gains optional APreserveSentinel constructor flag.
- Release no longer unlinks the sentinel when the flag is set.
- Required for Squish because the .SQL file is BOTH the lock
sentinel and the lastread store; deleting it on lock release
would wipe HWM data on every Open/Close cycle.
src/ma.api.pas:
- TMessageBase.Create passes APreserveSentinel = (Format = mbfSquish).
src/formats/ma.fmt.squish.pas (low-level open):
- EnsureSqlStream uses fpOpen directly on Unix instead of FPC's
FileOpen wrapper. FileOpen defaults to a fpflock that conflicts
with ma.lock's existing advisory lock on the same file (EAGAIN
on every attempt). fpOpen bypasses the auto-flock; cross-process
safety lives in ma.lock and doesn't need duplicating here.
tests/test_hwm.pas:
- New SeedSquish helper.
- TestSquishCapability + TestSquishSetGetPersistence verify
capability flag, set/get round-trip, multi-user independence,
and persistence across Open/Close.
Hudson, GoldBase and EzyCom were originally scoped for this
milestone but their per-(user, board) lastread layout needs more
design work (the LASTREAD.BBS format is shared across all 200
boards in a single per-user record, and TMessageBase doesn't
carry a "board" context). Deferred to a follow-up milestone.
Wildcat (SDK call) coming in 0.3.2.
Suite: 38/38 across 9 programs (test_hwm now 9/9 with Squish
coverage).
Adds the per-user High-Water Mark API to TMessageBase:
function SupportsHWM: boolean;
function GetHWM(const UserName: AnsiString): longint;
procedure SetHWM(const UserName: AnsiString; MsgNum: longint);
procedure MapUser(const UserName: AnsiString; UserId: longint);
property ActiveUser: AnsiString;
GetHWM returns -1 when the format has no HWM mechanism, the user
isn't registered (number-keyed formats), or no HWM has been set
for that user yet.
Two-flavour native dispatch:
- Name-keyed backends (JAM, later Squish/Wildcat) override
DoGetHWMByName / DoSetHWMByName.
- Number-keyed backends (later Hudson/GoldBase/EzyCom) override
DoGetHWMById / DoSetHWMById; caller must register name->id via
MapUser first.
Auto-bump: when ActiveUser is non-empty, ReadMessage compares the
just-read msg.num to GetHWM(ActiveUser); if higher, calls SetHWM.
Never decrements -- reading a lower-numbered message is a no-op.
Default off (ActiveUser = '').
JAM implementation:
- Adds .JLR (lastread) handling to TJamBase: lazy open via
EnsureLrStream, GetLastRead/GetHighRead/SetLastRead methods,
proper Close cleanup.
- TJamMessageBase wires DoGetHWMByName / DoSetHWMByName to
CalcUserCRC + GetLastRead / SetLastRead. SetHWM also keeps
HighReadMsg monotonic.
Coverage map (this milestone):
- JAM: native ✓
- Squish, Hudson, GoldBase, EzyCom, Wildcat: -1 (planned 0.3.1/2)
- PCBoard, MSG, PKT: -1 (no HWM in spec)
Tests: tests/test_hwm.pas covers SupportsHWM, set/get round-trip,
persistence across Open/Close, auto-bump via ActiveUser (advances),
auto-bump never decrements, MSG/PKT correctly return -1, Hudson
returns -1 even after MapUser (until 0.3.1 lands the impl). 7/7
new tests pass; full suite 38/38 across 9 programs.
Recommended caller pattern:
base.ActiveUser := 'NetReader';
for i := 0 to base.MessageCount - 1 do begin
base.ReadMessage(i, msg);
{ process msg ... HWM auto-tracks the high-water for NetReader }
end;
New docs/attributes-registry.md publishes the canonical attribute
key catalog in four tiers:
1. Universal headers — msg.num, from, to, subject, date.*, addr.*,
area, board, cost. Every Fido format carries them.
2. Canonical attribute bits — attr.private, attr.crash, etc.,
mapped to/from the FTS-1 attribute word.
3. FTSC kludges — msgid, replyid, pid, tid, flags, chrs, tzutc,
seen-by, path, via. Multi-line keys use #13 between lines.
4. Format-specific — jam.*, squish.*, hudson.*, goldbase.*, ezy.*,
pcb.*, wildcat.*, pkt.*, msg.*. Each backend's namespace.
Plus a per-format support matrix showing which keys each backend
carries. Authoritative source remains each backend's
ClassSupportedAttributes -- the matrix can drift; SupportsAttribute()
is the runtime-correct query.
docs/architecture.md TUniMessage section rewritten:
- Documents the strict two-area model (Body + Attributes only).
- Body holds only the message text, never kludges or headers.
- Library never composes presentation -- consumers walk Attributes
and assemble their own display.
- Adds the capabilities API section pointing at the registry.
- Removes the stale "kludge lines intact and CR-separated" promise
the previous adapter implementations didn't honor.
docs/PROPOSAL.md flags the original Extras-bag section as
SUPERSEDED 2026-04-17, points to the registry + architecture docs
as the live design. Original text retained as historical context
since it captures the conversation that drove the redesign.
README.md:
- Features list now leads with the lossless two-area model and the
capabilities API.
- Adds a Status note flagging 0.2 as a breaking change vs 0.1 with
a one-paragraph migration sketch (msg.WhoFrom -> Attributes.Get
('from'), etc.).
- Documentation index links to the new registry doc.
Adds tests/test_roundtrip_attrs.pas covering:
1. Capabilities API smoke test — confirms SupportsAttribute('msgid')
returns true on JAM/Squish/MSG/PKT, false on Hudson/GoldBase/
EzyCom/Wildcat/PCBoard. Confirms backend-private keys are gated
correctly (Squish.SupportsAttribute('jam.msgidcrc') = false).
2. Per-format kludge round-trip across all 5 storage formats —
builds a synthetic message with universal headers + FTSC kludges
(msgid, replyid, pid, flags, multi-line seen-by + path), writes,
reopens, reads back, asserts every key the backend's capability
list advertises survives byte-for-byte. Backends that don't
support a given key are silently skipped via SupportsAttribute
gating so the test exercises each format's actual contract.
3. Cross-format JAM → Squish copy — seeds JAM with the kludge
message, copies to a fresh Squish base via the unified API,
reopens both, asserts:
- intersection of capabilities lists is preserved verbatim
(msgid, seen-by, path, etc. all survive JAM → Squish)
- jam.* keys not in Squish's capability list are dropped
(no silent corruption of Squish's data with foreign keys)
Result: 7/7 new tests pass. Total suite now 31/31 across 8 programs.
This is the regression suite that locks the Body+Attributes contract
and proves the showstopper fix holds across every backend.
Hooked into run_tests.sh so CI catches future drift.
Replaces TUniMessage's 13-field flat record with a strict two-area
model: Body holds only the message text; Attributes holds everything
else (from/to/subject/dates/addresses/MSGID/SEEN-BY/PATH/format-
specific fields) as namespaced key/value pairs.
Why this fix is required NOW: the previous JAM adapter dropped
MSGID, ReplyID, PID, Flags, SEEN-BY and PATH on every Read/Write
through the unified API. A NetReader parity test surfaced it (17/21
pass with 4 kludge failures). All 9 adapters had the same bug. For
tossers and scanners the impact is silent corruption: dropped MSGID
→ dupe storms, dropped PATH → mail loops, dropped SEEN-BY → broken
routing. Three downstream consumers (Fimail's codex-transport branch,
NetReader, future Allfix) had halted integration work pending this
fix. Without it, anyone vendoring fpc-msgbase 0.1 ships with a
known-corrupting adapter.
Design choice: per Ken's call, "message is just the message text;
everything else is an attribute, including from/to/subject/dates."
Same architecture as RFC 822 email (headers + body). Each backend
fills attributes it knows on Read; reads attributes it understands
on Write; ignores unknown attributes silently (RFC 822 X-header
semantics). Forward-compatible -- a new backend (e.g. a planned SQL
message store) just adds its own attribute keys; old backends ignore
them.
Composition is the consumer's job. The library never reassembles
Body + Attributes into kludge-laden display text. A BBS that wants
inline kludges walks Attributes and prepends ^aMSGID etc. to its
own display. A tosser that needs MSGID for dupe detection reads
Attributes.Get('msgid') directly -- no body parsing required.
src/ma.types.pas:
- New TMsgAttribute / TMsgAttributes records with Get/SetValue,
typed accessors (GetInt/GetBool/GetDate/GetAddr), Has/Remove,
iteration. Linear-search lookup, fine for the ~30-50 keys per
message. Switch to hash later if profiling shows need.
- Replaced TUniMessage with the minimal Body + Attributes record.
- New UniAttrBitsToAttributes / UniAttrBitsFromAttributes helpers
to bridge the canonical MSG_ATTR_* cardinal bitset to/from
individual `attr.*` boolean keys.
- {$modeswitch advancedrecords} added so records have methods.
src/ma.api.pas:
- New capabilities API: TStringDynArray return type,
ClassSupportedAttributes (virtual class fn, default empty),
SupportedAttributes (instance sugar), SupportsAttribute (per-key
query). Each backend overrides ClassSupportedAttributes with the
static list of keys it knows. Callers query before setting so a
BBS UI can hide controls the underlying backend has no slot for.
src/formats/ma.fmt.*.uni.pas (all 9):
- Rewrote each XxxToUni and XxxFromUni for the new model. Read
populates Attributes with universal/FTSC/format-specific keys per
the attribute registry (to be published in phase 5). Write reads
attributes back and writes native form.
- JAM walks SubFields[] for SEEN-BY/PATH/TZUTC/TRACE plus passthrough
of unknown subfield IDs as `jam.subfield.<id>` for round-trip
safety. Squish parses CtrlInfo (NUL-separated ^A lines) into
individual attributes, rebuilds on Write. MSG and PKT (which keep
kludges inline in body per FTS-1) parse leading ^A lines and
trailing SEEN-BY/PATH out of the body so TUniMessage.Body is
always plain user text; on Write they reassemble the on-disk form.
- Each backend ships ClassSupportedAttributes with its key list.
src/ma.batch.pas: PktToUni signature updated to (in,out var) form.
tests/* + examples/*: migrated all callers from Msg.WhoFrom (etc.)
to Msg.Attributes.Get('from'). MakeMsg helpers now use SetValue/
SetBool/SetAddr.
Verified: 24/24 tests pass across all 7 test programs (read,
roundtrip, lock, batch, wildcat, write_existing, pack). Wildcat
walks all 7 vendored conferences clean.
Out of scope (next phases):
- docs/attributes-registry.md publishing the full key list with
per-format support matrix
- cross-format round-trip + capabilities-driven copy test
- update architecture.md / PROPOSAL.md to reflect the new model
Copies the working WildCat 4 conference set (7 conferences,
MAKEWILD.DAT, CONFDESC.*, MSG/MSG0..MSG6.DAT/IX, DATA/ALLUSERS.*,
etc., 228KB total) into the repo so the Wildcat backend test runs
without an external path dependency. Mirrors how the other format
tests carry their own sample data.
test_wildcat.pas SRC constant updated to point at tests/data/wildcat
instead of ~/allfix_dev/...; the test still copies to /tmp/ma_wildcat
before running so the vendored fixture stays untouched.
Project renamed from message_api → fpc-msgbase. Folder, README title,
docs, build.sh, fpc.cfg, and test banners all updated for consistency
with the planned remote at kjgr.io:2222/kenjreno/fpc-msgbase.git.
Also scrubbed claims that backends were "ported from Allfix" or
"match Allfix's msgutil/domsg" — none of this code was ported from
Allfix; it was implemented from FTSC documents and the original
format authors' published specs (jam.txt, squish.doc, pcboard.doc,
EzyCom reference, WildCat 4 SDK headers). Author credits live in
docs/ftsc-compliance.md.
Real interop facts that mention Allfix-the-product stay documented:
the PCB Extra2 sent-bit ($40) Allfix sets when tossing, and the
FTSC-registered product code $EB. These describe external software
behavior we interoperate with, not provenance.
docs/format-notes/hudson.md removed — stale planning doc that
predates the working ma.fmt.hudson backend.
Six-week reshape plan: lossless TMsgRecord with Extras bag, ITsmIO
abstraction, TOutboundBatch, single-callback events, explicit locking,
typed exception tree. Captures the cross-project conversation with
NetReader and the byte-agreement cross-verifier as the first
actionable step. Pre-rename baseline.
- ma.lock.Release now unlinks the .lck sentinel after releasing
the OS lock (SysUtils.DeleteFile to avoid Windows API collision)
- Unit headers trimmed to standard dev-comment form (purpose only)
- docs/API.md: complete API reference with runnable examples,
event types, locking semantics, tosser wiring, native-backend
drop-down pattern, per-format cheat-sheet
- README links to the new doc
- TUniMessage.Board: conference/board number preserved across
the adapter layer for Hudson/GoldBase/EzyCom/Wildcat/PCBoard
- ma.api.Open: momCreate now ensures the parent directory exists
before the sentinel lock tries to create itself
- tests/tools/make_hudson_sample: builds a 20-conference / 7793-
message Hudson base at /tmp/ma_hudson_sample from real JAM
source areas. Respects Hudson's 32767 MsgNum ceiling. Emits
CONFERENCES.TXT manifest mapping board numbers to area names.
- tests/tools/verify_hudson_sample: per-board count verifier
that cross-checks against the manifest.
- test_write_existing: append to 291-msg JAM base, 27-msg netmail
dir, and a Hudson base seeded on-demand by copying 50 messages
from the JAM source (no binary samples committed).
- test_pack: no-op Pack preserves JAM count + fields; purge-Pack
drops 5 deleted JAM messages; Hudson seed+mark-deleted+Pack
drops 7 of 50 and survivors stay readable.
- Source trees at ~/fidonet/msg/jam + ~/fidonet/msg/netmail are
never touched; all writes go to /tmp scratch copies.
The TWildcatBase.OpenConference was creating a TMsgDatabase without
populating the SDK globals, causing access violations on MwConfig.
Now calls the full Register sequence: InitWCglobal, LoadMakeWild,
BTInitIsam before opening; BTExitIsam + DisposeWCglobal on close.
test_wildcat reads conferences 0..6 from vendored testdata (copied
to /tmp/ma_wildcat so the source tree stays read-only).
