Ask 1 from fpc-binkp consumer thread: non-storage libraries
(fpc-ftn-transport, fpc-binkp, future fpc-comet-proto / fpc-emsi,
SQL-backed messaging like Fastway) only need TFTNAddress, not the
full 1041-line mb.types. Extract to src/mb.address.pas (~90 lines,
only SysUtils) so they can cp a single file into their project.
mb.types continues to uses mb.address so existing callers see the
type transitively -- BUT FPC does not propagate record-field access
through re-export, so consumers that touch TFTNAddress.Zone/Net/
Node/Point directly must add mb.address to their own uses clause.
All 7 in-tree .uni adapters, 2 examples, 5 test harnesses updated.
No behavioural change. Full suite passes, multi-target build
green (x86_64-linux, i386-{linux,freebsd,win32,os2,go32v2}).
Two stale phrasings on the same line:
- "Unit-name convention follows the Fimail style" -- meaningless;
was a preference expressed during early layout, not a convention
that lives anywhere or means anything to a reader
- "ma.<category>.<name>.pas" -- stale since the 0.5.0 rename
to mb.* (the earlier doc-sweep caught every other instance,
this one was in a line I hadn't touched)
Replaced with a direct statement of the current convention.
NR Message 23 (v0.6.2 verification) flagged 21% of year-09
archive messages mismatching HPT by exactly N leading #13 bytes.
Root cause in mb.kludge.SplitKludgeBlob: the rejoin loop used
`bodyOut = ''` as a proxy for "haven't emitted yet", conflating
"empty string so far" with "no line committed yet". A message
whose raw body started with blank lines (leading CRs) lost
those CRs because the bodyOut-stays-empty branch fired and
suppressed the separator.
Fix: track emission with a dedicated `emitted: boolean` flag.
Once any line has been committed (empty or not), subsequent
lines always use CR as separator. Leading empty lines now
contribute their CRs to the output.
Regression test: test_fuzz_kludge.TestLeadingCrPreserved covers
leading-CR-x2, leading-CR-x1, and kludge-prefix + blank-line +
user-text mixed case.
Expected outcome on NR's re-run: the 2055 messages previously
mismatching by one leading CR and the 15 by 2-4 leading CRs
should all match HPT byte-for-byte post-fix. That pushes the
66% body-parity to ~99.5%.
NR Message 22 verification of v0.6.1 showed the OADDRESS/DADDRESS
fix was only half-applied: my v0.6.1 edit removed the emission from
the uni adapter (JamFromUni), but the native TJamBase.BuildSubFields
in mb.fmt.jam.pas still unconditionally emitted them from the
Msg.OrigAddr / Msg.DestAddr record fields. Two-layer bug.
Fixed at the root:
- BuildSubFields drops the AddSF(JAM_OADDRESS) / AddSF(JAM_DADDRESS)
named-field calls. Addresses still round-trip via INTL FTSKLUDGE
for netmail, implicit envelope for echomail.
- AddExtraSF skip guard no longer includes OADDRESS/DADDRESS, so
callers who genuinely want them can push explicit IDs 0/1 into
Msg.SubFields[] and they emit via the extras path.
Plus a small R2 byte-parity cleanup while I was there: JamFromUni
now emits subfields in the order HPT uses (TRACE + FTSKLUDGE
before SEEN-BY + PATH). JAM-001 doesn't mandate order so any
arrangement was spec-compliant; matching HPT just keeps NR's
byte-compare signal clean.
All 27 consumer tests + 31 fuzz tests green; 6-target build clean.
Closes the last of the per-format fuzz gap Fimail flagged on the
v0.6.1 scan (their LOW priority; "we don't open those formats
today" but good to have before anyone does).
test_fuzz_ezycom (F-EZ-1..4): missing AREA1/ dir, empty MH/MT
files, truncated half-header, garbage-filled bundle.
test_fuzz_wildcat (F-WC-1..3): empty dir, garbage MAKEWILD.DAT
only, garbage MSGDB*/MSGTXT files. The WC SDK's historic
Mark/Release + LogFatalError pattern is the thing we're
testing for -- our wrapper must return False from Open rather
than raising or halting.
Not cutting a release just for these; the behaviour under test
is whether Open refuses cleanly on corrupt input, which was
already the case. Rides along on the next real release as
pinned regression coverage.
Wired into run_tests.sh; all 7 happy-path + 31 fuzz tests green.
Four interlocking fixes that NR's 16-year archive load through
the library surfaced. See CHANGELOG.md#0.6.1 for the full
per-bug writeup with JAM-001 / FTSC references.
1. ParseKludgeLine: first-wins on singleton kludges (msgid /
reply / pid / tid / flags / chrs / tzutc / intl / fmpt / topt
/ area). Quoted SOH-preserving body lines can no longer
overwrite the prefix kludges -- stops tossers flagging
legitimate messages as dupes of the quoted parent.
2. SplitKludgeBlob + BuildKludgePrefix added to Hudson, GoldBase,
PCBoard, EzyCom, Wildcat uni adapters (all five were previously
doing u.Body := native.Body verbatim, so msgid / pid / seen-by
/ path / chrs / tzutc / intl / vendor X-* kludges were all
getting dropped on Read AND Write through those formats).
3. JAM-001 spec compliance:
- TJamBase.CalcMsgIdCRC (new); JamFromUni auto-computes
MsgIdCRC and ReplyCRC per message so external readers can
walk reply chains (was writing 0 everywhere, collapsed
threading on GoldED / MsgEd / hptlink).
- PasswordCRC / ReplyCRC default to 0xFFFFFFFF sentinel, not
0 (which is indistinguishable from a real CRC match).
- InitHeader: DateCreated := Now, BaseMsgNum := 1,
PasswordCRC := 0xFFFFFFFF (was all zeros).
- JamFromUni appends trailing CR to body (FTS-0001 requires
CR-terminated lines; PKT-sourced bodies have it, manually-
built bodies may not -- library now ensures it).
- CHRS and TID emitted as JAM_FTSKLUDGE subfields (were
being dropped entirely; CHRS is load-bearing for non-ASCII
display).
- OADDRESS/DADDRESS no longer emitted (spec-valid but adds
34 bytes/msg; HPT/fmail/GoldED don't; callers that need
them can set jam.subfield.0 / .1 explicitly).
- .jlr not created on base-init (HPT creates lazily on first
lastread write; matching removes a file-tree diff).
4. Tests:
- test_fuzz_kludge gains F-KL-4 (quoted ^AMSGID in body
doesn't clobber prefix) and F-KL-5 (SOH mid-line is body
text, enforcing the position-0 rule).
- test_consumer_round1 gains TestJamCrcComputed,
TestJamBodyTrailingCR, TestJamChrsTidRoundTrip,
TestHudsonKludgeRoundTrip, TestGoldBaseKludgeRoundTrip.
- test_roundtrip_attrs updated: Hudson now advertises
msgid/seen-by via the new round-trip path.
All 21 consumer tests pass; 6-target build clean.
Two related doc passes:
1. UpdateMessage docs. docs/API.md "Updating & deleting" section
previously said "Optional operations. Not every backend
implements them -- default returns False." Since v0.6.0 that's
wrong: UpdateMessage has header-metadata-only semantics on
every backend. Rewrote the section to describe the semantics,
typical use cases (flipping attr bits, reply-chain pointers),
and the boundary (body / CtrlInfo / SubFields / inline kludges
untouched; need Delete + WriteMessage for body changes). Added
a note by the .Native escape-hatch example that most previously-
native-only operations now have uni-API equivalents.
2. ma.* -> mb.* namespace sweep. The 0.5.0 rename renamed every
source unit but the docs kept the old names in code-block
`uses` clauses and architecture diagrams. Copy-paste consumers
hitting those would get compile errors. Replaced word-boundary
ma.<api|types|events|lock|paths|kludge|fmt> references in
README.md, docs/API.md, docs/architecture.md, docs/ftsc-
compliance.md, and docs/format-notes/dependencies.md with the
mb.* form. Left docs/PROPOSAL.md alone -- that is the pre-
rename design doc and reads as historical record.
All 75 tests pass; full six-target build clean. Retagging v0.6.0
in place since no downstream has pinned yet.
Noticed right after tagging: the new Squish primitive I added in
the 0.6.0 release was named UpdateMsgHeader (pattern-matching on
the nearby UpdateMsgAttr). Every other backend uses UpdateHeader.
Fixed before v0.6.0 propagated to any downstream pin.
- TSquishBase.UpdateMsgHeader -> TSquishBase.UpdateHeader
- TPCBoardBase: add UpdateHeader(Index, Hdr) as a thin wrapper
over the existing UpdateHeaderAt(Offset, Hdr) so the common
index-based call matches the family. UpdateHeaderAt stays for
callers that already have an offset.
- TWildcatBase: UpdateHeader alias forwards to the existing
UpdateMsgHeader (the latter keeps the WC SDK nomenclature,
the former matches the family).
All three uni adapters call the UpdateHeader name now. All 16
consumer-round1 tests pass; full six-target build clean.
Retagging v0.6.0 in place since no downstream consumer had
pinned to it yet.
NR's Message 17 on the joint inbox flagged that
TMessageBase.UpdateMessage returned False on every format because
no backend overrode DoUpdateMessage. Was blocking nr.linker's
Pass 4 (header-only reply-chain rewrite), the last holdout before
nr.msgbase.* retirement.
Shipped DoUpdateMessage on all eight backends -- not just the
three NR asked for:
JAM -> TJamBase.UpdateHeader (existing primitive)
Squish -> new TSquishBase.UpdateMsgHeader rewrites full
SqMsgHdr in place, preserves CtrlInfo + body
*.MSG -> TMsgFile.UpdateHeader rewrites 190-byte NetMail
Hudson -> UpdateHeader writes HudsonHdrRec at idx*siz
GoldBase -> UpdateHeader, ReplyTo/SeeAlsoNum map to
goldbase.prevreply/nextreply attrs
PCBoard -> UpdateHeaderAt rewrites PCBMsgRec block; RefNum
goes through LongToBSReal encoding on write
EzyCom -> UpdateHeader rewrites EzyMsgHdrRecord
Wildcat -> WC SDK UpdateMsgHeader via the vendored wrapper
Semantics (header-metadata only, documented in each override):
apply attrs from incoming TUniMessage to header slots, leave
body / CtrlInfo / SubFields / inline kludges alone. Callers
needing body changes should Delete + WriteMessage.
Adjacent fix: PCBoard DoOpen now auto-initialises an empty
128-byte PCBBaseRec + empty .IDX on momCreate so a base can be
created from scratch via mb.api. Previously needed pre-
population by another tool.
Tests: seven UpdateMessage regression tests in
test_consumer_round1 (one per backend that supports create from
scratch; Wildcat uses the vendored sample base at
tests/data/wildcat/). All 16 consumer tests pass across six
targets.
NR spotted a read-compat gap while verifying TEST 8 byte-diff
regions: when a Squish base was written by HPT (or any tosser
that places SEEN-BY / PATH inline in the message body rather than
in the SMAPI CtrlInfo area), consumers reading via mb.api got
attr.seen-by / attr.path empty and the raw kludge lines embedded
in Uni.Body.
SquishToUni was calling ParseCtrlInfo on s.CtrlInfo (which catches
writer-emitted-to-CtrlInfo metadata) but nothing equivalent on
s.Body. Fix: SplitKludgeBlob(s.Body, plainBody, attrs) after the
ParseCtrlInfo call -- same pattern mb.fmt.msg.uni already uses for
always-inline-kludge situations.
Either placement now populates the attribute bag; Uni.Body is
always pure user text. CtrlInfo-borne values are extracted first;
if the same key also appears in body epilogue the values
concatenate with #13 via AppendAttr -- no legitimate writer does
this, but the semantics are safe.
Test: test_consumer_round1.TestSquishHptBodyEpilogue writes a
Squish message with MSGID in CtrlInfo and SEEN-BY/PATH in the body
epilogue via the low-level API, reopens via mb.api, asserts all
three attrs populate and body is plain text.
Thanks to NR for catching this during TEST 8 verification.
Sibling of cd2cc90. cd2cc90 fixed BuildKludgePrefix in mb.kludge
(the shared inline-kludge emitter used by PKT / *.MSG / Squish
body-epilogue). Squish has a second, Squish-specific emission
path in BuildCtrlInfo that writes kludges into the SMAPI CtrlInfo
area; that path was still emitting '^AFLAGS: <flags>' with a
colon, so any Squish message written with a FLAGS value carried
the same FTS-4001-violating output cd2cc90 fixed in the other
path. Caught by review while triaging NR's TEST 8 byte-diff
repro (analysis.md in that bundle didn't flag it because the
test message had no FLAGS set).
Fix: drop the colon from the AppendCtrlSingle call for FLAGS in
BuildCtrlInfo. One-line, matching cd2cc90's shape.
Also documented in CHANGELOG: NR's TEST 8 priority regions
(AreaTag base-header fill, SEEN-BY/PATH in CtrlInfo vs body-
epilogue) intentionally left as-is -- both are spec-tolerant per
FTS-5005 and our current choices are SMAPI-native. The residual
2-byte diff vs HPT is cosmetic and non-blocking per NR's own
framing.
Rolls up two consumer-surfaced interop fixes that landed earlier
today (cd2cc90 FLAGS space-separator, ee0f1ca AREA: envelope
recognition) into a tagged release so NR can stabilise its pin.
Both caught by NetReader's hpt-vs-nr byte-diff harness. See
CHANGELOG.md for the full writeup.
No API change. Pure behaviour fix.
AREA: is the echomail transport envelope -- FTS-0004 puts it at the
head of the packet-message body, before any ^A kludges. It has the
same no-^A-prefix shape as the FTS-4 trailing control lines (SEEN-BY:
/ PATH:) the parser already handled.
Without this, SplitKludgeBlob would leave AREA:<tag> sitting in the
"body" output and msgbase writers would store the echomail
envelope-tag as the first 12+ characters of the on-disk body. HPT
and every other tosser strip AREA: before write since the area tag
is implicit in the destination container (JAM directory, Squish
basename, SDM directory).
Fix: add AREA: to isTrailControl, add 'area' to the dispatch case.
The attribute bag now carries area under the 'area' key (same slot
0.5.0 already populates from `MessageBaseOpen`'s area-tag argument
on reads), consumers that need the tag read it there.
Broke test_uue_hpt_compare when NR's `post -u` path built a body
blob starting with "AREA:<tag>" and handed it to WriteMessage via
SplitKludgeBlob; the envelope leaked through into the JAM body,
inflating decoded UUE by the envelope length per section. With
the parse fix every consumer now gets a clean body + a populated
'area' attribute.
FTS-4001 §4 specifies `^AFLAGS flag_string`, space-separated, no
colon. fmail's jamfun.c:321 ("FLAGS ") and ftr.c:113
("\1FLAGS DIR\r") both emit the same form; HPT's ctrl-buffer
parser accepts it verbatim without a colon.
BuildKludgePrefix was previously emitting `^AFLAGS: DIR` with a
colon, which caused consumer parsers (NR's ParseKludges, HPT's
getFlags, fmail's getKludge) to miss the FLAGS kludge entirely.
In NR that manifested as FLAGS DIR / IMM routing being silently
ignored -- the message would fall into normal routing instead of
direct/immediate delivery, and the outbound pkt's destination
node ended up unset (from the route resolver rather than the
message's actual destination).
One-line fix: drop the colon, keep the rest of the kludge
emission unchanged. MSGID: / REPLY: / PID: / CHRS: / TZUTC:
keep their colons per their respective specs (FTS-0009, FRL-1028,
FSC-0054, FSC-0093).
Process gap fix. Up to today the library had no version constant
and no git tags -- downstream consumers were pinning commit
hashes, which is fragile (any force-push or rebase invalidates
the pin silently, and the hash doesn't convey "breaking" vs
"patch"). Two pieces land together:
src/mb.version.pas MB_VERSION = '0.5.1' + split major/minor/patch
constants. Included from build.sh so every
target gets it.
CHANGELOG.md Index of prior milestones (0.3.0 through 0.5.0)
with the commit each one maps to, and the
release notes for 0.5.1 (today's security-
hardening round).
Retroactive tags follow in the next invocation. Going forward:
every release bumps MB_VERSION and places a matching v<ver> tag
in the same commit.
README "Status" section now points consumers at CHANGELOG.md and
the v0.5.1 tag.
NR asked (Message 11 in the joint inbox) for a way to round-trip
TSquishMessage.Replies[1..MAX_REPLY] through the attribute bag so
nr.linker can retire its nr.msgbase.squish.pas and write reply-
chain metadata via mb.api like it already does for JAM and SDM.
Mirror JAM's naming for uniformity across formats:
squish.replyto -- parent (scalar; existed already)
squish.reply1st -- first child (scalar, = Replies[1])
squish.replynext -- remaining chain (list, = Replies[2..MAX_REPLY])
JAM's `replynext` is a single longint because JAM walks a linked
list sibling-to-sibling. Squish stores all direct children on the
parent, so `replynext` here is a LIST attribute (via TMsgAttributes
GetList/SetList). Same key names, shape reflects the on-disk
truth -- consumers that only care about the primary reply hit the
scalar on both formats; consumers that need the full chain
(nr.linker) call GetList on Squish and walk sibling records on JAM.
SquishFromUni now rebuilds Replies[] from these keys instead of
unconditionally zeroing the array, closing the write-side drop
that blocked NR's migration.
ClassSupportedAttributes advertises the new keys alongside the
existing `squish.umsgid`.
Test: test_consumer_round1.TestSquishReplyChain -- writes a
message with reply1st=101 and replynext=[102,103,104], closes,
reopens, reads, and asserts the full chain survives.
Audit pass over the vendored Wildcat! 4 SDK (~11 K LoC, circa 2003).
Most findings are either defended at the wrapper layer already
(mb.fmt.wildcat clamps MsgBytes) or live in DOS-era 16-bit segment
math that is unreachable under 32/64-bit FPC. One credible DoS did
warrant a direct patch:
vrec.pas (IsamDeleteVariableRec): the repeat-loop walks a
NextRefNr chain with no cycle guard. A corrupted .MDF that points
NextRefNr back at an earlier record caused the deleter to loop
forever, deleting the same records repeatedly. Added a 10 M hop
cap matching the pattern used in the Squish ReIndex fix.
SECURITY.md: documents the threat model for the vendored code,
what is defended (MsgBytes clamp, O_NOFOLLOW on lock file, new
cycle cap), and what is not (arbitrary-offset Seek via
UserConfData, tree-walks below the variable-record layer,
FileOpen without O_NOFOLLOW on data files). Consumers get
deployment guidance: run Wildcat under an isolated account, mount
externally-sourced bases read-only in a sandbox.
New tests/adversarial/ suite covers each driver plus mb.kludge with
crafted-input scenarios: empty files, truncated headers, garbage
payloads, oversized length fields, infinite-loop bait. The
invariant under test is graceful degradation: no crash, no hang,
no OOM. Every allocation caps, every loop terminates, every
unreadable record returns False cleanly.
Coverage:
test_fuzz_jam 7 cases (.JHR/.JDT/.JDX/.JLR corruption)
test_fuzz_squish 5 cases (clen underflow, 2 GB clen, garbage idx)
test_fuzz_hudson 3 cases (bundle-file corruption)
test_fuzz_goldbase 2 cases
test_fuzz_pcboard 2 cases
test_fuzz_msg 4 cases (50 MB no-NUL body, strange names)
test_fuzz_kludge 3 cases (100 K CRs, 1 M CRs, legit round-trip)
run_tests.sh builds and runs them after the happy-path suite.
All 26 fuzz cases pass; all 47 existing tests still pass.
Audit pass targeting attacker-controlled binary inputs (.JHR, .SQD,
*.MSG, Wildcat ISAM) and shared-directory sentinels. Caps bound
allocations driven by in-file length fields; a few forward-only
invariants bound chain walks; O_NOFOLLOW plugs a lock-file
symlink-swap window.
JAM (mb.fmt.jam): cap SubfieldLen (ReadSubFields) and TxtLen
(ReadBody) at 64 MiB. Reject negative TxtOffset before seek.
Squish (mb.fmt.squish): reject clen/msg_length outside the frame,
and clen > msg_length - SizeOf(SqMsgHdr) (prevents bodyLen
underflow into a negative SetLength argument). ReIndex now
rejects non-forward next_frame and caps total hops.
Wildcat (mb.fmt.wildcat): clamp Hdr.MsgBytes to SizeOf(TMsgText)
in ReadBody so Move() cannot read past the fixed-size buffer.
*.MSG (mb.fmt.msg): cap ReadBody growth at 16 MiB; pre-size the
AnsiString so concatenation is O(n), not O(n^2).
mb.kludge: pre-count CRs so SplitKludgeBlob is O(n) instead of
O(n^2), and cap the parsed line count at 10 K.
mb.lock: POSIX fpOpen now passes O_NOFOLLOW so a symlink that an
attacker drops in place of the sentinel between FileExists and
fpOpen does not redirect us. Advisory flock semantics unchanged.
mb.paths: new IsSafePathComponent / EnforceSafePathComponent
helpers. Reject empty, '.', '..', absolute, drive-prefixed, or
separator-bearing tails; used by callers that accept area tags or
filenames from outside data.
NR caught a real durability gap during migration prep: the
sequence
base.WriteMessage(msg);
source.MarkSent(srcMsg);
looks atomic, but the OS write buffer hasn't necessarily reached
the platter when MarkSent runs. Crash in that window = silent
message drop.
Add `Sync` virtual on TMessageBase (default no-op for read-only
and in-memory backends). Six writable backends override and
flush every open stream they own via SysUtils.FileFlush
(fpfsync on Unix, FlushFileBuffers on Windows):
JAM .JHR / .JDT / .JDX / .JLR
Squish .SQD / .SQI / .SQL
Hudson msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.BBS
PCBoard MSGS file + index
EzyCom header + text
GoldBase msginfo / msgidx / msghdr / msgtxt / msgtoidx / LASTREAD.DAT
MSG inherits no-op (per-write open/close — buffer flushes on
close, but dir entry isn't fsynced; future enhancement).
Wildcat inherits no-op (legacy `file` IO, not TFileStream).
Helper for backend authors: TMessageBase.FlushStream(S: TStream)
class method that handles the TFileStream cast + nil-safety.
`Sync` raises after Close (data is finalized; nothing to flush).
Test: TestSyncWriteable in test_consumer_round1 -- writes a
message via JAM and Squish, calls Sync (no raise), Close, calls
Sync again (raise expected).
docs/API.md: new "Sync (durability)" section explaining the
commit-after-fsync pattern with the canonical example.
Symmetric to fpc-ftn-transport TPktWriter.Sync (commit ee8c6ad)
that NR's review prompted on the transport side.
Suite: 48/48 (added TestSyncWriteable to test_consumer_round1).
Bug-hunt round (test_hwm Hudson per-(user,board) round-trip
appeared to hang at high UserId values). Root cause: SetLastRead
extended the .LRD file by writing zeros one byte at a time:
for i := 1 to gap do FLrStream.Write(zero, 1);
For UserId=60001, gap is ~24M bytes (Hudson) or ~60M bytes
(GoldBase) — millions of syscalls. Replace with
`FLrStream.Size := needSize`, which on Unix calls ftruncate
(sparse file, unwritten slots read as zero) and on Windows
calls SetEndOfFile. One syscall instead of millions.
Bonus: build.sh UNITS array still listed the pre-rename ma.*
paths after 0.5.0 — fresh clone + ./build.sh would have failed
immediately. Updated to mb.* paths and added the missing
mb.kludge.pas entry (was always missing — never built into a
.ppu cross-target).
Suite: 47/47 (test_hwm now finishes in milliseconds rather
than minutes).
Across-the-board rename so the unit prefix matches the repo
name (mb = msgbase). Brings naming into line with
fpc-ftn-transport's tt.* prefix and avoids the historical
"ma" abbreviation that meant nothing to new readers.
Files renamed via git mv:
src/ma.{api,events,kludge,lock,paths,types}.pas
-> src/mb.{...}.pas
src/formats/ma.fmt.{jam,squish,hudson,msg,pcboard,ezycom,
goldbase,wildcat,wcutil}{,.uni}.pas
-> src/formats/mb.fmt.*.pas
All `unit ma.X` declarations and `uses ma.X` clauses rewritten
to `mb.X` across src/, examples/, tests/.
Suite: 47/47 (read 7, hwm 11, lock 4, pack 4, write 5,
wildcat 5, consumer_round1 5, batch's gone w/ PKT relocation,
plus testutil).
Consumer impact: anyone with `uses ma.api;` etc. needs to
update to `uses mb.api;`. No semantic changes; a search/replace
on the consumer's source tree is the only migration step.
NR's notes (~/.MSGAPI_MSGS.md round 3) align this against
their already-pinned 8130b40; the next NR pin bump rolls in
both this rename and any further work in one step.
Pack in fpc-msgbase means "compact the message base" — wrong
semantic for fpc-ftn-transport's TPktWriter, which is doing a
transactional commit (atomic .tmp -> .pkt rename), not compaction.
Add distinct event types so writers across the ecosystem can
signal commit/rollback without overloading Pack semantics.
Additive: existing Pack* types unchanged.
Removes all PKT code from fpc-msgbase. The wire format and its
container concerns now live in the sibling fpc-ftn-transport
library (units tt.pkt.format, tt.pkt.reader, tt.pkt.writer,
tt.pkt.batch). Pair this commit with fpc-ftn-transport's
0.2.0 (commit 6bb71a6).
Why: the previous "reader here, writer there" split (briefly
landed in 0.3.5) baked in a coupling that didn't survive a
fresh look. The writer reached into fpc-msgbase for types,
the wire format lived in the wrong house, and consumers reading
fpc-msgbase saw "PKT support" that was actually only half-
support. Cleanest split: PKT is a wire format, both directions
belong with the wire-format-aware library; fpc-msgbase becomes
purely real message bases (Hudson / JAM / Squish / MSG /
PCBoard / EzyCom / GoldBase / Wildcat).
Also a cleaner separation-of-concerns story: a BBS that just
reads JAM/Squish never needs fpc-ftn-transport. A pure store-
and-forward node doing only ArcMail unbundle never depends on
storage formats. Each library = one concern.
Removed:
src/formats/ma.fmt.pkt.pas -> tt.pkt.format
src/formats/ma.fmt.pkt.uni.pas -> tt.pkt.reader
(TPktMessageBase -> TPktReader)
src/ma.batch.pas -> tt.pkt.batch
(TPacketBatch class name unchanged)
tests/test_batch.pas -> tests/test_pkt_writer.pas
(consolidated PKT tests)
examples/example_tosser.pas -> moves with the batch helper
Reduced in src/ma.types.pas:
- PacketRecord
- FlavourType / FlavourTypeSet / DateTimeArray
- FlagsToFido / FidoToFlags
- VersionNum (PKT-product-code stamping)
All moved to tt.pkt.format.
Kept in src/ma.types.pas:
- mbfPkt enum value (so tt.pkt.reader can register the backend
with the unified-API factory; consumers still use the
standard MessageBaseOpen(mbfPkt, ...) shape)
Migration for vendoring consumers:
before: after:
uses ma.fmt.pkt; uses tt.pkt.format;
uses ma.fmt.pkt.uni; uses tt.pkt.reader;
uses ma.batch; uses tt.pkt.batch;
(no writer surface) uses tt.pkt.writer;
TPktMessageBase TPktReader
TPktFile, TPktMessage, (unchanged class names)
TPktHeaderInfo, etc.
TPacketBatch (unchanged)
Docs sweep:
- README: PKT row called out as "moved to fpc-ftn-transport";
TPacketBatch removed from features.
- docs/architecture.md: layer diagram drops PKT + ma.batch;
new sibling-library box added for fpc-ftn-transport.
- docs/attributes-registry.md: PKT column dropped from per-
format support matrix; pointer to fpc-ftn-transport.
- docs/API.md: PKT cheat-sheet entry redirects to
fpc-ftn-transport; TPacketBatch section reduced to a
"moved" pointer with the new uses-clause shape.
- docs/ftsc-compliance.md: Type-2 / 2+ / 2.2 / AuxNet rows
annotated as living in tt.pkt.format.
Suite: 47/47 across 9 programs (was 9 with test_batch; now 9
with the PKT bits dropped from test_consumer_round1 and
test_hwm). All other tests untouched.
Decision 2026-04-18: container-level packet concerns (atomic
.tmp/rename, multi-pkt rotation, BSO .flo/.bsy, ArcMail bundles,
packet-header construction) belong in a new sibling library
fpc-ftn-transport, not in fpc-msgbase. ma.fmt.pkt(.uni) stays
here as the per-message reader/iterator surface.
This commit updates the PKT entry in docs/API.md's format
cheat-sheet so consumers reading it today know where the write-
side container API will live, instead of expecting it under
fpc-msgbase eventually. fpc-ftn-transport itself doesn't exist
yet -- this is a scope-boundary signal.
Driven by NetReader / Fimail consumer-round-1 feedback: the
storage-vs-transport split that already separates fpc-msgbase
from comet generalizes well; PKT is 70% transport with a
message-iteration angle, and the angle is the only reason it
fit here at all.
No code change.
Five consumer-feedback items, one milestone:
(1) Shared FTSC kludge plumbing in src/ma.kludge.pas
ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix,
BuildKludgeSuffix. Single source of truth for kludge naming,
INTL/FMPT/TOPT recognition, and the kludge.<lowername>
forward-compat passthrough. Eliminates the four near-identical
parsers MSG/PKT/Squish were carrying; JAM's FTSKLUDGE subfield
walking also routes through ParseKludgeLine so its unknown
kludges land in the same `kludge.<name>` slot as the others.
Bug fix folded in: the parser previously split kludge name from
value at the first ':' it found, which broke INTL (the value
contains an FTN address with ':' in it). Now picks the earlier
of space and colon, which handles both colon-form ("MSGID: foo")
and space-form ("INTL <to> <from>") kludges correctly.
(2) INTL / FMPT / TOPT slots in attributes registry
FSC-4008 cross-zone routing kludges every netmail tosser carries.
Added to JAM/Squish/MSG/PKT capability lists, parsed natively,
emitted on Write. Round-trip covered by tests.
(3) Unified `kludge.*` namespace for unknown FTSC kludges
Squish's `squish.kludge.<name>`, MSG's `msg.kludge.<name>`, and
PKT's `pkt.kludge.<name>` all collapse to plain `kludge.<name>`.
Consumers find passthrough kludges without switching on format.
JAM's numeric `jam.subfield.<id>` stays — those are JAM-specific
binary subfields, not FTSC-form kludges.
(4) `area` auto-populated from base.AreaTag on Read
When the caller passes AAreaTag to MessageBaseOpen (or sets
the AreaTag property post-construction), every successful
ReadMessage fills msg.Attributes['area'] unless the adapter
already populated it from on-disk data (e.g. PKT AREA kludge).
Saves echomail consumers from copying AreaTag into every
message attribute manually.
(5) TMsgAttributes multi-line helpers
GetList / SetList / AppendListItem on TMsgAttributes for the
multi-instance attributes (seen-by, path, via, trace) that
store with #13 between entries. Consumers don't have to roll
their own split/join.
Plus two PKT polish items from the same feedback round:
(6) ma.fmt.pkt.uni.DoWriteMessage now raises EMessageBase
explicitly with a pointer to the Native API instead of
silently returning False.
(7) TPktFile.CreateFromStream / CreateNewToStream constructors
accept any TStream (with optional ownership), so unit tests
that round-trip via TMemoryStream don't have to tempfile-dance.
FStream is now TStream; FOwnsStream gates Free in destructor.
TStringDynArray moved from ma.api.pas to ma.types.pas so both
the capabilities API and the new attribute helpers can share it.
Docs sweep:
- docs/attributes-registry.md: intl/fmpt/topt added; unknown-kludge
convention documented; multi-line helper section added.
- docs/architecture.md: ma.kludge layer surfaced; .uni adapter
registration gotcha called out loudly with the recommended
uses clause; area auto-pop documented.
- docs/API.md: TUniMessage section rewritten for Body+Attributes
model (was still pre-0.2); HWM API documented; PKT cheat-sheet
notes Native + CreateFromStream; tests/programs list updated.
- README.md: Building section flags the .uni gotcha first
thing; ma.kludge added to features.
tests/test_consumer_round1.pas: 7 new tests covering INTL/FMPT/
TOPT round-trip on JAM/Squish/MSG, area auto-pop, GetList/SetList/
AppendListItem, PKT raise, and TPktFile in-memory stream
round-trip.
Suite: 47/47 across 10 programs (test_consumer_round1 adds 7).
EzyCom's per-user state lives in the BBS user records, not in
the message base. There is no msg-base-side LASTREAD file to
plumb (cf. Hudson's LASTREAD.BBS or JAM's .JLR). Per the design
principle adopted earlier ("if a format wasn't designed with
HWM, it doesn't get one — that's on the BBS"), EzyCom stays at
-1 instead of getting a faked sidecar.
Final HWM coverage map:
JAM ✓ .JLR (CRC32-keyed)
Squish ✓ .SQL (CRC32-keyed)
Hudson ✓ LASTREAD.BBS (per user-id, per board)
GoldBase ✓ LASTREAD.DAT (per user-id, per board)
EzyCom -1 BBS-side state, no msg-base file
Wildcat -1 SDK has no per-user HWM primitive
PCBoard -1 USERS file lastread, deferred
MSG -1 spec has no HWM concept
PKT -1 spec has no HWM concept
4 native of 9 formats — covers JAM (most common), Squish (second
most), and the QuickBBS family. NetReader and similar consumers
fall back to their own state for the remaining 5 (e.g. dupedb
keyed by area).
README.md feature bullet updated to reflect the four native
formats.
Suite still 40/40 across 9 programs; no code change in this
commit.
Adds per-(user, board) HWM for the QuickBBS-family multi-board
formats. The same physical Hudson/GoldBase base file set holds
ALL boards (1..200 for Hudson, 1..500 for GoldBase) in one
LASTREAD.BBS / LASTREAD.DAT file, indexed by user number with
one word slot per board. Caller has to provide both pieces of
context before HWM operations make sense:
- base.MapUser('NetReader', 60001) - pick a numeric user ID
- base.Board := 5 - which board this scan is for
src/ma.api.pas:
- New TMessageBase.Board property (longint, default 0).
- Single-area formats (JAM, Squish) ignore it.
- Multi-board formats return -1 from GetHWM when Board <= 0.
src/formats/ma.fmt.hudson.pas:
- New HudsonLastRead record matching QuickBBS LASTREAD.BBS layout.
- TJamBase pattern: FLrStream lazy + EnsureLrStream +
GetLastRead(user, board) + SetLastRead(user, board, msgnum).
- SetLastRead extends file with zeros to reach the user slot,
matching QuickBBS convention.
- Uses fpOpen on Unix (same FPC auto-flock workaround as Squish).
src/formats/ma.fmt.goldbase.pas:
- Same shape, GoldBaseLastRead with GOLDBASE_MAX_BOARDS (500)
slots, file is LASTREAD.DAT.
Both .uni adapters wire DoSupportsHWM/DoGetHWMById/DoSetHWMById
to the new native methods, gating on Board > 0.
tests/test_hwm.pas: 3 new tests covering Hudson + GoldBase:
- TestHudsonRequiresMapUserAndBoard verifies -1 returns when
MapUser missing, Board missing, or both.
- TestHudsonSetGetPersistence covers two users on two boards
with cross-session persistence.
- TestGoldBaseSetGet covers a high board number (250) to
exercise the wider GOLDBASE_MAX_BOARDS range.
Updated docs/architecture.md HWM coverage map: Hudson and
GoldBase moved from deferred to native. EzyCom still deferred
(per-area layout differs); Wildcat/PCBoard still -1.
Suite: 40/40 across 9 programs (test_hwm now 11/11).
NetReader and similar consumers can now register tossers as
high-numbered users (60000+) and walk per-board HWM the way
Allfix has historically done. Tossers coexist with human BBS
users in the same LASTREAD file (different user slots).
Documents the HWM API in architecture.md and surfaces it in
README.md's feature list. Includes the auto-bump pattern, the
multi-tenant convention (each tosser registers as a named user
in the same lastread file), and the per-format coverage map.
Coverage decisions for 0.3.x:
JAM -- native (.JLR) [shipped 0.3.0]
Squish -- native (.SQL) [shipped 0.3.1]
MSG / PKT -- spec has no HWM, returns -1 [structural]
PCBoard -- USERS file too entangled, deferred
Wildcat -- WC SDK exposes only per-message MarkMsgRead,
no per-user HWM primitive; defer until either
the SDK gains the call or we reverse-engineer
the user-conference state file
Hudson -- LASTREAD.BBS is per-(user, board) and the
GoldBase base instance doesn't carry board context;
EzyCom needs API design before impl
For deferred formats, GetHWM honestly returns -1 and the caller
falls back to its own state (e.g. NR's dupedb keyed by area).
This matches the "no fakery" principle: don't pretend a format
supports HWM when it doesn't, and don't silently sidecar in a
location consumers can't discover.
The 0.3.0 / 0.3.1 trio gives NetReader native HWM coverage for
the two formats that account for the overwhelming majority of
real-world FidoNet areas (JAM, Squish). Everything else falls
back to dupedb.
No code changes in this commit -- docs only.
Adds Squish HWM via the .SQL lastread file (CRC32-keyed by
lowercased username, identical layout to JAM's .JLR per
Squish.doc).
src/formats/ma.fmt.squish.pas:
- New SqLastRead record type matching the Squish.doc spec.
- TSquishBase gains FSqlStream (lazy) + EnsureSqlStream +
GetLastRead/GetHighRead/SetLastRead methods, mirroring JAM's
.JLR pattern.
- Close releases the lastread stream alongside .SQD/.SQI.
src/formats/ma.fmt.squish.uni.pas:
- TSquishMessageBase wires DoGetHWMByName / DoSetHWMByName,
reusing TJamBase.CalcUserCRC for the (shared) CRC32 algorithm.
- DoSetHWMByName preserves HighReadMsg monotonicity.
src/ma.lock.pas:
- TMessageLock gains optional APreserveSentinel constructor flag.
- Release no longer unlinks the sentinel when the flag is set.
- Required for Squish because the .SQL file is BOTH the lock
sentinel and the lastread store; deleting it on lock release
would wipe HWM data on every Open/Close cycle.
src/ma.api.pas:
- TMessageBase.Create passes APreserveSentinel = (Format = mbfSquish).
src/formats/ma.fmt.squish.pas (low-level open):
- EnsureSqlStream uses fpOpen directly on Unix instead of FPC's
FileOpen wrapper. FileOpen defaults to a fpflock that conflicts
with ma.lock's existing advisory lock on the same file (EAGAIN
on every attempt). fpOpen bypasses the auto-flock; cross-process
safety lives in ma.lock and doesn't need duplicating here.
tests/test_hwm.pas:
- New SeedSquish helper.
- TestSquishCapability + TestSquishSetGetPersistence verify
capability flag, set/get round-trip, multi-user independence,
and persistence across Open/Close.
Hudson, GoldBase and EzyCom were originally scoped for this
milestone but their per-(user, board) lastread layout needs more
design work (the LASTREAD.BBS format is shared across all 200
boards in a single per-user record, and TMessageBase doesn't
carry a "board" context). Deferred to a follow-up milestone.
Wildcat (SDK call) coming in 0.3.2.
Suite: 38/38 across 9 programs (test_hwm now 9/9 with Squish
coverage).
Adds the per-user High-Water Mark API to TMessageBase:
function SupportsHWM: boolean;
function GetHWM(const UserName: AnsiString): longint;
procedure SetHWM(const UserName: AnsiString; MsgNum: longint);
procedure MapUser(const UserName: AnsiString; UserId: longint);
property ActiveUser: AnsiString;
GetHWM returns -1 when the format has no HWM mechanism, the user
isn't registered (number-keyed formats), or no HWM has been set
for that user yet.
Two-flavour native dispatch:
- Name-keyed backends (JAM, later Squish/Wildcat) override
DoGetHWMByName / DoSetHWMByName.
- Number-keyed backends (later Hudson/GoldBase/EzyCom) override
DoGetHWMById / DoSetHWMById; caller must register name->id via
MapUser first.
Auto-bump: when ActiveUser is non-empty, ReadMessage compares the
just-read msg.num to GetHWM(ActiveUser); if higher, calls SetHWM.
Never decrements -- reading a lower-numbered message is a no-op.
Default off (ActiveUser = '').
JAM implementation:
- Adds .JLR (lastread) handling to TJamBase: lazy open via
EnsureLrStream, GetLastRead/GetHighRead/SetLastRead methods,
proper Close cleanup.
- TJamMessageBase wires DoGetHWMByName / DoSetHWMByName to
CalcUserCRC + GetLastRead / SetLastRead. SetHWM also keeps
HighReadMsg monotonic.
Coverage map (this milestone):
- JAM: native ✓
- Squish, Hudson, GoldBase, EzyCom, Wildcat: -1 (planned 0.3.1/2)
- PCBoard, MSG, PKT: -1 (no HWM in spec)
Tests: tests/test_hwm.pas covers SupportsHWM, set/get round-trip,
persistence across Open/Close, auto-bump via ActiveUser (advances),
auto-bump never decrements, MSG/PKT correctly return -1, Hudson
returns -1 even after MapUser (until 0.3.1 lands the impl). 7/7
new tests pass; full suite 38/38 across 9 programs.
Recommended caller pattern:
base.ActiveUser := 'NetReader';
for i := 0 to base.MessageCount - 1 do begin
base.ReadMessage(i, msg);
{ process msg ... HWM auto-tracks the high-water for NetReader }
end;
New docs/attributes-registry.md publishes the canonical attribute
key catalog in four tiers:
1. Universal headers — msg.num, from, to, subject, date.*, addr.*,
area, board, cost. Every Fido format carries them.
2. Canonical attribute bits — attr.private, attr.crash, etc.,
mapped to/from the FTS-1 attribute word.
3. FTSC kludges — msgid, replyid, pid, tid, flags, chrs, tzutc,
seen-by, path, via. Multi-line keys use #13 between lines.
4. Format-specific — jam.*, squish.*, hudson.*, goldbase.*, ezy.*,
pcb.*, wildcat.*, pkt.*, msg.*. Each backend's namespace.
Plus a per-format support matrix showing which keys each backend
carries. Authoritative source remains each backend's
ClassSupportedAttributes -- the matrix can drift; SupportsAttribute()
is the runtime-correct query.
docs/architecture.md TUniMessage section rewritten:
- Documents the strict two-area model (Body + Attributes only).
- Body holds only the message text, never kludges or headers.
- Library never composes presentation -- consumers walk Attributes
and assemble their own display.
- Adds the capabilities API section pointing at the registry.
- Removes the stale "kludge lines intact and CR-separated" promise
the previous adapter implementations didn't honor.
docs/PROPOSAL.md flags the original Extras-bag section as
SUPERSEDED 2026-04-17, points to the registry + architecture docs
as the live design. Original text retained as historical context
since it captures the conversation that drove the redesign.
README.md:
- Features list now leads with the lossless two-area model and the
capabilities API.
- Adds a Status note flagging 0.2 as a breaking change vs 0.1 with
a one-paragraph migration sketch (msg.WhoFrom -> Attributes.Get
('from'), etc.).
- Documentation index links to the new registry doc.
Adds tests/test_roundtrip_attrs.pas covering:
1. Capabilities API smoke test — confirms SupportsAttribute('msgid')
returns true on JAM/Squish/MSG/PKT, false on Hudson/GoldBase/
EzyCom/Wildcat/PCBoard. Confirms backend-private keys are gated
correctly (Squish.SupportsAttribute('jam.msgidcrc') = false).
2. Per-format kludge round-trip across all 5 storage formats —
builds a synthetic message with universal headers + FTSC kludges
(msgid, replyid, pid, flags, multi-line seen-by + path), writes,
reopens, reads back, asserts every key the backend's capability
list advertises survives byte-for-byte. Backends that don't
support a given key are silently skipped via SupportsAttribute
gating so the test exercises each format's actual contract.
3. Cross-format JAM → Squish copy — seeds JAM with the kludge
message, copies to a fresh Squish base via the unified API,
reopens both, asserts:
- intersection of capabilities lists is preserved verbatim
(msgid, seen-by, path, etc. all survive JAM → Squish)
- jam.* keys not in Squish's capability list are dropped
(no silent corruption of Squish's data with foreign keys)
Result: 7/7 new tests pass. Total suite now 31/31 across 8 programs.
This is the regression suite that locks the Body+Attributes contract
and proves the showstopper fix holds across every backend.
Hooked into run_tests.sh so CI catches future drift.
Replaces TUniMessage's 13-field flat record with a strict two-area
model: Body holds only the message text; Attributes holds everything
else (from/to/subject/dates/addresses/MSGID/SEEN-BY/PATH/format-
specific fields) as namespaced key/value pairs.
Why this fix is required NOW: the previous JAM adapter dropped
MSGID, ReplyID, PID, Flags, SEEN-BY and PATH on every Read/Write
through the unified API. A NetReader parity test surfaced it (17/21
pass with 4 kludge failures). All 9 adapters had the same bug. For
tossers and scanners the impact is silent corruption: dropped MSGID
→ dupe storms, dropped PATH → mail loops, dropped SEEN-BY → broken
routing. Three downstream consumers (Fimail's codex-transport branch,
NetReader, future Allfix) had halted integration work pending this
fix. Without it, anyone vendoring fpc-msgbase 0.1 ships with a
known-corrupting adapter.
Design choice: per Ken's call, "message is just the message text;
everything else is an attribute, including from/to/subject/dates."
Same architecture as RFC 822 email (headers + body). Each backend
fills attributes it knows on Read; reads attributes it understands
on Write; ignores unknown attributes silently (RFC 822 X-header
semantics). Forward-compatible -- a new backend (e.g. a planned SQL
message store) just adds its own attribute keys; old backends ignore
them.
Composition is the consumer's job. The library never reassembles
Body + Attributes into kludge-laden display text. A BBS that wants
inline kludges walks Attributes and prepends ^aMSGID etc. to its
own display. A tosser that needs MSGID for dupe detection reads
Attributes.Get('msgid') directly -- no body parsing required.
src/ma.types.pas:
- New TMsgAttribute / TMsgAttributes records with Get/SetValue,
typed accessors (GetInt/GetBool/GetDate/GetAddr), Has/Remove,
iteration. Linear-search lookup, fine for the ~30-50 keys per
message. Switch to hash later if profiling shows need.
- Replaced TUniMessage with the minimal Body + Attributes record.
- New UniAttrBitsToAttributes / UniAttrBitsFromAttributes helpers
to bridge the canonical MSG_ATTR_* cardinal bitset to/from
individual `attr.*` boolean keys.
- {$modeswitch advancedrecords} added so records have methods.
src/ma.api.pas:
- New capabilities API: TStringDynArray return type,
ClassSupportedAttributes (virtual class fn, default empty),
SupportedAttributes (instance sugar), SupportsAttribute (per-key
query). Each backend overrides ClassSupportedAttributes with the
static list of keys it knows. Callers query before setting so a
BBS UI can hide controls the underlying backend has no slot for.
src/formats/ma.fmt.*.uni.pas (all 9):
- Rewrote each XxxToUni and XxxFromUni for the new model. Read
populates Attributes with universal/FTSC/format-specific keys per
the attribute registry (to be published in phase 5). Write reads
attributes back and writes native form.
- JAM walks SubFields[] for SEEN-BY/PATH/TZUTC/TRACE plus passthrough
of unknown subfield IDs as `jam.subfield.<id>` for round-trip
safety. Squish parses CtrlInfo (NUL-separated ^A lines) into
individual attributes, rebuilds on Write. MSG and PKT (which keep
kludges inline in body per FTS-1) parse leading ^A lines and
trailing SEEN-BY/PATH out of the body so TUniMessage.Body is
always plain user text; on Write they reassemble the on-disk form.
- Each backend ships ClassSupportedAttributes with its key list.
src/ma.batch.pas: PktToUni signature updated to (in,out var) form.
tests/* + examples/*: migrated all callers from Msg.WhoFrom (etc.)
to Msg.Attributes.Get('from'). MakeMsg helpers now use SetValue/
SetBool/SetAddr.
Verified: 24/24 tests pass across all 7 test programs (read,
roundtrip, lock, batch, wildcat, write_existing, pack). Wildcat
walks all 7 vendored conferences clean.
Out of scope (next phases):
- docs/attributes-registry.md publishing the full key list with
per-format support matrix
- cross-format round-trip + capabilities-driven copy test
- update architecture.md / PROPOSAL.md to reflect the new model
Copies the working WildCat 4 conference set (7 conferences,
MAKEWILD.DAT, CONFDESC.*, MSG/MSG0..MSG6.DAT/IX, DATA/ALLUSERS.*,
etc., 228KB total) into the repo so the Wildcat backend test runs
without an external path dependency. Mirrors how the other format
tests carry their own sample data.
test_wildcat.pas SRC constant updated to point at tests/data/wildcat
instead of ~/allfix_dev/...; the test still copies to /tmp/ma_wildcat
before running so the vendored fixture stays untouched.
Project renamed from message_api → fpc-msgbase. Folder, README title,
docs, build.sh, fpc.cfg, and test banners all updated for consistency
with the planned remote at kjgr.io:2222/kenjreno/fpc-msgbase.git.
Also scrubbed claims that backends were "ported from Allfix" or
"match Allfix's msgutil/domsg" — none of this code was ported from
Allfix; it was implemented from FTSC documents and the original
format authors' published specs (jam.txt, squish.doc, pcboard.doc,
EzyCom reference, WildCat 4 SDK headers). Author credits live in
docs/ftsc-compliance.md.
Real interop facts that mention Allfix-the-product stay documented:
the PCB Extra2 sent-bit ($40) Allfix sets when tossing, and the
FTSC-registered product code $EB. These describe external software
behavior we interoperate with, not provenance.
docs/format-notes/hudson.md removed — stale planning doc that
predates the working ma.fmt.hudson backend.
Six-week reshape plan: lossless TMsgRecord with Extras bag, ITsmIO
abstraction, TOutboundBatch, single-callback events, explicit locking,
typed exception tree. Captures the cross-project conversation with
NetReader and the byte-agreement cross-verifier as the first
actionable step. Pre-rename baseline.
- ma.lock.Release now unlinks the .lck sentinel after releasing
the OS lock (SysUtils.DeleteFile to avoid Windows API collision)
- Unit headers trimmed to standard dev-comment form (purpose only)
- docs/API.md: complete API reference with runnable examples,
event types, locking semantics, tosser wiring, native-backend
drop-down pattern, per-format cheat-sheet
- README links to the new doc
- TUniMessage.Board: conference/board number preserved across
the adapter layer for Hudson/GoldBase/EzyCom/Wildcat/PCBoard
- ma.api.Open: momCreate now ensures the parent directory exists
before the sentinel lock tries to create itself
- tests/tools/make_hudson_sample: builds a 20-conference / 7793-
message Hudson base at /tmp/ma_hudson_sample from real JAM
source areas. Respects Hudson's 32767 MsgNum ceiling. Emits
CONFERENCES.TXT manifest mapping board numbers to area names.
- tests/tools/verify_hudson_sample: per-board count verifier
that cross-checks against the manifest.
- test_write_existing: append to 291-msg JAM base, 27-msg netmail
dir, and a Hudson base seeded on-demand by copying 50 messages
from the JAM source (no binary samples committed).
- test_pack: no-op Pack preserves JAM count + fields; purge-Pack
drops 5 deleted JAM messages; Hudson seed+mark-deleted+Pack
drops 7 of 50 and survivors stay readable.
- Source trees at ~/fidonet/msg/jam + ~/fidonet/msg/netmail are
never touched; all writes go to /tmp scratch copies.
The TWildcatBase.OpenConference was creating a TMsgDatabase without
populating the SDK globals, causing access violations on MwConfig.
Now calls the full Register sequence: InitWCglobal, LoadMakeWild,
BTInitIsam before opening; BTExitIsam + DisposeWCglobal on close.
test_wildcat reads conferences 0..6 from vendored testdata (copied
to /tmp/ma_wildcat so the source tree stays read-only).
