Phase 5: attribute registry + arch / proposal / README updates

New docs/attributes-registry.md publishes the canonical attribute key catalog in four tiers: 1. Universal headers — msg.num, from, to, subject, date.*, addr.*, area, board, cost. Every Fido format carries them. 2. Canonical attribute bits — attr.private, attr.crash, etc., mapped to/from the FTS-1 attribute word. 3. FTSC kludges — msgid, replyid, pid, tid, flags, chrs, tzutc, seen-by, path, via. Multi-line keys use #13 between lines. 4. Format-specific — jam.*, squish.*, hudson.*, goldbase.*, ezy.*, pcb.*, wildcat.*, pkt.*, msg.*. Each backend's namespace. Plus a per-format support matrix showing which keys each backend carries. Authoritative source remains each backend's ClassSupportedAttributes -- the matrix can drift; SupportsAttribute() is the runtime-correct query. docs/architecture.md TUniMessage section rewritten: - Documents the strict two-area model (Body + Attributes only). - Body holds only the message text, never kludges or headers. - Library never composes presentation -- consumers walk Attributes and assemble their own display. - Adds the capabilities API section pointing at the registry. - Removes the stale "kludge lines intact and CR-separated" promise the previous adapter implementations didn't honor. docs/PROPOSAL.md flags the original Extras-bag section as SUPERSEDED 2026-04-17, points to the registry + architecture docs as the live design. Original text retained as historical context since it captures the conversation that drove the redesign. README.md: - Features list now leads with the lossless two-area model and the capabilities API. - Adds a Status note flagging 0.2 as a breaking change vs 0.1 with a one-paragraph migration sketch (msg.WhoFrom -> Attributes.Get ('from'), etc.). - Documentation index links to the new registry doc.
2026-04-17 14:35:19 -07:00
parent d7e58932e9
commit 1e253e8a78
4 changed files with 334 additions and 14 deletions
--- a/README.md
+++ b/README.md
@@ -25,6 +25,14 @@ can target a single interface regardless of the underlying format on disk.

 - One `TMessageBase` abstract class — read, write, pack, reindex through the
  same methods regardless of format.
+- **Lossless two-area message model.** `TUniMessage` = `Body` (just the
+  message text) + `Attributes` (key/value bag holding from/to/subject/dates/
+  addresses/MSGID/SEEN-BY/PATH and per-format extras). Same shape as RFC 822
+  email. Round-trip preservation enforced by the test suite.
+- **Capabilities API** — `base.SupportsAttribute('attr.returnreceipt')` lets
+  UIs hide controls the underlying backend has no slot for. Each backend
+  publishes its key list via `ClassSupportedAttributes`. Full per-format
+  matrix in [`docs/attributes-registry.md`](docs/attributes-registry.md).
 - Layered locking: in-process `TRTLCriticalSection` + cross-process advisory
  lock (`fpflock` on Unix, `LockFileEx` on Windows, `.LCK` sentinel fallback)
  + the existing `fmShareDenyWrite` / `fmShareDenyNone` share modes.
@@ -68,11 +76,32 @@ Unit-name convention follows the Fimail style: `ma.<category>.<name>.pas`
 ## Documentation

 - [`docs/API.md`](docs/API.md) — full API reference with examples
- [`docs/architecture.md`](docs/architecture.md) — layered design
+- [`docs/architecture.md`](docs/architecture.md) — layered design + Body/Attributes contract
+- [`docs/attributes-registry.md`](docs/attributes-registry.md) — canonical attribute keys + per-format support matrix
 - [`docs/ftsc-compliance.md`](docs/ftsc-compliance.md) — spec notes
 - [`docs/format-notes/`](docs/format-notes/) — per-format quirks

 ## Status

-Early development. APIs may move until 1.0. Format backends are
-spec-driven implementations validated against real-world sample bases.
+Early development. APIs may move until 1.0.
+
+**0.2 is a breaking change vs 0.1.** `TUniMessage` lost its 13 named
+fields (WhoFrom/WhoTo/Subject/MsgNum/Attr/etc.) in favour of a strict
+`Body` + `Attributes` two-area model. Migration:
+
+```pascal
+{ before }                       { after }
+msg.WhoFrom                       msg.Attributes.Get('from')
+msg.Subject := 'foo';             msg.Attributes.SetValue('subject', 'foo');
+msg.Attr := MSG_ATTR_LOCAL;       msg.Attributes.SetBool('attr.local', true);
+msg.OrigAddr                      msg.Attributes.GetAddr('addr.orig')
+msg.DateWritten                   msg.Attributes.GetDate('date.written')
+```
+
+This change makes the unified API lossless — kludges (MSGID, SEEN-BY,
+PATH, etc.) round-trip cleanly, where 0.1 silently dropped them. See
+[`docs/attributes-registry.md`](docs/attributes-registry.md) for the
+full key catalog.
+
+Format backends are spec-driven implementations validated against
+real-world sample bases.
--- a/docs/PROPOSAL.md
+++ b/docs/PROPOSAL.md
@@ -134,6 +134,20 @@ Caller (BBS, tosser, editor, importer)

 ### `TMsgRecord` with lossless `Extras`

+> **SUPERSEDED 2026-04-17.** The "named fields + Extras bag" hybrid
+> below was rejected during implementation in favour of a stricter
+> two-area model: `Body` holds only the message text; **everything
+> else** (from/to/subject/dates/addresses + every kludge + every
+> format-specific field) is an attribute. See
+> [`docs/attributes-registry.md`](attributes-registry.md) for the
+> key catalog and [`docs/architecture.md`](architecture.md) for the
+> updated TUniMessage contract. The original Extras-bag design is
+> retained below as historical context.
+>
+> The capabilities API discussed in this section landed essentially
+> as proposed (`base.SupportsAttribute(K)` + class-level
+> `ClassSupportedAttributes`).
+
 ```pascal
 TMsgRecord = record
  { Universal fields every format has: }
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -35,20 +35,72 @@ contract. Callers can either:
   `TSquishBase.SqHashName`) when they need behaviour the unified API cannot
   express. Each backend keeps its rich API public.

-## TUniMessage
+## TUniMessage — two-area model

-A single message record covering every backend. Format-specific bit fields
-(Hudson byte attr, JAM 32-bit attr, Squish attr, MSG word attr, PCB status,
-EzyCom dual byte) are mapped into a canonical `Attr: cardinal` bit set
-defined in `ma.types` (`MSG_ATTR_PRIVATE`, `MSG_ATTR_LOCAL`,
-`MSG_ATTR_RECEIVED`, …). Conversion helpers expose the original encoding
-when needed.
+```pascal
+TUniMessage = record
+  Body:       AnsiString;       { only the message text }
+  Attributes: TMsgAttributes;   { everything else, key/value }
+end;
+```

-Dates land in `TDateTime` regardless of how the backend stored them
-(Hudson `MM-DD-YY` strings with 1950 pivot, Squish FTS-0001 strings, JAM
-Unix timestamps, PCBoard / EzyCom DOS PackTime).
+Two areas, no surprises:

-The body is an `AnsiString` with kludge lines intact and CR-separated.
+- **Body** carries the user-visible message text and nothing else.
+  Never kludge lines, never headers, never SEEN-BY/PATH. Always a
+  ready-to-display blob.
+- **Attributes** carries every other piece of data: From, To,
+  Subject, dates, addresses, attribute bits, FTSC kludges (MSGID,
+  ReplyID, PID, SEEN-BY, PATH, …), and per-format extras
+  (`jam.msgidcrc`, `squish.umsgid`, `pcb.confnum`, …).
+
+Same model as RFC 822 email (headers + body). Lossless round-trip
+across Read → Write → Read is enforced by the regression suite in
+`tests/test_roundtrip_attrs.pas`.
+
+**The library never composes presentation.** A BBS that wants to
+display kludges inline walks `Attributes` and prepends `^aMSGID:`
+etc. to its own display. A BBS that hides kludges just shows
+`Body`. A tosser that needs MSGID for dupe detection reads
+`Attributes.Get('msgid')` directly — no body parsing required.
+
+Dates land in `TDateTime` regardless of how the backend stored
+them (Hudson `MM-DD-YY` strings with 1950 pivot, Squish FTS-0001
+strings, JAM Unix timestamps, PCBoard / EzyCom DOS PackTime).
+Stored in attributes as `date.written` / `date.received` via
+`SetDate` / `GetDate`.
+
+Format-specific bit fields (Hudson byte attr, JAM 32-bit attr,
+Squish attr, MSG word attr, PCB status, EzyCom dual byte) are
+unrolled into individual `attr.*` boolean attributes on Read via
+`UniAttrBitsToAttributes` and recomposed on Write via
+`UniAttrBitsFromAttributes` and the per-format `XxxAttrFromUni`
+helpers. The canonical `MSG_ATTR_*` cardinal bitset stays as the
+internal pivot.
+
+### Capabilities API — backend self-description
+
+Each backend declares the canonical list of attribute keys it
+understands via a class function:
+
+```pascal
+class function TMessageBase.ClassSupportedAttributes: TStringDynArray;
+```
+
+Callers query before setting:
+
+```pascal
+if base.SupportsAttribute('attr.returnreceipt') then
+  RenderReceiptCheckbox
+else
+  HideReceiptCheckbox;
+```
+
+Backends silently ignore unknown attributes on Write (RFC 822
+X-header semantics — fine for forward compatibility); the
+capabilities API exists so callers know in advance which keys won't
+survive on a given format. The full per-format support matrix lives
+in `docs/attributes-registry.md`.

 ## Locking

--- a/docs/attributes-registry.md
+++ b/docs/attributes-registry.md
@@ -0,0 +1,225 @@
+# Attribute registry
+
+This document is the source of truth for attribute key names used
+across all `fpc-msgbase` backends.
+
+`TUniMessage` has only two areas:
+
+```pascal
+TUniMessage = record
+  Body:       AnsiString;       { only the message text }
+  Attributes: TMsgAttributes;   { everything else, key/value }
+end;
+```
+
+Every header field, kludge, control line, and per-format flag a
+backend understands lives in `Attributes` under one of the keys
+below.  Backends drop keys they don't recognise on Write (RFC 822
+X-header semantics).  Callers can query `base.SupportsAttribute(K)`
+before setting a key to know up-front which backends carry it.
+
+Naming convention: lowercase, dot-namespaced.  Universal /
+FTSC-defined keys are unqualified (`from`, `msgid`).  Format-
+specific keys are prefixed with the format name (`jam.msgidcrc`,
+`squish.umsgid`, `pcb.confnum`).
+
+## Tier 1 — Universal headers
+
+Every Fido format sets these on Read; every backend reads them on
+Write.  Equivalent to the always-present headers in any classic
+BBS message.
+
+| Key | Type | Meaning |
+|---|---|---|
+| `msg.num` | int | Backend-assigned message number / index |
+| `from` | string | Author name |
+| `to` | string | Recipient name |
+| `subject` | string | Message subject |
+| `date.written` | date | When author wrote the message |
+| `date.received` | date | When local system received |
+| `addr.orig` | ftn | Originating FTN address (`zone:net/node[.point]`) |
+| `addr.dest` | ftn | Destination FTN address |
+| `area` | string | Echo area tag (echomail) |
+| `board` | int | Conference / board number for multi-board formats |
+| `cost` | int | Cost in cents (FTS-1 backends) |
+
+## Tier 2 — Canonical attribute bits
+
+Boolean flags derived from the FTS-1 attribute word, plus extensions
+some backends carry as separate bits.  Always written via
+`SetBool(K, true)`; absent (or false) when not set.
+
+| Key | FTS-1 | Meaning |
+|---|---|---|
+| `attr.private` | 0x0001 | Private message |
+| `attr.crash` | 0x0002 | Crash priority |
+| `attr.received` | 0x0004 | Recipient has read |
+| `attr.sent` | 0x0008 | Sent to destination |
+| `attr.fileattach` | 0x0010 | File attached |
+| `attr.intransit` | 0x0020 | In transit |
+| `attr.orphan` | 0x0040 | No matching destination |
+| `attr.killsent` | 0x0080 | Kill after sending |
+| `attr.local` | 0x0100 | Originated locally |
+| `attr.hold` | 0x0200 | Hold for pickup |
+| `attr.filereq` | 0x0400 | File request |
+| `attr.returnreceipt` | 0x0800 | Return receipt requested |
+| `attr.isreceipt` | 0x1000 | This message is a receipt |
+| `attr.auditreq` | 0x2000 | Audit trail requested |
+| `attr.fileupdreq` | 0x4000 | File update request |
+| `attr.deleted` | — | Tombstoned (per-base) |
+| `attr.read` | — | Marked-read (per-user) |
+| `attr.echo` | — | Echomail (vs netmail) |
+| `attr.direct` | — | Direct flavour |
+| `attr.immediate` | — | Immediate flavour |
+| `attr.locked` | — | Locked (no edit) |
+| `attr.netpending` | — | Pending netmail toss |
+| `attr.echopending` | — | Pending echomail toss |
+| `attr.nokill` | — | Protected from purge |
+
+## Tier 3 — FTSC kludges
+
+Standard FTSC kludge lines, named after the kludge name without
+the leading `^A`.  Multi-line attributes (SEEN-BY, PATH, Via) join
+their lines with `#13`.
+
+| Key | Type | Spec | Meaning |
+|---|---|---|---|
+| `msgid` | string | FTS-9 | Globally unique message ID |
+| `replyid` | string | FTS-9 | Reply linkage to a previous MSGID |
+| `pid` | string | FRL-1004 | Producer ID (creating tosser/editor) |
+| `tid` | string | FSC-46 | Tosser ID |
+| `flags` | string | FRL-1005 | Routing/handling flags |
+| `chrs` | string | FTS-5003 | Character set declaration |
+| `tzutc` | string | FTS-4001 | Time-zone offset from UTC |
+| `seen-by` | multi-string | FTS-4 | SEEN-BY lines (one node-list per line) |
+| `path` | multi-string | FTS-4 | PATH lines (one node-list per line) |
+| `via` | multi-string | FTS-4009 | Via lines (one per relay) |
+
+## Tier 4 — Format-specific keys
+
+These are namespaced and only meaningful to the format that
+produces them.  Other backends ignore them on Write (silently
+dropped — fine).
+
+### JAM
+
+| Key | Type | Meaning |
+|---|---|---|
+| `jam.msgidcrc` | int | Index fast-path CRC of MSGID |
+| `jam.replycrc` | int | Index fast-path CRC of ReplyID |
+| `jam.dateprocessed` | unix-int | Tosser timestamp |
+| `jam.passwordcrc` | int | Per-message password CRC |
+| `jam.cost` | int | JAM-level cost (separate from `cost`) |
+| `jam.timesread` | int | Times-read counter |
+| `jam.replyto` | int | Parent in reply chain |
+| `jam.reply1st` | int | First child in reply chain |
+| `jam.replynext` | int | Next sibling in reply chain |
+| `jam.attribute2` | int | Reserved JAM attribute2 word |
+| `jam.ftskludge` | multi | Passthrough for JAM_FTSKLUDGE subfields |
+| `jam.subfield.<id>` | multi | Passthrough for unknown subfield IDs |
+
+### Squish
+
+| Key | Type | Meaning |
+|---|---|---|
+| `squish.umsgid` | int | UMsgID (per-area unique number) |
+| `squish.utcofs` | int | UTC offset in minutes |
+| `squish.replyto` | int | Reply chain parent |
+| `squish.kludge.<name>` | multi | Passthrough for unknown CtrlInfo kludges |
+
+### Hudson / GoldBase
+
+| Key | Type | Meaning |
+|---|---|---|
+| `hudson.prevreply` | int | Previous message in reply chain |
+| `hudson.nextreply` | int | Next message in reply chain |
+| `hudson.timesread` | int | Times-read counter |
+| `goldbase.prevreply` | int | (GoldBase variant) |
+| `goldbase.nextreply` | int | (GoldBase variant) |
+| `goldbase.timesread` | int | (GoldBase variant) |
+
+### EzyCom
+
+| Key | Type | Meaning |
+|---|---|---|
+| `ezy.extattr` | int | EzyCom ExtAttr byte |
+| `ezy.prevreply` | int | Reply chain prev |
+| `ezy.nextreply` | int | Reply chain next |
+
+### PCBoard
+
+| Key | Type | Meaning |
+|---|---|---|
+| `pcb.refnum` | int | RefNum field |
+| `pcb.status` | int | Raw PCB status byte |
+| `pcb.active` | int | Active flag |
+| `pcb.echo` | int | Echo flag |
+| `pcb.export` | int | Export flag |
+| `pcb.extra2` | int | Extra2 byte (incl. Allfix-sent bit 6) |
+| `pcb.extra3` | int | Extra3 byte |
+| `pcb.hastags` | int | HasTags flag |
+| `pcb.origin` | int | Origin flag |
+| `pcb.readnum` | int | ReadNum word |
+| `pcb.extendedstatus` | int | Extended status byte |
+| `pcb.password` | string | Per-message password (max 12 chars) |
+
+### Wildcat
+
+| Key | Type | Meaning |
+|---|---|---|
+| `wildcat.confnum` | int | Conference number this message lives in |
+| `wildcat.mflags` | int | Raw mFlags word from TMsgHeader |
+
+### PKT
+
+| Key | Type | Meaning |
+|---|---|---|
+| `pkt.cost` | int | Packet-level cost (separate from `cost`) |
+| `pkt.kludge.<name>` | multi | Passthrough for unknown body kludges |
+
+### MSG (FTS-1)
+
+| Key | Type | Meaning |
+|---|---|---|
+| `msg.replyto` | int | Reply chain parent |
+| `msg.nextreply` | int | Reply chain next |
+| `msg.timesread` | int | Times-read counter |
+| `msg.kludge.<name>` | multi | Passthrough for unknown body kludges |
+
+## Per-format support matrix
+
+`X` = the backend's `ClassSupportedAttributes` lists the key.
+Blank = backend has no slot for it; setting the key has no effect
+on Write, and the key won't appear in `Attributes` after a Read.
+
+| Key | JAM | Squish | MSG | PKT | Hudson | GoldBase | EzyCom | PCB | WC |
+|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| `msg.num` | X | X | X | X | X | X | X | X | X |
+| `from` / `to` / `subject` | X | X | X | X | X | X | X | X | X |
+| `addr.orig` / `addr.dest` | X | X | X | X | X | X | X | X | X |
+| `date.written` | X | X | X | X | X | X | X | X | X |
+| `date.received` | X | X |   |   |   |   | X |   | X |
+| `area` | X | X | X | X | X | X | X | X | X |
+| `board` |   |   |   |   | X | X | X |   |   |
+| `cost` |   |   | X |   | X | X | X |   | X |
+| `attr.private` | X | X | X | X | X | X | X | X | X |
+| `attr.crash` | X | X | X | X | X | X | X |   |   |
+| `attr.received` | X | X | X | X | X | X | X |   | X |
+| `attr.sent` | X | X | X | X | X | X | X |   | X |
+| `attr.killsent` | X | X | X | X | X | X | X |   |   |
+| `attr.local` | X | X | X | X | X | X | X | X |   |
+| `attr.hold` | X | X | X | X | X | X |   |   |   |
+| `attr.fileattach` | X | X | X | X | X | X | X |   |   |
+| `attr.returnreceipt` | X | X | X | X |   |   | X |   | X |
+| `attr.deleted` | X | X |   |   | X | X | X | X | X |
+| `msgid` | X | X | X | X |   |   |   |   |   |
+| `replyid` | X | X | X | X |   |   |   |   |   |
+| `pid` | X | X | X | X |   |   |   |   |   |
+| `flags` | X | X | X | X |   |   |   |   |   |
+| `seen-by` | X | X | X | X |   |   |   |   |   |
+| `path` | X | X | X | X |   |   |   |   |   |
+| Format-specific (`<fmt>.*`) | X | X | X | X | X | X | X | X | X |
+
+(Always check at runtime via `SupportsAttribute(K)` rather than
+relying on this table — it can drift.  The capability list each
+backend ships in `ClassSupportedAttributes` is authoritative.)