kenjreno/fpc-msgbase
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

0.3.5: ma.kludge shared helper, INTL/FMPT/TOPT, area auto-pop, list helpers, PKT polish

Browse Source 
Five consumer-feedback items, one milestone:

(1) Shared FTSC kludge plumbing in src/ma.kludge.pas

  ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix,
  BuildKludgeSuffix.  Single source of truth for kludge naming,
  INTL/FMPT/TOPT recognition, and the kludge.<lowername>
  forward-compat passthrough.  Eliminates the four near-identical
  parsers MSG/PKT/Squish were carrying; JAM's FTSKLUDGE subfield
  walking also routes through ParseKludgeLine so its unknown
  kludges land in the same `kludge.<name>` slot as the others.

  Bug fix folded in: the parser previously split kludge name from
  value at the first ':' it found, which broke INTL (the value
  contains an FTN address with ':' in it).  Now picks the earlier
  of space and colon, which handles both colon-form ("MSGID: foo")
  and space-form ("INTL <to> <from>") kludges correctly.

(2) INTL / FMPT / TOPT slots in attributes registry

  FSC-4008 cross-zone routing kludges every netmail tosser carries.
  Added to JAM/Squish/MSG/PKT capability lists, parsed natively,
  emitted on Write.  Round-trip covered by tests.

(3) Unified `kludge.*` namespace for unknown FTSC kludges

  Squish's `squish.kludge.<name>`, MSG's `msg.kludge.<name>`, and
  PKT's `pkt.kludge.<name>` all collapse to plain `kludge.<name>`.
  Consumers find passthrough kludges without switching on format.
  JAM's numeric `jam.subfield.<id>` stays — those are JAM-specific
  binary subfields, not FTSC-form kludges.

(4) `area` auto-populated from base.AreaTag on Read

  When the caller passes AAreaTag to MessageBaseOpen (or sets
  the AreaTag property post-construction), every successful
  ReadMessage fills msg.Attributes['area'] unless the adapter
  already populated it from on-disk data (e.g. PKT AREA kludge).
  Saves echomail consumers from copying AreaTag into every
  message attribute manually.

(5) TMsgAttributes multi-line helpers

  GetList / SetList / AppendListItem on TMsgAttributes for the
  multi-instance attributes (seen-by, path, via, trace) that
  store with #13 between entries.  Consumers don't have to roll
  their own split/join.

Plus two PKT polish items from the same feedback round:

(6) ma.fmt.pkt.uni.DoWriteMessage now raises EMessageBase
    explicitly with a pointer to the Native API instead of
    silently returning False.

(7) TPktFile.CreateFromStream / CreateNewToStream constructors
    accept any TStream (with optional ownership), so unit tests
    that round-trip via TMemoryStream don't have to tempfile-dance.
    FStream is now TStream; FOwnsStream gates Free in destructor.

TStringDynArray moved from ma.api.pas to ma.types.pas so both
the capabilities API and the new attribute helpers can share it.

Docs sweep:

- docs/attributes-registry.md: intl/fmpt/topt added; unknown-kludge
  convention documented; multi-line helper section added.
- docs/architecture.md: ma.kludge layer surfaced; .uni adapter
  registration gotcha called out loudly with the recommended
  uses clause; area auto-pop documented.
- docs/API.md: TUniMessage section rewritten for Body+Attributes
  model (was still pre-0.2); HWM API documented; PKT cheat-sheet
  notes Native + CreateFromStream; tests/programs list updated.
- README.md: Building section flags the .uni gotcha first
  thing; ma.kludge added to features.

tests/test_consumer_round1.pas: 7 new tests covering INTL/FMPT/
TOPT round-trip on JAM/Squish/MSG, area auto-pop, GetList/SetList/
AppendListItem, PKT raise, and TPktFile in-memory stream
round-trip.

Suite: 47/47 across 10 programs (test_consumer_round1 adds 7).
This commit is contained in:
Ken Johnson
2026-04-18 09:14:33 -07:00
parent 13ff9bf88a
commit e876d98b83
14 changed files with 1113 additions and 470 deletions
Download Patch File Download Diff File Expand all files Collapse all files

185
docs/API.md
View File

@@ -44,7 +44,9 @@ begin
if not base.Open then Halt(1);
for i := 0 to base.MessageCount - 1 do
if base.ReadMessage(i, msg) then
WriteLn(msg.WhoFrom, ' -> ', msg.WhoTo, ': ', msg.Subject);
WriteLn(msg.Attributes.Get('from'),
' -> ', msg.Attributes.Get('to'),
': ', msg.Attributes.Get('subject'));
finally
base.Close;
base.Free;
@@ -147,31 +149,90 @@ unlinks the `.lck` file, and fires `metBaseClosed`.
---
## TUniMessage
## TUniMessage — two-area model
Single format-agnostic message record. Backends convert between
their native record and this on each read/write.
Single format-agnostic record. **Body holds only the message
text; everything else lives in `Attributes`** as namespaced
key/value pairs. Backends convert between their native record
and this on each read/write.
```pascal
TUniMessage = record
MsgNum: longint; { backend-assigned number }
WhoFrom: AnsiString;
WhoTo: AnsiString;
Subject: AnsiString;
DateWritten: TDateTime;
DateReceived: TDateTime; { 0 if not received }
Attr: cardinal; { MSG_ATTR_* bitset, see below }
OrigAddr: TFTNAddress;
DestAddr: TFTNAddress;
Cost: word;
Body: AnsiString; { CR-separated, kludges intact }
AreaTag: AnsiString; { optional echo-area tag }
Board: word; { conference/board; 0 = default }
Body: AnsiString; { only the message text }
Attributes: TMsgAttributes; { everything else, key/value }
end;
```
The full attribute key catalog with per-format support matrix
lives in [`docs/attributes-registry.md`](attributes-registry.md).
### TMsgAttributes accessors
```pascal
{ Setters }
procedure SetValue(const K, V: AnsiString);
procedure SetInt(const K: AnsiString; V: longint);
procedure SetInt64(const K: AnsiString; V: int64);
procedure SetBool(const K: AnsiString; V: boolean);
procedure SetDate(const K: AnsiString; V: TDateTime);
procedure SetAddr(const K: AnsiString; const V: TFTNAddress);
procedure SetList(const K: AnsiString; const V: TStringDynArray);
procedure AppendListItem(const K, Item: AnsiString);
{ Getters }
function Get(const K: AnsiString;
const Def: AnsiString = ''): AnsiString;
function GetInt (const K: AnsiString; Def: longint = 0): longint;
function GetInt64 (const K: AnsiString; Def: int64 = 0): int64;
function GetBool (const K: AnsiString; Def: boolean = false): boolean;
function GetDate (const K: AnsiString; Def: TDateTime = 0): TDateTime;
function GetAddr (const K: AnsiString): TFTNAddress;
function GetList (const K: AnsiString): TStringDynArray;
{ Inspection }
function Has(const K: AnsiString): boolean;
procedure Remove(const K: AnsiString);
procedure Clear;
function Count: longint;
function KeyAt(I: longint): AnsiString;
function ValueAt(I: longint): AnsiString;
```
Common keys — see [`docs/attributes-registry.md`](attributes-registry.md):
| Key | Type | Meaning |
|---|---|---|
| `msg.num` | int | backend-assigned message number |
| `from` / `to` / `subject` | string | universal headers |
| `date.written` / `date.received` | date | timestamps |
| `addr.orig` / `addr.dest` | ftn | FTN addresses |
| `area` | string | echo area tag (auto-pop from base.AreaTag) |
| `board` | int | multi-board format conference number |
| `attr.*` | bool | private/crash/sent/read/etc. (see registry) |
| `msgid` / `replyid` / `pid` / `flags` | string | FTSC kludges |
| `seen-by` / `path` / `via` | multi | FTS-4 routing kludges (use GetList) |
| `intl` / `fmpt` / `topt` | string | FSC-4008 cross-zone routing |
| `kludge.<name>` | string | unknown FTSC kludge passthrough |
### Capabilities API
Each backend declares the canonical list of attribute keys it
understands. Callers query before setting:
```pascal
function TMessageBase.SupportsAttribute(const Key: AnsiString): boolean;
function TMessageBase.SupportedAttributes: TStringDynArray;
if base.SupportsAttribute('attr.returnreceipt') then
RenderReceiptCheckbox;
```
### Canonical attribute bits (`ma.types`)
The MSG_ATTR_* cardinal constants stay as the internal pivot
between native flag words and the individual `attr.*` boolean
attributes:
```
MSG_ATTR_PRIVATE $00000001 MSG_ATTR_DELETED $00010000
MSG_ATTR_CRASH $00000002 MSG_ATTR_READ $00020000
@@ -193,6 +254,10 @@ MSG_ATTR_FILE_UPD_REQ $00004000
Bits 0..15 match FTS-0001 exactly; 16+ are for storage-layer flags
(deleted/read/echo/etc.) that aren't part of the FTN wire format.
Use `UniAttrBitsToAttributes` / `UniAttrBitsFromAttributes`
helpers in `ma.types` to bridge the bitset to/from individual
`attr.*` boolean attributes.
### FTN addressing
```pascal
@@ -211,14 +276,28 @@ function FTNAddressEqual(const A, B: TFTNAddress): boolean;
```pascal
function ReadMessage(Index: longint; var Msg: TUniMessage): boolean;
property MessageCount: longint;
property AreaTag: AnsiString; { auto-pop msg.area on Read }
property ActiveUser: AnsiString; { auto-bump HWM on Read }
```
Zero-based index. Returns False on EOF or backend failure.
Message numbers (`Msg.MsgNum`) are backend-assigned and typically
**don't** match the index (most formats keep a gap-tolerant index).
Fires `metMessageRead` on success.
Message numbers come back in `msg.Attributes.GetInt('msg.num')`
and are backend-assigned (typically **don't** match the index;
most formats keep a gap-tolerant index). Fires `metMessageRead`
on success.
If `AreaTag` is set (either via `MessageBaseOpen`'s `AAreaTag`
parameter or the property setter post-construction), every
successful Read auto-populates `msg.Attributes['area']` with
the tag, unless the adapter already populated it from on-disk
data (e.g. PKT's AREA kludge).
If `ActiveUser` is set and the backend supports HWM, every
successful Read advances the per-user HWM if `msg.num >`
current HWM (never decrements). See *HWM* below.
```pascal
base.ActiveUser := 'NetReader'; { optional: HWM auto-bump }
for i := 0 to base.MessageCount - 1 do
if base.ReadMessage(i, msg) then
Handle(msg);
@@ -232,21 +311,51 @@ for i := 0 to base.MessageCount - 1 do
function WriteMessage(var Msg: TUniMessage): boolean;
```
Appends a new message. Backend assigns `Msg.MsgNum` on success.
Fires `metMessageWritten`. Raises `EMessageBase` in read-only mode.
Appends a new message. Backend assigns `msg.Attributes['msg.num']`
on success. Fires `metMessageWritten`. Raises `EMessageBase` in
read-only mode. PKT raises `EMessageBase` regardless of mode —
write packets via `Native: TPktFile` directly.
```pascal
msg.WhoFrom := 'Sysop';
msg.WhoTo := 'All';
msg.Subject := 'Hello';
msg.DateWritten := Now;
msg.Attr := MSG_ATTR_LOCAL or MSG_ATTR_ECHO;
msg.OrigAddr := MakeFTNAddress(1, 100, 1, 0);
msg.DestAddr := MakeFTNAddress(1, 100, 2, 0);
msg.Body := 'Hello, world' + #13;
msg.Attributes.Clear;
msg.Attributes.SetValue('from', 'Sysop');
msg.Attributes.SetValue('to', 'All');
msg.Attributes.SetValue('subject', 'Hello');
msg.Attributes.SetDate ('date.written', Now);
msg.Attributes.SetBool ('attr.local', true);
msg.Attributes.SetBool ('attr.echo', true);
msg.Attributes.SetAddr ('addr.orig', MakeFTNAddress(1, 100, 1, 0));
msg.Attributes.SetAddr ('addr.dest', MakeFTNAddress(1, 100, 2, 0));
msg.Body := 'Hello, world';
base.WriteMessage(msg);
```
### High-Water Mark (HWM)
Per-user "last message I scanned" pointer. Native for JAM,
Squish, Hudson, GoldBase; -1 (unsupported) for the others.
```pascal
function SupportsHWM: boolean;
function GetHWM(const UserName: AnsiString): longint;
procedure SetHWM(const UserName: AnsiString; MsgNum: longint);
procedure MapUser(const UserName: AnsiString; UserId: longint);
property ActiveUser: AnsiString;
property Board: longint; { multi-board context }
```
Tossers / scanners register as named users (e.g. `'NetReader'`,
`'Allfix'`, `'FidoMail-Toss'`); each gets its own slot in the
format's native lastread file, so multiple consumers coexist.
Number-keyed formats (Hudson, GoldBase, EzyCom) need
`MapUser('NetReader', 60001)` (pick `60000+` to avoid colliding
with real BBS users) and `Board := <n>` before HWM ops; otherwise
return -1.
See [`docs/architecture.md`](architecture.md) HWM section for
the full coverage map and rationale.
---
## Updating & deleting
@@ -437,7 +546,8 @@ procedure TTosser.OnMsg(const Path: AnsiString;
var
base: TMessageBase;
begin
base := Batch.GetOrCreateBase(mbfJam, '/msg/echo/' + Msg.AreaTag);
base := Batch.GetOrCreateBase(mbfJam,
'/msg/echo/' + Msg.Attributes.Get('area'));
if base <> nil then
base.WriteMessage(Msg);
end;
@@ -503,7 +613,7 @@ Native class names:
| JAM | dir + basename, no ext | Adapter appends `.JHR/.JDT/.JDX/.JLR` |
| Squish | dir + basename, no ext | Adapter appends `.SQD/.SQI/.SQL` |
| FTS-1 MSG| directory | Numbered `*.msg` / `*.MSG` (case mixed) |
| FTN PKT | full packet filename | Stream-only; `WriteMessage` not supported through adapter — use `Native` |
| FTN PKT | full packet filename | Stream-only; `WriteMessage` raises `EMessageBase` — use `Native` (`TPktFile`). For in-memory tests, `TPktFile.CreateFromStream` / `CreateNewToStream` accept any `TStream`. |
| PCBoard | dir + basename, no ext | Adapter appends `.MSG/.IDX` |
| EzyCom | directory + `.Board` | Set `adapter.Board`/`.BBSType` before Open |
| Wildcat | WC data dir + `.Conference` | Set `adapter.Conference` before Open |
@@ -515,7 +625,14 @@ Native class names:
- `docs/architecture.md` — layered design
- `docs/ftsc-compliance.md` — spec refs
- `docs/format-notes/` — per-format quirks and dependencies
- `docs/attributes-registry.md` — full attribute key catalog +
per-format support matrix
- `examples/` — runnable `example_read`, `example_write`,
`example_tosser`
- `tests/` — test_read, test_roundtrip, test_lock, test_batch,
test_wildcat, test_write_existing, test_pack
- `tests/` — test_read, test_roundtrip, test_roundtrip_attrs,
test_lock, test_batch, test_wildcat, test_write_existing,
test_pack, test_hwm, test_consumer_round1
- `ma.kludge` — shared FTSC kludge parsing/emission helpers
(`ParseKludgeLine`, `SplitKludgeBlob`, `BuildKludgePrefix`,
`BuildKludgeSuffix`) for callers that need to handle raw FTSC
body blobs outside an adapter