fpc-msgbase/docs/API.md

# fpc-msgbase — API reference

Every callable the library exposes, with parameters, return values,
and runnable examples.  See `docs/architecture.md` for the big
picture and `docs/ftsc-compliance.md` for spec notes.

## Contents

- [Quick start](#quick-start)
- [Opening a base](#opening-a-base)
- [TUniMessage](#tunimessage)
- [Reading](#reading)
- [Writing](#writing)
- [Updating & deleting](#updating--deleting)
- [Packing & reindexing](#packing--reindexing)
- [Events & logging](#events--logging)
- [Locking](#locking)
- [Path helpers](#path-helpers)
- [Concurrent tossers — moved](#concurrent-tossers--moved)
- [Dropping to a native backend](#dropping-to-a-native-backend)
- [Format cheat-sheet](#format-cheat-sheet)

---

## Quick start

```pascal
program hello;
{$mode objfpc}{$H+}
uses
  SysUtils,
  ma.types, ma.events, ma.api,
  ma.fmt.jam, ma.fmt.jam.uni;

var
  base: TMessageBase;
  msg:  TUniMessage;
  i:    longint;
begin
  base := MessageBaseOpen(mbfJam,
                          '/home/ken/fidonet/msg/jam/10thamd',
                          momReadOnly);
  try
    if not base.Open then Halt(1);
    for i := 0 to base.MessageCount - 1 do
      if base.ReadMessage(i, msg) then
        WriteLn(msg.Attributes.Get('from'),
                ' -> ', msg.Attributes.Get('to'),
                ': ',   msg.Attributes.Get('subject'));
  finally
    base.Close;
    base.Free;
  end;
end.
```

**Add format support** by including the matching pair of units:

| Format    | Units to add to `uses`                      |
|-----------|---------------------------------------------|
| Hudson    | `ma.fmt.hudson, ma.fmt.hudson.uni`          |
| JAM       | `ma.fmt.jam, ma.fmt.jam.uni`                |
| Squish    | `ma.fmt.squish, ma.fmt.squish.uni`          |
| FTS-1 MSG | `ma.fmt.msg, ma.fmt.msg.uni`                |
| FTN PKT   | (moved to `fpc-ftn-transport`; `uses tt.pkt.reader, tt.pkt.writer`) |
| PCBoard   | `ma.fmt.pcboard, ma.fmt.pcboard.uni`        |
| EzyCom    | `ma.fmt.ezycom, ma.fmt.ezycom.uni`          |
| GoldBase  | `ma.fmt.goldbase, ma.fmt.goldbase.uni`      |
| Wildcat   | `ma.fmt.wildcat, ma.fmt.wildcat.uni`        |

The `.uni` unit's `initialization` section registers the factory;
without it, `MessageBaseOpen(<format>, ...)` raises `EMessageBase`.

Threads on Unix need `cthreads` first in the program's `uses`:

```pascal
uses
  {$IFDEF UNIX}cthreads,{$ENDIF}
  ma.api, ...;
```

---

## Opening a base

### `MessageBaseOpen`

```pascal
function MessageBaseOpen(AFormat: TMsgBaseFormat;
                         const AAreaPath: AnsiString;
                         AMode: TMsgOpenMode;
                         const AAreaTag: AnsiString = ''): TMessageBase;
```

Returns a **not-yet-opened** `TMessageBase` for the requested format.
Call `.Open` before reading/writing.  Raises `EMessageBase` if the
format has no registered backend.

| Parameter  | Meaning                                                     |
|------------|-------------------------------------------------------------|
| AFormat    | `mbfHudson` / `mbfJam` / `mbfSquish` / etc.                 |
| AAreaPath  | Directory for dir-based formats, basename for JAM/Squish/PCB, full filename for PKT (PKT lives in fpc-ftn-transport but the `mbfPkt` factory enum is here). |
| AMode      | `momReadOnly`, `momReadWrite`, or `momCreate`.              |
| AAreaTag   | Optional area tag; passed through to the adapter.           |

### `MessageBaseOpenAuto`

```pascal
function MessageBaseOpenAuto(const AAreaPath: AnsiString;
                             AMode: TMsgOpenMode): TMessageBase;
```

Sniffs the directory for signature files (`MSGINFO.BBS`, `*.JHR`,
`*.SQD`, numbered `*.MSG`, …) and picks the right backend.  Returns
nil when no format matches.

### `DetectFormat`

```pascal
function DetectFormat(const AAreaPath: AnsiString;
                      out AFormat: TMsgBaseFormat): boolean;
```

Same sniffer as `MessageBaseOpenAuto`, but just returns the format
without opening.

### `TMessageBase.Open` / `Close`

```pascal
function  Open: boolean;
procedure Close;
```

`Open` acquires the cross-process sentinel lock (exclusive for
writers, shared for readers), then delegates to the backend's
native open.  Fires `metBaseOpened`.  Returns False if the base
files are missing or another writer is holding the lock.

`Close` delegates to the backend, releases the sentinel lock,
unlinks the `.lck` file, and fires `metBaseClosed`.

### Mode semantics

| Mode          | Behaviour                                                          |
|---------------|--------------------------------------------------------------------|
| `momReadOnly` | Read-only.  No write/pack allowed.  Shared lock, failures tolerated (e.g. RO FS). |
| `momReadWrite`| Files must exist; exclusive lock.                                  |
| `momCreate`   | Creates empty format files if missing, then opens exclusive.       |

---

## TUniMessage — two-area model

Single format-agnostic record.  **Body holds only the message
text; everything else lives in `Attributes`** as namespaced
key/value pairs.  Backends convert between their native record
and this on each read/write.

```pascal
TUniMessage = record
  Body:       AnsiString;       { only the message text }
  Attributes: TMsgAttributes;   { everything else, key/value }
end;
```

The full attribute key catalog with per-format support matrix
lives in [`docs/attributes-registry.md`](attributes-registry.md).

### TMsgAttributes accessors

```pascal
{ Setters }
procedure SetValue(const K, V: AnsiString);
procedure SetInt(const K: AnsiString; V: longint);
procedure SetInt64(const K: AnsiString; V: int64);
procedure SetBool(const K: AnsiString; V: boolean);
procedure SetDate(const K: AnsiString; V: TDateTime);
procedure SetAddr(const K: AnsiString; const V: TFTNAddress);
procedure SetList(const K: AnsiString; const V: TStringDynArray);
procedure AppendListItem(const K, Item: AnsiString);

{ Getters }
function  Get(const K: AnsiString;
              const Def: AnsiString = ''): AnsiString;
function  GetInt   (const K: AnsiString; Def: longint   = 0): longint;
function  GetInt64 (const K: AnsiString; Def: int64     = 0): int64;
function  GetBool  (const K: AnsiString; Def: boolean   = false): boolean;
function  GetDate  (const K: AnsiString; Def: TDateTime = 0): TDateTime;
function  GetAddr  (const K: AnsiString): TFTNAddress;
function  GetList  (const K: AnsiString): TStringDynArray;

{ Inspection }
function  Has(const K: AnsiString): boolean;
procedure Remove(const K: AnsiString);
procedure Clear;
function  Count: longint;
function  KeyAt(I: longint): AnsiString;
function  ValueAt(I: longint): AnsiString;
```

Common keys — see [`docs/attributes-registry.md`](attributes-registry.md):

| Key | Type | Meaning |
|---|---|---|
| `msg.num` | int | backend-assigned message number |
| `from` / `to` / `subject` | string | universal headers |
| `date.written` / `date.received` | date | timestamps |
| `addr.orig` / `addr.dest` | ftn | FTN addresses |
| `area` | string | echo area tag (auto-pop from base.AreaTag) |
| `board` | int | multi-board format conference number |
| `attr.*` | bool | private/crash/sent/read/etc. (see registry) |
| `msgid` / `replyid` / `pid` / `flags` | string | FTSC kludges |
| `seen-by` / `path` / `via` | multi | FTS-4 routing kludges (use GetList) |
| `intl` / `fmpt` / `topt` | string | FSC-4008 cross-zone routing |
| `kludge.<name>` | string | unknown FTSC kludge passthrough |

### Capabilities API

Each backend declares the canonical list of attribute keys it
understands.  Callers query before setting:

```pascal
function  TMessageBase.SupportsAttribute(const Key: AnsiString): boolean;
function  TMessageBase.SupportedAttributes: TStringDynArray;

if base.SupportsAttribute('attr.returnreceipt') then
  RenderReceiptCheckbox;
```

### Canonical attribute bits (`ma.types`)

The MSG_ATTR_* cardinal constants stay as the internal pivot
between native flag words and the individual `attr.*` boolean
attributes:

```
MSG_ATTR_PRIVATE        $00000001    MSG_ATTR_DELETED       $00010000
MSG_ATTR_CRASH          $00000002    MSG_ATTR_READ          $00020000
MSG_ATTR_RECEIVED       $00000004    MSG_ATTR_ECHO          $00040000
MSG_ATTR_SENT           $00000008    MSG_ATTR_DIRECT        $00080000
MSG_ATTR_FILE_ATTACHED  $00000010    MSG_ATTR_IMMEDIATE     $00100000
MSG_ATTR_IN_TRANSIT     $00000020    MSG_ATTR_FREQ_PENDING  $00200000
MSG_ATTR_ORPHAN         $00000040    MSG_ATTR_SEEN          $00400000
MSG_ATTR_KILL_SENT      $00000080    MSG_ATTR_LOCKED        $00800000
MSG_ATTR_LOCAL          $00000100    MSG_ATTR_NO_KILL       $01000000
MSG_ATTR_HOLD           $00000200    MSG_ATTR_NET_PENDING   $02000000
MSG_ATTR_FILE_REQUEST   $00000400    MSG_ATTR_ECHO_PENDING  $04000000
MSG_ATTR_RETURN_RCPT    $00000800
MSG_ATTR_IS_RECEIPT     $00001000
MSG_ATTR_AUDIT_REQ      $00002000
MSG_ATTR_FILE_UPD_REQ   $00004000
```

Bits 0..15 match FTS-0001 exactly; 16+ are for storage-layer flags
(deleted/read/echo/etc.) that aren't part of the FTN wire format.

Use `UniAttrBitsToAttributes` / `UniAttrBitsFromAttributes`
helpers in `ma.types` to bridge the bitset to/from individual
`attr.*` boolean attributes.

### FTN addressing

```pascal
TFTNAddress = record Zone, Net, Node, Point: word; end;

function MakeFTNAddress(AZone, ANet, ANode, APoint: word): TFTNAddress;
function FTNAddressToString(const A: TFTNAddress): string;       { "1:100/200.3" }
function ParseFTNAddress(const S: string; out A: TFTNAddress): boolean;
function FTNAddressEqual(const A, B: TFTNAddress): boolean;
```

---

## Reading

```pascal
function ReadMessage(Index: longint; var Msg: TUniMessage): boolean;
property MessageCount: longint;
property AreaTag: AnsiString;            { auto-pop msg.area on Read }
property ActiveUser: AnsiString;         { auto-bump HWM on Read }
```

Zero-based index.  Returns False on EOF or backend failure.
Message numbers come back in `msg.Attributes.GetInt('msg.num')`
and are backend-assigned (typically **don't** match the index;
most formats keep a gap-tolerant index).  Fires `metMessageRead`
on success.

If `AreaTag` is set (either via `MessageBaseOpen`'s `AAreaTag`
parameter or the property setter post-construction), every
successful Read auto-populates `msg.Attributes['area']` with
the tag, unless the adapter already populated it from on-disk
data (e.g. PKT's AREA kludge).

If `ActiveUser` is set and the backend supports HWM, every
successful Read advances the per-user HWM if `msg.num >`
current HWM (never decrements).  See *HWM* below.

```pascal
base.ActiveUser := 'NetReader';   { optional: HWM auto-bump }
for i := 0 to base.MessageCount - 1 do
  if base.ReadMessage(i, msg) then
    Handle(msg);
```

---

## Writing

```pascal
function WriteMessage(var Msg: TUniMessage): boolean;
```

Appends a new message.  Backend assigns `msg.Attributes['msg.num']`
on success.  Fires `metMessageWritten`.  Raises `EMessageBase` in
read-only mode.  PKT (via `tt.pkt.reader` from fpc-ftn-transport)
raises `EMessageBase` on Write — use `tt.pkt.writer.TPktWriter`
for outbound packets.

```pascal
msg.Attributes.Clear;
msg.Attributes.SetValue('from',    'Sysop');
msg.Attributes.SetValue('to',      'All');
msg.Attributes.SetValue('subject', 'Hello');
msg.Attributes.SetDate ('date.written', Now);
msg.Attributes.SetBool ('attr.local', true);
msg.Attributes.SetBool ('attr.echo',  true);
msg.Attributes.SetAddr ('addr.orig', MakeFTNAddress(1, 100, 1, 0));
msg.Attributes.SetAddr ('addr.dest', MakeFTNAddress(1, 100, 2, 0));
msg.Body := 'Hello, world';
base.WriteMessage(msg);
```

### High-Water Mark (HWM)

Per-user "last message I scanned" pointer.  Native for JAM,
Squish, Hudson, GoldBase; -1 (unsupported) for the others.

```pascal
function  SupportsHWM: boolean;
function  GetHWM(const UserName: AnsiString): longint;
procedure SetHWM(const UserName: AnsiString; MsgNum: longint);
procedure MapUser(const UserName: AnsiString; UserId: longint);
property  ActiveUser: AnsiString;
property  Board: longint;          { multi-board context }
```

Tossers / scanners register as named users (e.g. `'NetReader'`,
`'Allfix'`, `'FidoMail-Toss'`); each gets its own slot in the
format's native lastread file, so multiple consumers coexist.

Number-keyed formats (Hudson, GoldBase, EzyCom) need
`MapUser('NetReader', 60001)` (pick `60000+` to avoid colliding
with real BBS users) and `Board := <n>` before HWM ops; otherwise
return -1.

See [`docs/architecture.md`](architecture.md) HWM section for
the full coverage map and rationale.

---

## Updating & deleting

```pascal
function UpdateMessage(Index: longint; var Msg: TUniMessage): boolean;
function DeleteMessage(Index: longint): boolean;
```

Optional operations.  Not every backend implements them — the
default returns False.  Attribute-only changes can also be made
through the native API (`.Native`) by flipping the MSG_ATTR_DELETED
bit and calling `Pack`; that pattern works on every backend.

---

## Packing & reindexing

```pascal
function Pack(PurgeAgeDays, PurgeMaxCount: longint;
              Backup: boolean = True): boolean;
function ReIndex: boolean;
```

`Pack` purges deleted/old messages and rebuilds the physical files.
`PurgeAgeDays = 0` and `PurgeMaxCount = 0` means "don't purge by
age or count; just compact deleted messages."  Fires
`metPackStarted` / `metPackComplete`.

`ReIndex` rebuilds the index file from the header file (JAM/Hudson).

```pascal
{ no-op pack: only drop anything already marked deleted }
base.Pack(0, 0, False);

{ purge anything older than 90 days, keep at most 500 messages }
base.Pack(90, 500, True);
```

---

## Events & logging

Every `TMessageBase` owns a `TMessageEvents` reachable as `.Events`.
Subscribe with either a log-shaped handler or a full-info handler,
or both.  Multiple hooks can be registered with `AddHook`.

```pascal
TMessageLogHandler =
  procedure(Level: TMsgEventType;
            const Source, Msg: AnsiString) of object;

TMessageEventHandler =
  procedure(Sender: TObject;
            const Info: TMessageEventInfo) of object;
```

### Example — console logger

```pascal
type
  TLogger = class
    procedure OnLog(Level: TMsgEventType;
                    const Source, Msg: AnsiString);
  end;

procedure TLogger.OnLog(Level: TMsgEventType;
                        const Source, Msg: AnsiString);
begin
  WriteLn('[', EventTypeToStr(Level), '] ', Source, ': ', Msg);
end;

var
  log: TLogger;
begin
  log := TLogger.Create;
  base.Events.OnLog := @log.OnLog;
  ...
```

### Event types (`TMsgEventType`)

| Group      | Values                                                              |
|------------|---------------------------------------------------------------------|
| Log levels | `metInfo`, `metWarning`, `metError`, `metDebug`                     |
| Lifecycle  | `metBaseOpened`, `metBaseClosed`, `metBaseCreated`                  |
| Messages   | `metMessageRead`, `metMessageWritten`, `metMessageDeleted`, `metMessageUpdated` |
| Locking    | `metLockAcquired`, `metLockReleased`, `metLockTimeout`, `metLockFailed` |
| Pack       | `metPackStarted`, `metPackProgress`, `metPackComplete`, `metReindexStarted`, `metReindexComplete` |
| Packets    | `metPacketStart`, `metPacketEnd`, `metPacketMessage`, `metPacketError` |

Filter by severity:

```pascal
base.Events.MinLevel := metWarning;  { drop info + debug }
```

---

## Locking

Three layers, all in `ma.lock`:

1. **TRTLCriticalSection** per `TMessageBase` instance — serialises
   concurrent Read/Write/Update/Delete calls on the same instance.
2. **Advisory sentinel** — `fpflock` (Unix), `LockFileEx`
   (Windows), `DosSetFileLocks` (OS/2), exclusive-open (DOS) on a
   per-base `.lck` file.  Released and unlinked on close.
3. **Native share modes** — each backend opens its data streams
   with `fmShareDenyWrite` / `fmShareDenyNone` as the backstop.

The sentinel timeout is configurable per instance:

```pascal
base.LockTimeoutMs := 5000;   { retry for 5 s, fail otherwise }
base.LockTimeoutMs := 0;      { fail fast — don't wait at all }
base.LockTimeoutMs := -1;     { wait forever }
```

Direct access: `base.Lock` returns the `TMessageLock` for the rare
case where you want to hold the lock across multiple operations.

---

## Path helpers

```pascal
function MessageBasePathFor(AFormat: TMsgBaseFormat;
                            const AAreaPath, AAreaTag: AnsiString): AnsiString;
function FindExistingFile(const APath: AnsiString): AnsiString;
function PathJoin(const ADir, ATail: AnsiString): AnsiString;
function LockFilePath(AFormat: TMsgBaseFormat;
                      const ABasePath, AAreaTag: AnsiString): AnsiString;
```

`MessageBasePathFor` produces the canonical constructor argument
each backend expects from an area directory + optional tag.
`FindExistingFile` does case-insensitive resolution (path → UPPER
→ lower) for Linux hosts where on-disk names may be mixed case.

---

## Concurrent tossers — moved

`TPacketBatch` (was `ma.batch` here pre-0.4.0) moved to
`fpc-ftn-transport` as `tt.pkt.batch` along with the rest of
the PKT code.  Class name and API surface unchanged for caller
compatibility.

Use it from there:

```pascal
uses
  {$IFDEF UNIX}cthreads,{$ENDIF}
  ma.api,                        { TMessageBase + factory  }
  ma.fmt.jam, ma.fmt.jam.uni,    { destination msgbase }
  tt.pkt.format,                 { wire-format types       }
  tt.pkt.reader,                 { registers mbfPkt        }
  tt.pkt.batch;                  { TPacketBatch + Run loop }
```

See `fpc-ftn-transport/docs/` for the worker-pool details.

---

## Dropping to a native backend

Every `.uni` adapter exposes `Native` for callers who need format-
specific features the unified API doesn't abstract (JAM subfields,
Squish UMsgId, EzyCom dual-byte attributes, Wildcat NextMsg
walking, etc.):

```pascal
var
  adapter: TJamMessageBase;
  hdr:     JamHdr;
  nat:     TJamMessage;
begin
  adapter := base as TJamMessageBase;
  adapter.Native.ReadMessage(Index, nat);
  adapter.Native.ReadHeader(nat.HdrOffset, hdr);
  hdr.Attribute := hdr.Attribute or longint($80000000);   { mark deleted }
  adapter.Native.UpdateHeader(nat.HdrOffset, hdr);
  adapter.Native.IncModCounter;
  adapter.Native.UpdateHdrInfo;
end;
```

Native class names:

| Format    | Native class      | Adapter              |
|-----------|-------------------|----------------------|
| Hudson    | `THudsonBase`     | `THudsonMessageBase` |
| JAM       | `TJamBase`        | `TJamMessageBase`    |
| Squish    | `TSquishBase`     | `TSquishMessageBase` |
| FTS-1 MSG | `TMsgDir`+`TMsgFile` | `TMsgMessageBase` |
| FTN PKT   | `TPktFile` *(in fpc-ftn-transport: tt.pkt.format)* | `TPktReader` *(tt.pkt.reader)* |
| PCBoard   | `TPCBoardBase`    | `TPCBoardMessageBase`|
| EzyCom    | `TEzyComBase`     | `TEzyComMessageBase` |
| GoldBase  | `TGoldBase`       | `TGoldBaseMessageBase`|
| Wildcat   | `TWildcatBase`    | `TWildcatMessageBase`|

---

## Format cheat-sheet

| Format   | Base path shape          | Caveats                                    |
|----------|--------------------------|--------------------------------------------|
| Hudson   | directory                | 32767-message ceiling (smallint MsgNum)    |
| GoldBase | directory                | Widened Hudson (word MsgAttr, 500 boards)  |
| JAM      | dir + basename, no ext   | Adapter appends `.JHR/.JDT/.JDX/.JLR`      |
| Squish   | dir + basename, no ext   | Adapter appends `.SQD/.SQI/.SQL`           |
| FTS-1 MSG| directory                | Numbered `*.msg` / `*.MSG` (case mixed)    |
| FTN PKT  | full packet filename     | Lives in `fpc-ftn-transport`. `uses tt.pkt.reader` registers `mbfPkt` with the unified factory; iterate with `MessageBaseOpen(mbfPkt, ...)`. Outbound writes via `tt.pkt.writer.TPktWriter` (atomic `.tmp→rename`, file/stream targets, variant selection). |
| PCBoard  | dir + basename, no ext   | Adapter appends `.MSG/.IDX`                |
| EzyCom   | directory + `.Board`     | Set `adapter.Board`/`.BBSType` before Open |
| Wildcat  | WC data dir + `.Conference` | Set `adapter.Conference` before Open    |

---

## See also

- `docs/architecture.md` — layered design
- `docs/ftsc-compliance.md` — spec refs
- `docs/format-notes/` — per-format quirks and dependencies
- `docs/attributes-registry.md` — full attribute key catalog +
  per-format support matrix
- `examples/` — runnable `example_read`, `example_write`,
  `example_tosser`
- `tests/` — test_read, test_roundtrip, test_roundtrip_attrs,
  test_lock, test_batch, test_wildcat, test_write_existing,
  test_pack, test_hwm, test_consumer_round1
- `ma.kludge` — shared FTSC kludge parsing/emission helpers
  (`ParseKludgeLine`, `SplitKludgeBlob`, `BuildKludgePrefix`,
  `BuildKludgeSuffix`) for callers that need to handle raw FTSC
  body blobs outside an adapter