# fpc-msgbase — API reference Every callable the library exposes, with parameters, return values, and runnable examples. See `docs/architecture.md` for the big picture and `docs/ftsc-compliance.md` for spec notes. ## Contents - [Quick start](#quick-start) - [Opening a base](#opening-a-base) - [TUniMessage](#tunimessage) - [Reading](#reading) - [Writing](#writing) - [Updating & deleting](#updating--deleting) - [Packing & reindexing](#packing--reindexing) - [Events & logging](#events--logging) - [Locking](#locking) - [Path helpers](#path-helpers) - [Concurrent tossers — moved](#concurrent-tossers--moved) - [Dropping to a native backend](#dropping-to-a-native-backend) - [Format cheat-sheet](#format-cheat-sheet) --- ## Quick start ```pascal program hello; {$mode objfpc}{$H+} uses SysUtils, mb.types, mb.events, mb.api, mb.fmt.jam, mb.fmt.jam.uni; var base: TMessageBase; msg: TUniMessage; i: longint; begin base := MessageBaseOpen(mbfJam, '/home/ken/fidonet/msg/jam/10thamd', momReadOnly); try if not base.Open then Halt(1); for i := 0 to base.MessageCount - 1 do if base.ReadMessage(i, msg) then WriteLn(msg.Attributes.Get('from'), ' -> ', msg.Attributes.Get('to'), ': ', msg.Attributes.Get('subject')); finally base.Close; base.Free; end; end. ``` **Add format support** by including the matching pair of units: | Format | Units to add to `uses` | |-----------|---------------------------------------------| | Hudson | `mb.fmt.hudson, mb.fmt.hudson.uni` | | JAM | `mb.fmt.jam, mb.fmt.jam.uni` | | Squish | `mb.fmt.squish, mb.fmt.squish.uni` | | FTS-1 MSG | `mb.fmt.msg, mb.fmt.msg.uni` | | FTN PKT | (moved to `fpc-ftn-transport`; `uses tt.pkt.reader, tt.pkt.writer`) | | PCBoard | `mb.fmt.pcboard, mb.fmt.pcboard.uni` | | EzyCom | `mb.fmt.ezycom, mb.fmt.ezycom.uni` | | GoldBase | `mb.fmt.goldbase, mb.fmt.goldbase.uni` | | Wildcat | `mb.fmt.wildcat, mb.fmt.wildcat.uni` | The `.uni` unit's `initialization` section registers the factory; without it, `MessageBaseOpen(, ...)` raises `EMessageBase`. Threads on Unix need `cthreads` first in the program's `uses`: ```pascal uses {$IFDEF UNIX}cthreads,{$ENDIF} mb.api, ...; ``` --- ## Opening a base ### `MessageBaseOpen` ```pascal function MessageBaseOpen(AFormat: TMsgBaseFormat; const AAreaPath: AnsiString; AMode: TMsgOpenMode; const AAreaTag: AnsiString = ''): TMessageBase; ``` Returns a **not-yet-opened** `TMessageBase` for the requested format. Call `.Open` before reading/writing. Raises `EMessageBase` if the format has no registered backend. | Parameter | Meaning | |------------|-------------------------------------------------------------| | AFormat | `mbfHudson` / `mbfJam` / `mbfSquish` / etc. | | AAreaPath | Directory for dir-based formats, basename for JAM/Squish/PCB, full filename for PKT (PKT lives in fpc-ftn-transport but the `mbfPkt` factory enum is here). | | AMode | `momReadOnly`, `momReadWrite`, or `momCreate`. | | AAreaTag | Optional area tag; passed through to the adapter. | ### `MessageBaseOpenAuto` ```pascal function MessageBaseOpenAuto(const AAreaPath: AnsiString; AMode: TMsgOpenMode): TMessageBase; ``` Sniffs the directory for signature files (`MSGINFO.BBS`, `*.JHR`, `*.SQD`, numbered `*.MSG`, …) and picks the right backend. Returns nil when no format matches. ### `DetectFormat` ```pascal function DetectFormat(const AAreaPath: AnsiString; out AFormat: TMsgBaseFormat): boolean; ``` Same sniffer as `MessageBaseOpenAuto`, but just returns the format without opening. ### `TMessageBase.Open` / `Close` ```pascal function Open: boolean; procedure Close; ``` `Open` acquires the cross-process sentinel lock (exclusive for writers, shared for readers), then delegates to the backend's native open. Fires `metBaseOpened`. Returns False if the base files are missing or another writer is holding the lock. `Close` delegates to the backend, releases the sentinel lock, unlinks the `.lck` file, and fires `metBaseClosed`. ### Mode semantics | Mode | Behaviour | |---------------|--------------------------------------------------------------------| | `momReadOnly` | Read-only. No write/pack allowed. Shared lock, failures tolerated (e.g. RO FS). | | `momReadWrite`| Files must exist; exclusive lock. | | `momCreate` | Creates empty format files if missing, then opens exclusive. | --- ## TUniMessage — two-area model Single format-agnostic record. **Body holds only the message text; everything else lives in `Attributes`** as namespaced key/value pairs. Backends convert between their native record and this on each read/write. ```pascal TUniMessage = record Body: AnsiString; { only the message text } Attributes: TMsgAttributes; { everything else, key/value } end; ``` The full attribute key catalog with per-format support matrix lives in [`docs/attributes-registry.md`](attributes-registry.md). ### TMsgAttributes accessors ```pascal { Setters } procedure SetValue(const K, V: AnsiString); procedure SetInt(const K: AnsiString; V: longint); procedure SetInt64(const K: AnsiString; V: int64); procedure SetBool(const K: AnsiString; V: boolean); procedure SetDate(const K: AnsiString; V: TDateTime); procedure SetAddr(const K: AnsiString; const V: TFTNAddress); procedure SetList(const K: AnsiString; const V: TStringDynArray); procedure AppendListItem(const K, Item: AnsiString); { Getters } function Get(const K: AnsiString; const Def: AnsiString = ''): AnsiString; function GetInt (const K: AnsiString; Def: longint = 0): longint; function GetInt64 (const K: AnsiString; Def: int64 = 0): int64; function GetBool (const K: AnsiString; Def: boolean = false): boolean; function GetDate (const K: AnsiString; Def: TDateTime = 0): TDateTime; function GetAddr (const K: AnsiString): TFTNAddress; function GetList (const K: AnsiString): TStringDynArray; { Inspection } function Has(const K: AnsiString): boolean; procedure Remove(const K: AnsiString); procedure Clear; function Count: longint; function KeyAt(I: longint): AnsiString; function ValueAt(I: longint): AnsiString; ``` Common keys — see [`docs/attributes-registry.md`](attributes-registry.md): | Key | Type | Meaning | |---|---|---| | `msg.num` | int | backend-assigned message number | | `from` / `to` / `subject` | string | universal headers | | `date.written` / `date.received` | date | timestamps | | `addr.orig` / `addr.dest` | ftn | FTN addresses | | `area` | string | echo area tag (auto-pop from base.AreaTag) | | `board` | int | multi-board format conference number | | `attr.*` | bool | private/crash/sent/read/etc. (see registry) | | `msgid` / `replyid` / `pid` / `flags` | string | FTSC kludges | | `seen-by` / `path` / `via` | multi | FTS-4 routing kludges (use GetList) | | `intl` / `fmpt` / `topt` | string | FSC-4008 cross-zone routing | | `kludge.` | string | unknown FTSC kludge passthrough | ### Sync (durability) `TMessageBase.Sync` forces every open writer stream to durable storage (`fpfsync` on Unix, `FlushFileBuffers` on Windows). Override default no-op for read-only / in-memory backends; six of the nine backends (JAM, Squish, Hudson, PCBoard, EzyCom, GoldBase) flush every open stream they own. MSG inherits no-op (each WriteMessage opens / closes its own .msg file — the OS write buffer is flushed but not fsynced; Sync would have to fsync the directory entry which is a future enhancement). Wildcat inherits no-op (legacy `file` IO, not TFileStream). Tossers needing crash-safe per-message acknowledgement use the commit-after-fsync pattern: ```pascal base.WriteMessage(msg); base.Sync; { msg on platter } source.MarkSent(srcMsg); { commit the source pointer } ``` Without Sync, a crash after MarkSent but before the OS flushes the write buffer = silent message drop. Same durability problem fpc-ftn-transport's `TPktWriter.Sync` solves on the transport side; the symmetric surface is here for tossers writing to message bases. `Sync` raises after `Close` (data is finalized; nothing to flush). ### Capabilities API Each backend declares the canonical list of attribute keys it understands. Callers query before setting: ```pascal function TMessageBase.SupportsAttribute(const Key: AnsiString): boolean; function TMessageBase.SupportedAttributes: TStringDynArray; if base.SupportsAttribute('attr.returnreceipt') then RenderReceiptCheckbox; ``` ### Canonical attribute bits (`mb.types`) The MSG_ATTR_* cardinal constants stay as the internal pivot between native flag words and the individual `attr.*` boolean attributes: ``` MSG_ATTR_PRIVATE $00000001 MSG_ATTR_DELETED $00010000 MSG_ATTR_CRASH $00000002 MSG_ATTR_READ $00020000 MSG_ATTR_RECEIVED $00000004 MSG_ATTR_ECHO $00040000 MSG_ATTR_SENT $00000008 MSG_ATTR_DIRECT $00080000 MSG_ATTR_FILE_ATTACHED $00000010 MSG_ATTR_IMMEDIATE $00100000 MSG_ATTR_IN_TRANSIT $00000020 MSG_ATTR_FREQ_PENDING $00200000 MSG_ATTR_ORPHAN $00000040 MSG_ATTR_SEEN $00400000 MSG_ATTR_KILL_SENT $00000080 MSG_ATTR_LOCKED $00800000 MSG_ATTR_LOCAL $00000100 MSG_ATTR_NO_KILL $01000000 MSG_ATTR_HOLD $00000200 MSG_ATTR_NET_PENDING $02000000 MSG_ATTR_FILE_REQUEST $00000400 MSG_ATTR_ECHO_PENDING $04000000 MSG_ATTR_RETURN_RCPT $00000800 MSG_ATTR_IS_RECEIPT $00001000 MSG_ATTR_AUDIT_REQ $00002000 MSG_ATTR_FILE_UPD_REQ $00004000 ``` Bits 0..15 match FTS-0001 exactly; 16+ are for storage-layer flags (deleted/read/echo/etc.) that aren't part of the FTN wire format. Use `UniAttrBitsToAttributes` / `UniAttrBitsFromAttributes` helpers in `mb.types` to bridge the bitset to/from individual `attr.*` boolean attributes. ### FTN addressing ```pascal TFTNAddress = record Zone, Net, Node, Point: word; end; function MakeFTNAddress(AZone, ANet, ANode, APoint: word): TFTNAddress; function FTNAddressToString(const A: TFTNAddress): string; { "1:100/200.3" } function ParseFTNAddress(const S: string; out A: TFTNAddress): boolean; function FTNAddressEqual(const A, B: TFTNAddress): boolean; ``` --- ## Reading ```pascal function ReadMessage(Index: longint; var Msg: TUniMessage): boolean; property MessageCount: longint; property AreaTag: AnsiString; { auto-pop msg.area on Read } property ActiveUser: AnsiString; { auto-bump HWM on Read } ``` Zero-based index. Returns False on EOF or backend failure. Message numbers come back in `msg.Attributes.GetInt('msg.num')` and are backend-assigned (typically **don't** match the index; most formats keep a gap-tolerant index). Fires `metMessageRead` on success. If `AreaTag` is set (either via `MessageBaseOpen`'s `AAreaTag` parameter or the property setter post-construction), every successful Read auto-populates `msg.Attributes['area']` with the tag, unless the adapter already populated it from on-disk data (e.g. PKT's AREA kludge). If `ActiveUser` is set and the backend supports HWM, every successful Read advances the per-user HWM if `msg.num >` current HWM (never decrements). See *HWM* below. ```pascal base.ActiveUser := 'NetReader'; { optional: HWM auto-bump } for i := 0 to base.MessageCount - 1 do if base.ReadMessage(i, msg) then Handle(msg); ``` --- ## Writing ```pascal function WriteMessage(var Msg: TUniMessage): boolean; ``` Appends a new message. Backend assigns `msg.Attributes['msg.num']` on success. Fires `metMessageWritten`. Raises `EMessageBase` in read-only mode. PKT (via `tt.pkt.reader` from fpc-ftn-transport) raises `EMessageBase` on Write — use `tt.pkt.writer.TPktWriter` for outbound packets. ```pascal msg.Attributes.Clear; msg.Attributes.SetValue('from', 'Sysop'); msg.Attributes.SetValue('to', 'All'); msg.Attributes.SetValue('subject', 'Hello'); msg.Attributes.SetDate ('date.written', Now); msg.Attributes.SetBool ('attr.local', true); msg.Attributes.SetBool ('attr.echo', true); msg.Attributes.SetAddr ('addr.orig', MakeFTNAddress(1, 100, 1, 0)); msg.Attributes.SetAddr ('addr.dest', MakeFTNAddress(1, 100, 2, 0)); msg.Body := 'Hello, world'; base.WriteMessage(msg); ``` ### High-Water Mark (HWM) Per-user "last message I scanned" pointer. Native for JAM, Squish, Hudson, GoldBase; -1 (unsupported) for the others. ```pascal function SupportsHWM: boolean; function GetHWM(const UserName: AnsiString): longint; procedure SetHWM(const UserName: AnsiString; MsgNum: longint); procedure MapUser(const UserName: AnsiString; UserId: longint); property ActiveUser: AnsiString; property Board: longint; { multi-board context } ``` Tossers / scanners register as named users (e.g. `'NetReader'`, `'Allfix'`, `'FidoMail-Toss'`); each gets its own slot in the format's native lastread file, so multiple consumers coexist. Number-keyed formats (Hudson, GoldBase, EzyCom) need `MapUser('NetReader', 60001)` (pick `60000+` to avoid colliding with real BBS users) and `Board := ` before HWM ops; otherwise return -1. See [`docs/architecture.md`](architecture.md) HWM section for the full coverage map and rationale. --- ## Updating & deleting ```pascal function UpdateMessage(Index: longint; var Msg: TUniMessage): boolean; function DeleteMessage(Index: longint): boolean; ``` **`UpdateMessage` — header-metadata-only, every backend (since v0.6.0).** Reads the existing message, applies header-shaped attrs from `Msg`, rewrites the fixed-size header / ISAM record in place. Body, CtrlInfo (Squish), inline kludges (`*.MSG`), and SubFields (JAM) are NOT touched. Typical use cases: flipping an attribute bit (deleted / received / sent), updating reply-chain pointers (`jam.reply1st` / `jam.replynext`, `squish.reply1st` / `squish.replynext` list, `hudson.prevreply` / `hudson.nextreply`, etc.), rewriting `cost` or `msg.timesread`. Callers that need body-level changes should `DeleteMessage` + `WriteMessage`. Per-backend header-slot coverage — the attrs each backend honours on Update are the same slots its `ClassSupportedAttributes` publishes as "header-shaped." See [`attributes-registry.md`](attributes-registry.md) for the full matrix. `DeleteMessage` is currently optional — not every backend implements it; the default returns False. Attribute-only deletion also works via the native API (`.Native`) by flipping the `attr.deleted` bit on `UpdateMessage` and calling `Pack`; that pattern works on every backend. --- ## Packing & reindexing ```pascal function Pack(PurgeAgeDays, PurgeMaxCount: longint; Backup: boolean = True): boolean; function ReIndex: boolean; ``` `Pack` purges deleted/old messages and rebuilds the physical files. `PurgeAgeDays = 0` and `PurgeMaxCount = 0` means "don't purge by age or count; just compact deleted messages." Fires `metPackStarted` / `metPackComplete`. `ReIndex` rebuilds the index file from the header file (JAM/Hudson). ```pascal { no-op pack: only drop anything already marked deleted } base.Pack(0, 0, False); { purge anything older than 90 days, keep at most 500 messages } base.Pack(90, 500, True); ``` --- ## Events & logging Every `TMessageBase` owns a `TMessageEvents` reachable as `.Events`. Subscribe with either a log-shaped handler or a full-info handler, or both. Multiple hooks can be registered with `AddHook`. ```pascal TMessageLogHandler = procedure(Level: TMsgEventType; const Source, Msg: AnsiString) of object; TMessageEventHandler = procedure(Sender: TObject; const Info: TMessageEventInfo) of object; ``` ### Example — console logger ```pascal type TLogger = class procedure OnLog(Level: TMsgEventType; const Source, Msg: AnsiString); end; procedure TLogger.OnLog(Level: TMsgEventType; const Source, Msg: AnsiString); begin WriteLn('[', EventTypeToStr(Level), '] ', Source, ': ', Msg); end; var log: TLogger; begin log := TLogger.Create; base.Events.OnLog := @log.OnLog; ... ``` ### Event types (`TMsgEventType`) | Group | Values | |------------|---------------------------------------------------------------------| | Log levels | `metInfo`, `metWarning`, `metError`, `metDebug` | | Lifecycle | `metBaseOpened`, `metBaseClosed`, `metBaseCreated` | | Messages | `metMessageRead`, `metMessageWritten`, `metMessageDeleted`, `metMessageUpdated` | | Locking | `metLockAcquired`, `metLockReleased`, `metLockTimeout`, `metLockFailed` | | Pack | `metPackStarted`, `metPackProgress`, `metPackComplete`, `metReindexStarted`, `metReindexComplete` | | Packets | `metPacketStart`, `metPacketEnd`, `metPacketMessage`, `metPacketError` | Filter by severity: ```pascal base.Events.MinLevel := metWarning; { drop info + debug } ``` --- ## Locking Three layers, all in `mb.lock`: 1. **TRTLCriticalSection** per `TMessageBase` instance — serialises concurrent Read/Write/Update/Delete calls on the same instance. 2. **Advisory sentinel** — `fpflock` (Unix), `LockFileEx` (Windows), `DosSetFileLocks` (OS/2), exclusive-open (DOS) on a per-base `.lck` file. Released and unlinked on close. 3. **Native share modes** — each backend opens its data streams with `fmShareDenyWrite` / `fmShareDenyNone` as the backstop. The sentinel timeout is configurable per instance: ```pascal base.LockTimeoutMs := 5000; { retry for 5 s, fail otherwise } base.LockTimeoutMs := 0; { fail fast — don't wait at all } base.LockTimeoutMs := -1; { wait forever } ``` Direct access: `base.Lock` returns the `TMessageLock` for the rare case where you want to hold the lock across multiple operations. --- ## Path helpers ```pascal function MessageBasePathFor(AFormat: TMsgBaseFormat; const AAreaPath, AAreaTag: AnsiString): AnsiString; function FindExistingFile(const APath: AnsiString): AnsiString; function PathJoin(const ADir, ATail: AnsiString): AnsiString; function LockFilePath(AFormat: TMsgBaseFormat; const ABasePath, AAreaTag: AnsiString): AnsiString; ``` `MessageBasePathFor` produces the canonical constructor argument each backend expects from an area directory + optional tag. `FindExistingFile` does case-insensitive resolution (path → UPPER → lower) for Linux hosts where on-disk names may be mixed case. --- ## Concurrent tossers — moved `TPacketBatch` (was `ma.batch` here pre-0.4.0) moved to `fpc-ftn-transport` as `tt.pkt.batch` along with the rest of the PKT code. Class name and API surface unchanged for caller compatibility. Use it from there: ```pascal uses {$IFDEF UNIX}cthreads,{$ENDIF} mb.api, { TMessageBase + factory } mb.fmt.jam, mb.fmt.jam.uni, { destination msgbase } tt.pkt.format, { wire-format types } tt.pkt.reader, { registers mbfPkt } tt.pkt.batch; { TPacketBatch + Run loop } ``` See `fpc-ftn-transport/docs/` for the worker-pool details. --- ## Dropping to a native backend Every `.uni` adapter exposes `Native` for callers who need format- specific features the unified API doesn't abstract (JAM subfields, Squish UMsgId, EzyCom dual-byte attributes, Wildcat NextMsg walking, etc.). Note that many previously-native-only operations have uni-API equivalents now — e.g. header rewrites go through `base.UpdateMessage` since v0.6.0 and don't need the `Native` dance below. Drop to `Native` only when the unified API really can't express what you need. Native header-rewrite example (pre-v0.6.0 pattern, still supported): ```pascal var adapter: TJamMessageBase; hdr: JamHdr; nat: TJamMessage; begin adapter := base as TJamMessageBase; adapter.Native.ReadMessage(Index, nat); adapter.Native.ReadHeader(nat.HdrOffset, hdr); hdr.Attribute := hdr.Attribute or longint($80000000); { mark deleted } adapter.Native.UpdateHeader(nat.HdrOffset, hdr); adapter.Native.IncModCounter; adapter.Native.UpdateHdrInfo; end; ``` Native class names: | Format | Native class | Adapter | |-----------|-------------------|----------------------| | Hudson | `THudsonBase` | `THudsonMessageBase` | | JAM | `TJamBase` | `TJamMessageBase` | | Squish | `TSquishBase` | `TSquishMessageBase` | | FTS-1 MSG | `TMsgDir`+`TMsgFile` | `TMsgMessageBase` | | FTN PKT | `TPktFile` *(in fpc-ftn-transport: tt.pkt.format)* | `TPktReader` *(tt.pkt.reader)* | | PCBoard | `TPCBoardBase` | `TPCBoardMessageBase`| | EzyCom | `TEzyComBase` | `TEzyComMessageBase` | | GoldBase | `TGoldBase` | `TGoldBaseMessageBase`| | Wildcat | `TWildcatBase` | `TWildcatMessageBase`| --- ## Format cheat-sheet | Format | Base path shape | Caveats | |----------|--------------------------|--------------------------------------------| | Hudson | directory | 32767-message ceiling (smallint MsgNum) | | GoldBase | directory | Widened Hudson (word MsgAttr, 500 boards) | | JAM | dir + basename, no ext | Adapter appends `.JHR/.JDT/.JDX/.JLR` | | Squish | dir + basename, no ext | Adapter appends `.SQD/.SQI/.SQL` | | FTS-1 MSG| directory | Numbered `*.msg` / `*.MSG` (case mixed) | | FTN PKT | full packet filename | Lives in `fpc-ftn-transport`. `uses tt.pkt.reader` registers `mbfPkt` with the unified factory; iterate with `MessageBaseOpen(mbfPkt, ...)`. Outbound writes via `tt.pkt.writer.TPktWriter` (atomic `.tmp→rename`, file/stream targets, variant selection). | | PCBoard | dir + basename, no ext | Adapter appends `.MSG/.IDX` | | EzyCom | directory + `.Board` | Set `adapter.Board`/`.BBSType` before Open | | Wildcat | WC data dir + `.Conference` | Set `adapter.Conference` before Open | --- ## See also - `docs/architecture.md` — layered design - `docs/ftsc-compliance.md` — spec refs - `docs/format-notes/` — per-format quirks and dependencies - `docs/attributes-registry.md` — full attribute key catalog + per-format support matrix - `examples/` — runnable `example_read`, `example_write`, `example_tosser` - `tests/` — test_read, test_roundtrip, test_roundtrip_attrs, test_lock, test_batch, test_wildcat, test_write_existing, test_pack, test_hwm, test_consumer_round1 - `mb.kludge` — shared FTSC kludge parsing/emission helpers (`ParseKludgeLine`, `SplitKludgeBlob`, `BuildKludgePrefix`, `BuildKludgeSuffix`) for callers that need to handle raw FTSC body blobs outside an adapter