fpc-msgbase/docs/API.md

fpc-msgbase — API reference

Every callable the library exposes, with parameters, return values, and runnable examples. See docs/architecture.md for the big picture and docs/ftsc-compliance.md for spec notes.

Contents

• Quick start
• Opening a base
• TUniMessage
• Reading
• Writing
• Updating & deleting
• Packing & reindexing
• Events & logging
• Locking
• Path helpers
• Concurrent tossers — moved
• Dropping to a native backend
• Format cheat-sheet

---

Quick start

program hello;
{$mode objfpc}{$H+}
uses
  SysUtils,
  ma.types, ma.events, ma.api,
  ma.fmt.jam, ma.fmt.jam.uni;

var
  base: TMessageBase;
  msg:  TUniMessage;
  i:    longint;
begin
  base := MessageBaseOpen(mbfJam,
                          '/home/ken/fidonet/msg/jam/10thamd',
                          momReadOnly);
  try
    if not base.Open then Halt(1);
    for i := 0 to base.MessageCount - 1 do
      if base.ReadMessage(i, msg) then
        WriteLn(msg.Attributes.Get('from'),
                ' -> ', msg.Attributes.Get('to'),
                ': ',   msg.Attributes.Get('subject'));
  finally
    base.Close;
    base.Free;
  end;
end.

Add format support by including the matching pair of units:

Format	Units to add to uses
Hudson	ma.fmt.hudson, ma.fmt.hudson.uni
JAM	ma.fmt.jam, ma.fmt.jam.uni
Squish	ma.fmt.squish, ma.fmt.squish.uni
FTS-1 MSG	ma.fmt.msg, ma.fmt.msg.uni
FTN PKT	(moved to fpc-ftn-transport; uses tt.pkt.reader, tt.pkt.writer)
PCBoard	ma.fmt.pcboard, ma.fmt.pcboard.uni
EzyCom	ma.fmt.ezycom, ma.fmt.ezycom.uni
GoldBase	ma.fmt.goldbase, ma.fmt.goldbase.uni
Wildcat	ma.fmt.wildcat, ma.fmt.wildcat.uni

The .uni unit's initialization section registers the factory; without it, MessageBaseOpen(<format>, ...) raises EMessageBase.

Threads on Unix need cthreads first in the program's uses:

uses
  {$IFDEF UNIX}cthreads,{$ENDIF}
  ma.api, ...;

---

Opening a base

MessageBaseOpen

function MessageBaseOpen(AFormat: TMsgBaseFormat;
                         const AAreaPath: AnsiString;
                         AMode: TMsgOpenMode;
                         const AAreaTag: AnsiString = ''): TMessageBase;

Returns a not-yet-opened TMessageBase for the requested format. Call .Open before reading/writing. Raises EMessageBase if the format has no registered backend.

Parameter	Meaning
AFormat	mbfHudson / mbfJam / mbfSquish / etc.
AAreaPath	Directory for dir-based formats, basename for JAM/Squish/PCB, full filename for PKT (PKT lives in fpc-ftn-transport but the mbfPkt factory enum is here).
AMode	momReadOnly, momReadWrite, or momCreate.
AAreaTag	Optional area tag; passed through to the adapter.

MessageBaseOpenAuto

function MessageBaseOpenAuto(const AAreaPath: AnsiString;
                             AMode: TMsgOpenMode): TMessageBase;

Sniffs the directory for signature files (MSGINFO.BBS, *.JHR, *.SQD, numbered *.MSG, …) and picks the right backend. Returns nil when no format matches.

DetectFormat

function DetectFormat(const AAreaPath: AnsiString;
                      out AFormat: TMsgBaseFormat): boolean;

Same sniffer as MessageBaseOpenAuto, but just returns the format without opening.

TMessageBase.Open / Close

function  Open: boolean;
procedure Close;

Open acquires the cross-process sentinel lock (exclusive for writers, shared for readers), then delegates to the backend's native open. Fires metBaseOpened. Returns False if the base files are missing or another writer is holding the lock.

Close delegates to the backend, releases the sentinel lock, unlinks the .lck file, and fires metBaseClosed.

Mode semantics

Mode	Behaviour
momReadOnly	Read-only. No write/pack allowed. Shared lock, failures tolerated (e.g. RO FS).
momReadWrite	Files must exist; exclusive lock.
momCreate	Creates empty format files if missing, then opens exclusive.

---

TUniMessage — two-area model

Single format-agnostic record. Body holds only the message text; everything else lives in Attributes as namespaced key/value pairs. Backends convert between their native record and this on each read/write.

TUniMessage = record
  Body:       AnsiString;       { only the message text }
  Attributes: TMsgAttributes;   { everything else, key/value }
end;

The full attribute key catalog with per-format support matrix lives in docs/attributes-registry.md.

TMsgAttributes accessors

{ Setters }
procedure SetValue(const K, V: AnsiString);
procedure SetInt(const K: AnsiString; V: longint);
procedure SetInt64(const K: AnsiString; V: int64);
procedure SetBool(const K: AnsiString; V: boolean);
procedure SetDate(const K: AnsiString; V: TDateTime);
procedure SetAddr(const K: AnsiString; const V: TFTNAddress);
procedure SetList(const K: AnsiString; const V: TStringDynArray);
procedure AppendListItem(const K, Item: AnsiString);

{ Getters }
function  Get(const K: AnsiString;
              const Def: AnsiString = ''): AnsiString;
function  GetInt   (const K: AnsiString; Def: longint   = 0): longint;
function  GetInt64 (const K: AnsiString; Def: int64     = 0): int64;
function  GetBool  (const K: AnsiString; Def: boolean   = false): boolean;
function  GetDate  (const K: AnsiString; Def: TDateTime = 0): TDateTime;
function  GetAddr  (const K: AnsiString): TFTNAddress;
function  GetList  (const K: AnsiString): TStringDynArray;

{ Inspection }
function  Has(const K: AnsiString): boolean;
procedure Remove(const K: AnsiString);
procedure Clear;
function  Count: longint;
function  KeyAt(I: longint): AnsiString;
function  ValueAt(I: longint): AnsiString;

Common keys — see docs/attributes-registry.md:

Key	Type	Meaning
msg.num	int	backend-assigned message number
from / to / subject	string	universal headers
date.written / date.received	date	timestamps
addr.orig / addr.dest	ftn	FTN addresses
area	string	echo area tag (auto-pop from base.AreaTag)
board	int	multi-board format conference number
attr.*	bool	private/crash/sent/read/etc. (see registry)
msgid / replyid / pid / flags	string	FTSC kludges
seen-by / path / via	multi	FTS-4 routing kludges (use GetList)
intl / fmpt / topt	string	FSC-4008 cross-zone routing
kludge.<name>	string	unknown FTSC kludge passthrough

Sync (durability)

TMessageBase.Sync forces every open writer stream to durable storage (fpfsync on Unix, FlushFileBuffers on Windows). Override default no-op for read-only / in-memory backends; six of the nine backends (JAM, Squish, Hudson, PCBoard, EzyCom, GoldBase) flush every open stream they own. MSG inherits no-op (each WriteMessage opens / closes its own .msg file — the OS write buffer is flushed but not fsynced; Sync would have to fsync the directory entry which is a future enhancement). Wildcat inherits no-op (legacy file IO, not TFileStream).

Tossers needing crash-safe per-message acknowledgement use the commit-after-fsync pattern:

base.WriteMessage(msg);
base.Sync;                    { msg on platter }
source.MarkSent(srcMsg);      { commit the source pointer }

Without Sync, a crash after MarkSent but before the OS flushes the write buffer = silent message drop. Same durability problem fpc-ftn-transport's TPktWriter.Sync solves on the transport side; the symmetric surface is here for tossers writing to message bases.

Sync raises after Close (data is finalized; nothing to flush).

Capabilities API

Each backend declares the canonical list of attribute keys it understands. Callers query before setting:

function  TMessageBase.SupportsAttribute(const Key: AnsiString): boolean;
function  TMessageBase.SupportedAttributes: TStringDynArray;

if base.SupportsAttribute('attr.returnreceipt') then
  RenderReceiptCheckbox;

Canonical attribute bits (ma.types)

The MSG_ATTR_* cardinal constants stay as the internal pivot between native flag words and the individual attr.* boolean attributes:

MSG_ATTR_PRIVATE        $00000001    MSG_ATTR_DELETED       $00010000
MSG_ATTR_CRASH          $00000002    MSG_ATTR_READ          $00020000
MSG_ATTR_RECEIVED       $00000004    MSG_ATTR_ECHO          $00040000
MSG_ATTR_SENT           $00000008    MSG_ATTR_DIRECT        $00080000
MSG_ATTR_FILE_ATTACHED  $00000010    MSG_ATTR_IMMEDIATE     $00100000
MSG_ATTR_IN_TRANSIT     $00000020    MSG_ATTR_FREQ_PENDING  $00200000
MSG_ATTR_ORPHAN         $00000040    MSG_ATTR_SEEN          $00400000
MSG_ATTR_KILL_SENT      $00000080    MSG_ATTR_LOCKED        $00800000
MSG_ATTR_LOCAL          $00000100    MSG_ATTR_NO_KILL       $01000000
MSG_ATTR_HOLD           $00000200    MSG_ATTR_NET_PENDING   $02000000
MSG_ATTR_FILE_REQUEST   $00000400    MSG_ATTR_ECHO_PENDING  $04000000
MSG_ATTR_RETURN_RCPT    $00000800
MSG_ATTR_IS_RECEIPT     $00001000
MSG_ATTR_AUDIT_REQ      $00002000
MSG_ATTR_FILE_UPD_REQ   $00004000

Bits 0..15 match FTS-0001 exactly; 16+ are for storage-layer flags (deleted/read/echo/etc.) that aren't part of the FTN wire format.

Use UniAttrBitsToAttributes / UniAttrBitsFromAttributes helpers in ma.types to bridge the bitset to/from individual attr.* boolean attributes.

FTN addressing

TFTNAddress = record Zone, Net, Node, Point: word; end;

function MakeFTNAddress(AZone, ANet, ANode, APoint: word): TFTNAddress;
function FTNAddressToString(const A: TFTNAddress): string;       { "1:100/200.3" }
function ParseFTNAddress(const S: string; out A: TFTNAddress): boolean;
function FTNAddressEqual(const A, B: TFTNAddress): boolean;

---

Reading

function ReadMessage(Index: longint; var Msg: TUniMessage): boolean;
property MessageCount: longint;
property AreaTag: AnsiString;            { auto-pop msg.area on Read }
property ActiveUser: AnsiString;         { auto-bump HWM on Read }

Zero-based index. Returns False on EOF or backend failure. Message numbers come back in msg.Attributes.GetInt('msg.num') and are backend-assigned (typically don't match the index; most formats keep a gap-tolerant index). Fires metMessageRead on success.

If AreaTag is set (either via MessageBaseOpen's AAreaTag parameter or the property setter post-construction), every successful Read auto-populates msg.Attributes['area'] with the tag, unless the adapter already populated it from on-disk data (e.g. PKT's AREA kludge).

If ActiveUser is set and the backend supports HWM, every successful Read advances the per-user HWM if msg.num > current HWM (never decrements). See HWM below.

base.ActiveUser := 'NetReader';   { optional: HWM auto-bump }
for i := 0 to base.MessageCount - 1 do
  if base.ReadMessage(i, msg) then
    Handle(msg);

---

Writing

function WriteMessage(var Msg: TUniMessage): boolean;

Appends a new message. Backend assigns msg.Attributes['msg.num'] on success. Fires metMessageWritten. Raises EMessageBase in read-only mode. PKT (via tt.pkt.reader from fpc-ftn-transport) raises EMessageBase on Write — use tt.pkt.writer.TPktWriter for outbound packets.

msg.Attributes.Clear;
msg.Attributes.SetValue('from',    'Sysop');
msg.Attributes.SetValue('to',      'All');
msg.Attributes.SetValue('subject', 'Hello');
msg.Attributes.SetDate ('date.written', Now);
msg.Attributes.SetBool ('attr.local', true);
msg.Attributes.SetBool ('attr.echo',  true);
msg.Attributes.SetAddr ('addr.orig', MakeFTNAddress(1, 100, 1, 0));
msg.Attributes.SetAddr ('addr.dest', MakeFTNAddress(1, 100, 2, 0));
msg.Body := 'Hello, world';
base.WriteMessage(msg);

High-Water Mark (HWM)

Per-user "last message I scanned" pointer. Native for JAM, Squish, Hudson, GoldBase; -1 (unsupported) for the others.

function  SupportsHWM: boolean;
function  GetHWM(const UserName: AnsiString): longint;
procedure SetHWM(const UserName: AnsiString; MsgNum: longint);
procedure MapUser(const UserName: AnsiString; UserId: longint);
property  ActiveUser: AnsiString;
property  Board: longint;          { multi-board context }

Tossers / scanners register as named users (e.g. 'NetReader', 'Allfix', 'FidoMail-Toss'); each gets its own slot in the format's native lastread file, so multiple consumers coexist.

Number-keyed formats (Hudson, GoldBase, EzyCom) need MapUser('NetReader', 60001) (pick 60000+ to avoid colliding with real BBS users) and Board := <n> before HWM ops; otherwise return -1.

See docs/architecture.md HWM section for the full coverage map and rationale.

---

Updating & deleting

function UpdateMessage(Index: longint; var Msg: TUniMessage): boolean;
function DeleteMessage(Index: longint): boolean;

Optional operations. Not every backend implements them — the default returns False. Attribute-only changes can also be made through the native API (.Native) by flipping the MSG_ATTR_DELETED bit and calling Pack; that pattern works on every backend.

---

Packing & reindexing

function Pack(PurgeAgeDays, PurgeMaxCount: longint;
              Backup: boolean = True): boolean;
function ReIndex: boolean;

Pack purges deleted/old messages and rebuilds the physical files. PurgeAgeDays = 0 and PurgeMaxCount = 0 means "don't purge by age or count; just compact deleted messages." Fires metPackStarted / metPackComplete.

ReIndex rebuilds the index file from the header file (JAM/Hudson).

{ no-op pack: only drop anything already marked deleted }
base.Pack(0, 0, False);

{ purge anything older than 90 days, keep at most 500 messages }
base.Pack(90, 500, True);

---

Events & logging

Every TMessageBase owns a TMessageEvents reachable as .Events. Subscribe with either a log-shaped handler or a full-info handler, or both. Multiple hooks can be registered with AddHook.

TMessageLogHandler =
  procedure(Level: TMsgEventType;
            const Source, Msg: AnsiString) of object;

TMessageEventHandler =
  procedure(Sender: TObject;
            const Info: TMessageEventInfo) of object;

Example — console logger

type
  TLogger = class
    procedure OnLog(Level: TMsgEventType;
                    const Source, Msg: AnsiString);
  end;

procedure TLogger.OnLog(Level: TMsgEventType;
                        const Source, Msg: AnsiString);
begin
  WriteLn('[', EventTypeToStr(Level), '] ', Source, ': ', Msg);
end;

var
  log: TLogger;
begin
  log := TLogger.Create;
  base.Events.OnLog := @log.OnLog;
  ...

Event types (TMsgEventType)

Group	Values
Log levels	metInfo, metWarning, metError, metDebug
Lifecycle	metBaseOpened, metBaseClosed, metBaseCreated
Messages	metMessageRead, metMessageWritten, metMessageDeleted, metMessageUpdated
Locking	metLockAcquired, metLockReleased, metLockTimeout, metLockFailed
Pack	metPackStarted, metPackProgress, metPackComplete, metReindexStarted, metReindexComplete
Packets	metPacketStart, metPacketEnd, metPacketMessage, metPacketError

Filter by severity:

base.Events.MinLevel := metWarning;  { drop info + debug }

---

Locking

Three layers, all in ma.lock:

1. TRTLCriticalSection per TMessageBase instance — serialises concurrent Read/Write/Update/Delete calls on the same instance.
2. Advisory sentinel — fpflock (Unix), LockFileEx (Windows), DosSetFileLocks (OS/2), exclusive-open (DOS) on a per-base .lck file. Released and unlinked on close.
3. Native share modes — each backend opens its data streams with fmShareDenyWrite / fmShareDenyNone as the backstop.

The sentinel timeout is configurable per instance:

base.LockTimeoutMs := 5000;   { retry for 5 s, fail otherwise }
base.LockTimeoutMs := 0;      { fail fast — don't wait at all }
base.LockTimeoutMs := -1;     { wait forever }

Direct access: base.Lock returns the TMessageLock for the rare case where you want to hold the lock across multiple operations.

---

Path helpers

function MessageBasePathFor(AFormat: TMsgBaseFormat;
                            const AAreaPath, AAreaTag: AnsiString): AnsiString;
function FindExistingFile(const APath: AnsiString): AnsiString;
function PathJoin(const ADir, ATail: AnsiString): AnsiString;
function LockFilePath(AFormat: TMsgBaseFormat;
                      const ABasePath, AAreaTag: AnsiString): AnsiString;

MessageBasePathFor produces the canonical constructor argument each backend expects from an area directory + optional tag. FindExistingFile does case-insensitive resolution (path → UPPER → lower) for Linux hosts where on-disk names may be mixed case.

---

Concurrent tossers — moved

TPacketBatch (was ma.batch here pre-0.4.0) moved to fpc-ftn-transport as tt.pkt.batch along with the rest of the PKT code. Class name and API surface unchanged for caller compatibility.

Use it from there:

uses
  {$IFDEF UNIX}cthreads,{$ENDIF}
  ma.api,                        { TMessageBase + factory  }
  ma.fmt.jam, ma.fmt.jam.uni,    { destination msgbase }
  tt.pkt.format,                 { wire-format types       }
  tt.pkt.reader,                 { registers mbfPkt        }
  tt.pkt.batch;                  { TPacketBatch + Run loop }

See fpc-ftn-transport/docs/ for the worker-pool details.

---

Dropping to a native backend

Every .uni adapter exposes Native for callers who need format- specific features the unified API doesn't abstract (JAM subfields, Squish UMsgId, EzyCom dual-byte attributes, Wildcat NextMsg walking, etc.):

var
  adapter: TJamMessageBase;
  hdr:     JamHdr;
  nat:     TJamMessage;
begin
  adapter := base as TJamMessageBase;
  adapter.Native.ReadMessage(Index, nat);
  adapter.Native.ReadHeader(nat.HdrOffset, hdr);
  hdr.Attribute := hdr.Attribute or longint($80000000);   { mark deleted }
  adapter.Native.UpdateHeader(nat.HdrOffset, hdr);
  adapter.Native.IncModCounter;
  adapter.Native.UpdateHdrInfo;
end;

Native class names:

Format	Native class	Adapter
Hudson	THudsonBase	THudsonMessageBase
JAM	TJamBase	TJamMessageBase
Squish	TSquishBase	TSquishMessageBase
FTS-1 MSG	TMsgDir+TMsgFile	TMsgMessageBase
FTN PKT	TPktFile (in fpc-ftn-transport: tt.pkt.format)	TPktReader (tt.pkt.reader)
PCBoard	TPCBoardBase	TPCBoardMessageBase
EzyCom	TEzyComBase	TEzyComMessageBase
GoldBase	TGoldBase	TGoldBaseMessageBase
Wildcat	TWildcatBase	TWildcatMessageBase

---

Format cheat-sheet

Format	Base path shape	Caveats
Hudson	directory	32767-message ceiling (smallint MsgNum)
GoldBase	directory	Widened Hudson (word MsgAttr, 500 boards)
JAM	dir + basename, no ext	Adapter appends .JHR/.JDT/.JDX/.JLR
Squish	dir + basename, no ext	Adapter appends .SQD/.SQI/.SQL
FTS-1 MSG	directory	Numbered *.msg / *.MSG (case mixed)
FTN PKT	full packet filename	Lives in fpc-ftn-transport. uses tt.pkt.reader registers mbfPkt with the unified factory; iterate with MessageBaseOpen(mbfPkt, ...). Outbound writes via tt.pkt.writer.TPktWriter (atomic .tmp→rename, file/stream targets, variant selection).
PCBoard	dir + basename, no ext	Adapter appends .MSG/.IDX
EzyCom	directory + .Board	Set adapter.Board/.BBSType before Open
Wildcat	WC data dir + .Conference	Set adapter.Conference before Open

---

See also

• docs/architecture.md — layered design
• docs/ftsc-compliance.md — spec refs
• docs/format-notes/ — per-format quirks and dependencies
• docs/attributes-registry.md — full attribute key catalog + per-format support matrix
• examples/ — runnable example_read, example_write, example_tosser
• tests/ — test_read, test_roundtrip, test_roundtrip_attrs, test_lock, test_batch, test_wildcat, test_write_existing, test_pack, test_hwm, test_consumer_round1
• ma.kludge — shared FTSC kludge parsing/emission helpers (ParseKludgeLine, SplitKludgeBlob, BuildKludgePrefix, BuildKludgeSuffix) for callers that need to handle raw FTSC body blobs outside an adapter