Ken Johnson
6181b6abce
Project renamed from message_api → fpc-msgbase. Folder, README title, docs, build.sh, fpc.cfg, and test banners all updated for consistency with the planned remote at kjgr.io:2222/kenjreno/fpc-msgbase.git. Also scrubbed claims that backends were "ported from Allfix" or "match Allfix's msgutil/domsg" — none of this code was ported from Allfix; it was implemented from FTSC documents and the original format authors' published specs (jam.txt, squish.doc, pcboard.doc, EzyCom reference, WildCat 4 SDK headers). Author credits live in docs/ftsc-compliance.md. Real interop facts that mention Allfix-the-product stay documented: the PCB Extra2 sent-bit ($40) Allfix sets when tossing, and the FTSC-registered product code $EB. These describe external software behavior we interoperate with, not provenance. docs/format-notes/hudson.md removed — stale planning doc that predates the working ma.fmt.hudson backend.
522 lines
16 KiB
Markdown
522 lines
16 KiB
Markdown
# fpc-msgbase — API reference
|
|
|
|
Every callable the library exposes, with parameters, return values,
|
|
and runnable examples. See `docs/architecture.md` for the big
|
|
picture and `docs/ftsc-compliance.md` for spec notes.
|
|
|
|
## Contents
|
|
|
|
- [Quick start](#quick-start)
|
|
- [Opening a base](#opening-a-base)
|
|
- [TUniMessage](#tunimessage)
|
|
- [Reading](#reading)
|
|
- [Writing](#writing)
|
|
- [Updating & deleting](#updating--deleting)
|
|
- [Packing & reindexing](#packing--reindexing)
|
|
- [Events & logging](#events--logging)
|
|
- [Locking](#locking)
|
|
- [Path helpers](#path-helpers)
|
|
- [Concurrent tossers — TPacketBatch](#concurrent-tossers--tpacketbatch)
|
|
- [Dropping to a native backend](#dropping-to-a-native-backend)
|
|
- [Format cheat-sheet](#format-cheat-sheet)
|
|
|
|
---
|
|
|
|
## Quick start
|
|
|
|
```pascal
|
|
program hello;
|
|
{$mode objfpc}{$H+}
|
|
uses
|
|
SysUtils,
|
|
ma.types, ma.events, ma.api,
|
|
ma.fmt.jam, ma.fmt.jam.uni;
|
|
|
|
var
|
|
base: TMessageBase;
|
|
msg: TUniMessage;
|
|
i: longint;
|
|
begin
|
|
base := MessageBaseOpen(mbfJam,
|
|
'/home/ken/fidonet/msg/jam/10thamd',
|
|
momReadOnly);
|
|
try
|
|
if not base.Open then Halt(1);
|
|
for i := 0 to base.MessageCount - 1 do
|
|
if base.ReadMessage(i, msg) then
|
|
WriteLn(msg.WhoFrom, ' -> ', msg.WhoTo, ': ', msg.Subject);
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
end.
|
|
```
|
|
|
|
**Add format support** by including the matching pair of units:
|
|
|
|
| Format | Units to add to `uses` |
|
|
|-----------|---------------------------------------------|
|
|
| Hudson | `ma.fmt.hudson, ma.fmt.hudson.uni` |
|
|
| JAM | `ma.fmt.jam, ma.fmt.jam.uni` |
|
|
| Squish | `ma.fmt.squish, ma.fmt.squish.uni` |
|
|
| FTS-1 MSG | `ma.fmt.msg, ma.fmt.msg.uni` |
|
|
| FTN PKT | `ma.fmt.pkt, ma.fmt.pkt.uni` |
|
|
| PCBoard | `ma.fmt.pcboard, ma.fmt.pcboard.uni` |
|
|
| EzyCom | `ma.fmt.ezycom, ma.fmt.ezycom.uni` |
|
|
| GoldBase | `ma.fmt.goldbase, ma.fmt.goldbase.uni` |
|
|
| Wildcat | `ma.fmt.wildcat, ma.fmt.wildcat.uni` |
|
|
|
|
The `.uni` unit's `initialization` section registers the factory;
|
|
without it, `MessageBaseOpen(<format>, ...)` raises `EMessageBase`.
|
|
|
|
Threads on Unix need `cthreads` first in the program's `uses`:
|
|
|
|
```pascal
|
|
uses
|
|
{$IFDEF UNIX}cthreads,{$ENDIF}
|
|
ma.api, ...;
|
|
```
|
|
|
|
---
|
|
|
|
## Opening a base
|
|
|
|
### `MessageBaseOpen`
|
|
|
|
```pascal
|
|
function MessageBaseOpen(AFormat: TMsgBaseFormat;
|
|
const AAreaPath: AnsiString;
|
|
AMode: TMsgOpenMode;
|
|
const AAreaTag: AnsiString = ''): TMessageBase;
|
|
```
|
|
|
|
Returns a **not-yet-opened** `TMessageBase` for the requested format.
|
|
Call `.Open` before reading/writing. Raises `EMessageBase` if the
|
|
format has no registered backend.
|
|
|
|
| Parameter | Meaning |
|
|
|------------|-------------------------------------------------------------|
|
|
| AFormat | `mbfHudson` / `mbfJam` / `mbfSquish` / etc. |
|
|
| AAreaPath | Directory for dir-based formats, basename for JAM/Squish/PCB, full filename for PKT. |
|
|
| AMode | `momReadOnly`, `momReadWrite`, or `momCreate`. |
|
|
| AAreaTag | Optional area tag; passed through to the adapter. |
|
|
|
|
### `MessageBaseOpenAuto`
|
|
|
|
```pascal
|
|
function MessageBaseOpenAuto(const AAreaPath: AnsiString;
|
|
AMode: TMsgOpenMode): TMessageBase;
|
|
```
|
|
|
|
Sniffs the directory for signature files (`MSGINFO.BBS`, `*.JHR`,
|
|
`*.SQD`, numbered `*.MSG`, …) and picks the right backend. Returns
|
|
nil when no format matches.
|
|
|
|
### `DetectFormat`
|
|
|
|
```pascal
|
|
function DetectFormat(const AAreaPath: AnsiString;
|
|
out AFormat: TMsgBaseFormat): boolean;
|
|
```
|
|
|
|
Same sniffer as `MessageBaseOpenAuto`, but just returns the format
|
|
without opening.
|
|
|
|
### `TMessageBase.Open` / `Close`
|
|
|
|
```pascal
|
|
function Open: boolean;
|
|
procedure Close;
|
|
```
|
|
|
|
`Open` acquires the cross-process sentinel lock (exclusive for
|
|
writers, shared for readers), then delegates to the backend's
|
|
native open. Fires `metBaseOpened`. Returns False if the base
|
|
files are missing or another writer is holding the lock.
|
|
|
|
`Close` delegates to the backend, releases the sentinel lock,
|
|
unlinks the `.lck` file, and fires `metBaseClosed`.
|
|
|
|
### Mode semantics
|
|
|
|
| Mode | Behaviour |
|
|
|---------------|--------------------------------------------------------------------|
|
|
| `momReadOnly` | Read-only. No write/pack allowed. Shared lock, failures tolerated (e.g. RO FS). |
|
|
| `momReadWrite`| Files must exist; exclusive lock. |
|
|
| `momCreate` | Creates empty format files if missing, then opens exclusive. |
|
|
|
|
---
|
|
|
|
## TUniMessage
|
|
|
|
Single format-agnostic message record. Backends convert between
|
|
their native record and this on each read/write.
|
|
|
|
```pascal
|
|
TUniMessage = record
|
|
MsgNum: longint; { backend-assigned number }
|
|
WhoFrom: AnsiString;
|
|
WhoTo: AnsiString;
|
|
Subject: AnsiString;
|
|
DateWritten: TDateTime;
|
|
DateReceived: TDateTime; { 0 if not received }
|
|
Attr: cardinal; { MSG_ATTR_* bitset, see below }
|
|
OrigAddr: TFTNAddress;
|
|
DestAddr: TFTNAddress;
|
|
Cost: word;
|
|
Body: AnsiString; { CR-separated, kludges intact }
|
|
AreaTag: AnsiString; { optional echo-area tag }
|
|
Board: word; { conference/board; 0 = default }
|
|
end;
|
|
```
|
|
|
|
### Canonical attribute bits (`ma.types`)
|
|
|
|
```
|
|
MSG_ATTR_PRIVATE $00000001 MSG_ATTR_DELETED $00010000
|
|
MSG_ATTR_CRASH $00000002 MSG_ATTR_READ $00020000
|
|
MSG_ATTR_RECEIVED $00000004 MSG_ATTR_ECHO $00040000
|
|
MSG_ATTR_SENT $00000008 MSG_ATTR_DIRECT $00080000
|
|
MSG_ATTR_FILE_ATTACHED $00000010 MSG_ATTR_IMMEDIATE $00100000
|
|
MSG_ATTR_IN_TRANSIT $00000020 MSG_ATTR_FREQ_PENDING $00200000
|
|
MSG_ATTR_ORPHAN $00000040 MSG_ATTR_SEEN $00400000
|
|
MSG_ATTR_KILL_SENT $00000080 MSG_ATTR_LOCKED $00800000
|
|
MSG_ATTR_LOCAL $00000100 MSG_ATTR_NO_KILL $01000000
|
|
MSG_ATTR_HOLD $00000200 MSG_ATTR_NET_PENDING $02000000
|
|
MSG_ATTR_FILE_REQUEST $00000400 MSG_ATTR_ECHO_PENDING $04000000
|
|
MSG_ATTR_RETURN_RCPT $00000800
|
|
MSG_ATTR_IS_RECEIPT $00001000
|
|
MSG_ATTR_AUDIT_REQ $00002000
|
|
MSG_ATTR_FILE_UPD_REQ $00004000
|
|
```
|
|
|
|
Bits 0..15 match FTS-0001 exactly; 16+ are for storage-layer flags
|
|
(deleted/read/echo/etc.) that aren't part of the FTN wire format.
|
|
|
|
### FTN addressing
|
|
|
|
```pascal
|
|
TFTNAddress = record Zone, Net, Node, Point: word; end;
|
|
|
|
function MakeFTNAddress(AZone, ANet, ANode, APoint: word): TFTNAddress;
|
|
function FTNAddressToString(const A: TFTNAddress): string; { "1:100/200.3" }
|
|
function ParseFTNAddress(const S: string; out A: TFTNAddress): boolean;
|
|
function FTNAddressEqual(const A, B: TFTNAddress): boolean;
|
|
```
|
|
|
|
---
|
|
|
|
## Reading
|
|
|
|
```pascal
|
|
function ReadMessage(Index: longint; var Msg: TUniMessage): boolean;
|
|
property MessageCount: longint;
|
|
```
|
|
|
|
Zero-based index. Returns False on EOF or backend failure.
|
|
Message numbers (`Msg.MsgNum`) are backend-assigned and typically
|
|
**don't** match the index (most formats keep a gap-tolerant index).
|
|
Fires `metMessageRead` on success.
|
|
|
|
```pascal
|
|
for i := 0 to base.MessageCount - 1 do
|
|
if base.ReadMessage(i, msg) then
|
|
Handle(msg);
|
|
```
|
|
|
|
---
|
|
|
|
## Writing
|
|
|
|
```pascal
|
|
function WriteMessage(var Msg: TUniMessage): boolean;
|
|
```
|
|
|
|
Appends a new message. Backend assigns `Msg.MsgNum` on success.
|
|
Fires `metMessageWritten`. Raises `EMessageBase` in read-only mode.
|
|
|
|
```pascal
|
|
msg.WhoFrom := 'Sysop';
|
|
msg.WhoTo := 'All';
|
|
msg.Subject := 'Hello';
|
|
msg.DateWritten := Now;
|
|
msg.Attr := MSG_ATTR_LOCAL or MSG_ATTR_ECHO;
|
|
msg.OrigAddr := MakeFTNAddress(1, 100, 1, 0);
|
|
msg.DestAddr := MakeFTNAddress(1, 100, 2, 0);
|
|
msg.Body := 'Hello, world' + #13;
|
|
base.WriteMessage(msg);
|
|
```
|
|
|
|
---
|
|
|
|
## Updating & deleting
|
|
|
|
```pascal
|
|
function UpdateMessage(Index: longint; var Msg: TUniMessage): boolean;
|
|
function DeleteMessage(Index: longint): boolean;
|
|
```
|
|
|
|
Optional operations. Not every backend implements them — the
|
|
default returns False. Attribute-only changes can also be made
|
|
through the native API (`.Native`) by flipping the MSG_ATTR_DELETED
|
|
bit and calling `Pack`; that pattern works on every backend.
|
|
|
|
---
|
|
|
|
## Packing & reindexing
|
|
|
|
```pascal
|
|
function Pack(PurgeAgeDays, PurgeMaxCount: longint;
|
|
Backup: boolean = True): boolean;
|
|
function ReIndex: boolean;
|
|
```
|
|
|
|
`Pack` purges deleted/old messages and rebuilds the physical files.
|
|
`PurgeAgeDays = 0` and `PurgeMaxCount = 0` means "don't purge by
|
|
age or count; just compact deleted messages." Fires
|
|
`metPackStarted` / `metPackComplete`.
|
|
|
|
`ReIndex` rebuilds the index file from the header file (JAM/Hudson).
|
|
|
|
```pascal
|
|
{ no-op pack: only drop anything already marked deleted }
|
|
base.Pack(0, 0, False);
|
|
|
|
{ purge anything older than 90 days, keep at most 500 messages }
|
|
base.Pack(90, 500, True);
|
|
```
|
|
|
|
---
|
|
|
|
## Events & logging
|
|
|
|
Every `TMessageBase` owns a `TMessageEvents` reachable as `.Events`.
|
|
Subscribe with either a log-shaped handler or a full-info handler,
|
|
or both. Multiple hooks can be registered with `AddHook`.
|
|
|
|
```pascal
|
|
TMessageLogHandler =
|
|
procedure(Level: TMsgEventType;
|
|
const Source, Msg: AnsiString) of object;
|
|
|
|
TMessageEventHandler =
|
|
procedure(Sender: TObject;
|
|
const Info: TMessageEventInfo) of object;
|
|
```
|
|
|
|
### Example — console logger
|
|
|
|
```pascal
|
|
type
|
|
TLogger = class
|
|
procedure OnLog(Level: TMsgEventType;
|
|
const Source, Msg: AnsiString);
|
|
end;
|
|
|
|
procedure TLogger.OnLog(Level: TMsgEventType;
|
|
const Source, Msg: AnsiString);
|
|
begin
|
|
WriteLn('[', EventTypeToStr(Level), '] ', Source, ': ', Msg);
|
|
end;
|
|
|
|
var
|
|
log: TLogger;
|
|
begin
|
|
log := TLogger.Create;
|
|
base.Events.OnLog := @log.OnLog;
|
|
...
|
|
```
|
|
|
|
### Event types (`TMsgEventType`)
|
|
|
|
| Group | Values |
|
|
|------------|---------------------------------------------------------------------|
|
|
| Log levels | `metInfo`, `metWarning`, `metError`, `metDebug` |
|
|
| Lifecycle | `metBaseOpened`, `metBaseClosed`, `metBaseCreated` |
|
|
| Messages | `metMessageRead`, `metMessageWritten`, `metMessageDeleted`, `metMessageUpdated` |
|
|
| Locking | `metLockAcquired`, `metLockReleased`, `metLockTimeout`, `metLockFailed` |
|
|
| Pack | `metPackStarted`, `metPackProgress`, `metPackComplete`, `metReindexStarted`, `metReindexComplete` |
|
|
| Packets | `metPacketStart`, `metPacketEnd`, `metPacketMessage`, `metPacketError` |
|
|
|
|
Filter by severity:
|
|
|
|
```pascal
|
|
base.Events.MinLevel := metWarning; { drop info + debug }
|
|
```
|
|
|
|
---
|
|
|
|
## Locking
|
|
|
|
Three layers, all in `ma.lock`:
|
|
|
|
1. **TRTLCriticalSection** per `TMessageBase` instance — serialises
|
|
concurrent Read/Write/Update/Delete calls on the same instance.
|
|
2. **Advisory sentinel** — `fpflock` (Unix), `LockFileEx`
|
|
(Windows), `DosSetFileLocks` (OS/2), exclusive-open (DOS) on a
|
|
per-base `.lck` file. Released and unlinked on close.
|
|
3. **Native share modes** — each backend opens its data streams
|
|
with `fmShareDenyWrite` / `fmShareDenyNone` as the backstop.
|
|
|
|
The sentinel timeout is configurable per instance:
|
|
|
|
```pascal
|
|
base.LockTimeoutMs := 5000; { retry for 5 s, fail otherwise }
|
|
base.LockTimeoutMs := 0; { fail fast — don't wait at all }
|
|
base.LockTimeoutMs := -1; { wait forever }
|
|
```
|
|
|
|
Direct access: `base.Lock` returns the `TMessageLock` for the rare
|
|
case where you want to hold the lock across multiple operations.
|
|
|
|
---
|
|
|
|
## Path helpers
|
|
|
|
```pascal
|
|
function MessageBasePathFor(AFormat: TMsgBaseFormat;
|
|
const AAreaPath, AAreaTag: AnsiString): AnsiString;
|
|
function FindExistingFile(const APath: AnsiString): AnsiString;
|
|
function PathJoin(const ADir, ATail: AnsiString): AnsiString;
|
|
function LockFilePath(AFormat: TMsgBaseFormat;
|
|
const ABasePath, AAreaTag: AnsiString): AnsiString;
|
|
```
|
|
|
|
`MessageBasePathFor` produces the canonical constructor argument
|
|
each backend expects from an area directory + optional tag.
|
|
`FindExistingFile` does case-insensitive resolution (path → UPPER
|
|
→ lower) for Linux hosts where on-disk names may be mixed case.
|
|
|
|
---
|
|
|
|
## Concurrent tossers — TPacketBatch
|
|
|
|
```pascal
|
|
TPacketBatch = class
|
|
constructor Create(const APacketDir: AnsiString; AThreadCount: integer = 4);
|
|
function Run: longint; { returns packet count }
|
|
procedure RequestStop;
|
|
function GetOrCreateBase(AFormat: TMsgBaseFormat;
|
|
const APath: AnsiString): TMessageBase;
|
|
property PacketDir: AnsiString;
|
|
property PacketMask: AnsiString; { default '*.pkt' }
|
|
property ThreadCount: integer;
|
|
property Processor: TPacketProcessor;
|
|
property Events: TMessageEvents;
|
|
end;
|
|
|
|
TPacketProcessor =
|
|
procedure(const APacketPath: AnsiString;
|
|
var Msg: TUniMessage;
|
|
var Stop: boolean) of object;
|
|
```
|
|
|
|
`Run` scans `PacketDir` for `PacketMask`, spawns `ThreadCount`
|
|
workers, and blocks until every packet has been processed.
|
|
Each worker opens one packet at a time (`TPktFile`), hands each
|
|
message to `Processor`. Writes to destination bases should go
|
|
through `GetOrCreateBase` so the batch caches and serialises
|
|
them.
|
|
|
|
### Minimal tosser
|
|
|
|
```pascal
|
|
uses {$IFDEF UNIX}cthreads,{$ENDIF} ma.api, ma.batch, ...;
|
|
|
|
type
|
|
TTosser = class
|
|
Batch: TPacketBatch;
|
|
procedure OnMsg(const Path: AnsiString;
|
|
var Msg: TUniMessage;
|
|
var Stop: boolean);
|
|
end;
|
|
|
|
procedure TTosser.OnMsg(const Path: AnsiString;
|
|
var Msg: TUniMessage;
|
|
var Stop: boolean);
|
|
var
|
|
base: TMessageBase;
|
|
begin
|
|
base := Batch.GetOrCreateBase(mbfJam, '/msg/echo/' + Msg.AreaTag);
|
|
if base <> nil then
|
|
base.WriteMessage(Msg);
|
|
end;
|
|
|
|
var
|
|
t: TTosser;
|
|
begin
|
|
t := TTosser.Create;
|
|
t.Batch := TPacketBatch.Create('/inbound', 4);
|
|
t.Batch.Processor := @t.OnMsg;
|
|
t.Batch.Run;
|
|
...
|
|
end.
|
|
```
|
|
|
|
---
|
|
|
|
## Dropping to a native backend
|
|
|
|
Every `.uni` adapter exposes `Native` for callers who need format-
|
|
specific features the unified API doesn't abstract (JAM subfields,
|
|
Squish UMsgId, EzyCom dual-byte attributes, Wildcat NextMsg
|
|
walking, etc.):
|
|
|
|
```pascal
|
|
var
|
|
adapter: TJamMessageBase;
|
|
hdr: JamHdr;
|
|
nat: TJamMessage;
|
|
begin
|
|
adapter := base as TJamMessageBase;
|
|
adapter.Native.ReadMessage(Index, nat);
|
|
adapter.Native.ReadHeader(nat.HdrOffset, hdr);
|
|
hdr.Attribute := hdr.Attribute or longint($80000000); { mark deleted }
|
|
adapter.Native.UpdateHeader(nat.HdrOffset, hdr);
|
|
adapter.Native.IncModCounter;
|
|
adapter.Native.UpdateHdrInfo;
|
|
end;
|
|
```
|
|
|
|
Native class names:
|
|
|
|
| Format | Native class | Adapter |
|
|
|-----------|-------------------|----------------------|
|
|
| Hudson | `THudsonBase` | `THudsonMessageBase` |
|
|
| JAM | `TJamBase` | `TJamMessageBase` |
|
|
| Squish | `TSquishBase` | `TSquishMessageBase` |
|
|
| FTS-1 MSG | `TMsgDir`+`TMsgFile` | `TMsgMessageBase` |
|
|
| FTN PKT | `TPktFile` | `TPktMessageBase` |
|
|
| PCBoard | `TPCBoardBase` | `TPCBoardMessageBase`|
|
|
| EzyCom | `TEzyComBase` | `TEzyComMessageBase` |
|
|
| GoldBase | `TGoldBase` | `TGoldBaseMessageBase`|
|
|
| Wildcat | `TWildcatBase` | `TWildcatMessageBase`|
|
|
|
|
---
|
|
|
|
## Format cheat-sheet
|
|
|
|
| Format | Base path shape | Caveats |
|
|
|----------|--------------------------|--------------------------------------------|
|
|
| Hudson | directory | 32767-message ceiling (smallint MsgNum) |
|
|
| GoldBase | directory | Widened Hudson (word MsgAttr, 500 boards) |
|
|
| JAM | dir + basename, no ext | Adapter appends `.JHR/.JDT/.JDX/.JLR` |
|
|
| Squish | dir + basename, no ext | Adapter appends `.SQD/.SQI/.SQL` |
|
|
| FTS-1 MSG| directory | Numbered `*.msg` / `*.MSG` (case mixed) |
|
|
| FTN PKT | full packet filename | Stream-only; `WriteMessage` not supported through adapter — use `Native` |
|
|
| PCBoard | dir + basename, no ext | Adapter appends `.MSG/.IDX` |
|
|
| EzyCom | directory + `.Board` | Set `adapter.Board`/`.BBSType` before Open |
|
|
| Wildcat | WC data dir + `.Conference` | Set `adapter.Conference` before Open |
|
|
|
|
---
|
|
|
|
## See also
|
|
|
|
- `docs/architecture.md` — layered design
|
|
- `docs/ftsc-compliance.md` — spec refs
|
|
- `docs/format-notes/` — per-format quirks and dependencies
|
|
- `examples/` — runnable `example_read`, `example_write`,
|
|
`example_tosser`
|
|
- `tests/` — test_read, test_roundtrip, test_lock, test_batch,
|
|
test_wildcat, test_write_existing, test_pack
|