From 426fb677d5d22dcfb888afa0749094f1839f1bee Mon Sep 17 00:00:00 2001 From: Ken Johnson Date: Tue, 14 Apr 2026 10:40:56 -0700 Subject: [PATCH] Initial scaffold: layout, fpc.cfg, README, architecture doc --- .gitignore | 25 +++++++++++++ README.md | 69 ++++++++++++++++++++++++++++++++++ docs/architecture.md | 88 ++++++++++++++++++++++++++++++++++++++++++++ fpc.cfg | 49 ++++++++++++++++++++++++ 4 files changed, 231 insertions(+) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 docs/architecture.md create mode 100644 fpc.cfg diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..18edecb --- /dev/null +++ b/.gitignore @@ -0,0 +1,25 @@ +# FPC build artifacts +*.o +*.ppu +*.compiled +*.or +*.a +*.lib +*.dcu + +# Lazarus +*.lps +backup/ +lib/ +units/ + +# Build output trees +exe/ +units/ + +# Editor / OS junk +.vscode/ +.idea/ +*.swp +*~ +.DS_Store diff --git a/README.md b/README.md new file mode 100644 index 0000000..f4bf2df --- /dev/null +++ b/README.md @@ -0,0 +1,69 @@ +# message_api + +A unified Free Pascal library for reading and writing classic BBS message bases. + +Wraps proven message-format handlers from Allfix behind one polymorphic API +(`TMessageBase`) so BBS software, mail tossers, message editors, and utilities +can target a single interface regardless of the underlying format on disk. + +## Supported formats + +| Format | Files | Backend unit | +|-----------|--------------------------------------|---------------------------| +| Hudson | MSGINFO/IDX/HDR/TXT/TOIDX.BBS | `ma.fmt.hudson.pas` | +| JAM | `*.JHR` `*.JDT` `*.JDX` `*.JLR` | `ma.fmt.jam.pas` | +| Squish | `*.SQD` `*.SQI` `*.SQL` | `ma.fmt.squish.pas` | +| FTS-1 MSG | numbered `*.MSG` per directory | `ma.fmt.msg.pas` | +| FTN PKT | `*.pkt` (Type-2 / 2+ / 2.2) | `ma.fmt.pkt.pas` | +| PCBoard | `*.MSG` + `*.IDX` | `ma.fmt.pcboard.pas` | +| EzyCom | `MH#####.BBS` / `MT#####.BBS` | `ma.fmt.ezycom.pas` | +| GoldBase | MSGINFO/IDX/HDR/TXT/TOIDX.DAT | `ma.fmt.goldbase.pas` | +| Wildcat 4 | WC SDK databases | `ma.fmt.wildcat.pas` | + +## Features + +- One `TMessageBase` abstract class — read, write, pack, reindex through the + same methods regardless of format. +- Layered locking: in-process `TRTLCriticalSection` + cross-process advisory + lock (`fpflock` on Unix, `LockFileEx` on Windows, `.LCK` sentinel fallback) + + the existing `fmShareDenyWrite` / `fmShareDenyNone` share modes. +- Event hooks for logging, progress, and status reporting. +- `TPacketBatch` worker pool for tossers that need to process many `.pkt` + files concurrently while serialising writes per destination base. +- Path / filename auto-derivation per format from a base directory plus + optional area tag. + +## Building + +Native Linux: + +```sh +fpc -Fusrc -Fusrc/formats examples/example_read.pas +``` + +Lazarus package: + +```sh +lazbuild message_api.lpk +``` + +The repo includes a `fpc.cfg` template mirroring the Allfix multi-target +build (`i386-go32v2`, `i386-win32`, `i386-linux`, `i386-os2`). + +## Layout + +``` +src/ ma.api, ma.types, ma.events, ma.lock, ma.paths, ma.batch +src/formats/ ma.fmt..pas — one per supported format +docs/ architecture, locking semantics, format notes +tests/ FPCUnit tests, sample data +examples/ small CLI programs that double as smoke tests +``` + +Unit-name convention follows the Fimail style: `ma...pas` +(`ma` = message_api). All units use `{$mode objfpc}{$H+}`. + +## Status + +Early development. APIs may move until 1.0. Format backends are ports of the +working Allfix code so behaviour matches what Allfix produces today. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..98db6e7 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,88 @@ +# message_api — architecture + +## Layers + +``` + ┌──────────────────────────────────────────────────┐ + │ Caller (BBS, tosser, editor, importer, …) │ + └──────────────────────────────────────────────────┘ + │ + ▼ + ┌──────────────────────────────────────────────────┐ + │ ma.api (TMessageBase, factory, TUniMessage) │ + ├──────────────────────────────────────────────────┤ + │ ma.events ma.lock ma.paths │ + │ ma.batch (concurrent tosser helper) │ + ├──────────────────────────────────────────────────┤ + │ Format backends — one .pas per format │ + │ ma.fmt.hudson ma.fmt.jam ma.fmt.squish │ + │ ma.fmt.msg ma.fmt.pkt ma.fmt.pcboard │ + │ ma.fmt.ezycom ma.fmt.goldbase ma.fmt.wildcat │ + ├──────────────────────────────────────────────────┤ + │ RTL: TFileStream, BaseUnix/Windows for locking │ + └──────────────────────────────────────────────────┘ +``` + +## Polymorphism + +Every backend descends from `TMessageBase` and implements the abstract +`DoOpen`, `DoClose`, `DoMessageCount`, `DoReadMessage`, `DoWriteMessage` +contract. Callers can either: + +1. Use the unified API — `MessageBaseOpen(format, path, mode)` returns a + `TMessageBase`. Read/write through `TUniMessage`. Format-agnostic. +2. Drop down to format-specific class methods (e.g. `TJamBase.IncModCounter`, + `TSquishBase.SqHashName`) when they need behaviour the unified API cannot + express. Each backend keeps its rich API public. + +## TUniMessage + +A single message record covering every backend. Format-specific bit fields +(Hudson byte attr, JAM 32-bit attr, Squish attr, MSG word attr, PCB status, +EzyCom dual byte) are mapped into a canonical `Attr: cardinal` bit set +defined in `ma.types` (`MSG_ATTR_PRIVATE`, `MSG_ATTR_LOCAL`, +`MSG_ATTR_RECEIVED`, …). Conversion helpers expose the original encoding +when needed. + +Dates land in `TDateTime` regardless of how the backend stored them +(Hudson `MM-DD-YY` strings with 1950 pivot, Squish FTS-0001 strings, JAM +Unix timestamps, PCBoard / EzyCom DOS PackTime). + +The body is an `AnsiString` with kludge lines intact and CR-separated +(matches what Allfix already produces). + +## Locking + +Three layers, applied in order on every `Open`: + +1. **In-process** — `TRTLCriticalSection` per `TMessageBase` instance. +2. **Cross-process** — advisory lock on a sentinel file + (`.lck` or, for Squish, `.SQL` so we coexist with other + Squish-aware tools). `fpflock(LOCK_EX|LOCK_SH)` on Unix, + `LockFileEx` on Windows. Retry with backoff up to a configurable + timeout (default 30s). Lock acquire/release fires events. +3. **OS share modes** — `fmShareDenyWrite` for writers, + `fmShareDenyNone` for readers. Same as the existing Allfix code. + +## Events + +`TMessageEvents` lets callers subscribe one or more handlers to receive +`metBaseOpened`, `metMessageRead`, `metMessageWritten`, `metLockAcquired`, +`metPackProgress`, etc. Internally the dispatcher serialises calls so +handlers do not need to be reentrant. + +## Concurrent tossers + +`TPacketBatch` owns a queue of `.pkt` paths and a worker thread pool. +Each worker opens its packet, reads messages, hands each to the +caller-provided processor. The batch caches one `TMessageBase` per +destination area so writes serialise through layer-1 locking; layer-2 +keeps separate processes (e.g. an editor) safe at the same time. + +## Behavioural fidelity + +Every format backend is a port of the proven Allfix implementation. Tests +read/write the same sample bases that Allfix tests use; outputs are +diffed against captures produced by the existing `msgutil` / `domsg` +code paths. Anything that works in Allfix today must keep working through +this library. diff --git a/fpc.cfg b/fpc.cfg new file mode 100644 index 0000000..7eeb48c --- /dev/null +++ b/fpc.cfg @@ -0,0 +1,49 @@ +# message_api fpc.cfg — mirrors the Allfix template. +# +# Pulls compiler-bundled RTL paths from /opt/fpcup, then adds the library +# source roots so callers can build with a single -Fucfg=fpc.cfg. + +#IFNDEF FPC_CROSSCOMPILING + #IFDEF LINUX + #INCLUDE /opt/fpcup/fpc/bin/x86_64-linux/fpc.cfg + #ENDIF LINUX + #IFDEF OS2 + #INCLUDE P:\fpc\bin\os2\fpc.cfg + #ENDIF OS2 + #IFDEF DOS + -Fuc:\pp/units/i386-$FPCTARGET/ + -Fuc:\pp/units/i386-$FPCTARGET/* + -Fuc:\pp/units/i386-$FPCTARGET/rtl + #ENDIF DOS +#ELSE + -Fu/opt/fpcup/fpc/units/$FPCTARGET/rtl + -Fu/opt/fpcup/fpc/units/$FPCTARGET/* +#ENDIF + +# Allow GOTO/LABEL, inline, C-operators (matches Allfix dialect) +-Sgic + +# Library source roots +-Fusrc +-Fusrc/formats + +# Output trees per target +-FE./exe/$FPCTARGET +-FU./units/$FPCTARGET + +# Cross-compile binutils prefixes (mirrors Allfix) +#IFDEF FPC_CROSSCOMPILING +#IFDEF CPUI386 +#IFDEF OS2 + -FD/opt/fpcup/cross/i386-pc-os2-emx + -XPi386-emx- + -k-e + -k0x10000 +#ELSE + -FD/opt/fpcup/cross/bin + -XP$FPCTARGET- +#ENDIF OS2 +#ENDIF CPUI386 +#ENDIF FPC_CROSSCOMPILING + +-Xs