Replaces TUniMessage's 13-field flat record with a strict two-area
model: Body holds only the message text; Attributes holds everything
else (from/to/subject/dates/addresses/MSGID/SEEN-BY/PATH/format-
specific fields) as namespaced key/value pairs.
Why this fix is required NOW: the previous JAM adapter dropped
MSGID, ReplyID, PID, Flags, SEEN-BY and PATH on every Read/Write
through the unified API. A NetReader parity test surfaced it (17/21
pass with 4 kludge failures). All 9 adapters had the same bug. For
tossers and scanners the impact is silent corruption: dropped MSGID
→ dupe storms, dropped PATH → mail loops, dropped SEEN-BY → broken
routing. Three downstream consumers (Fimail's codex-transport branch,
NetReader, future Allfix) had halted integration work pending this
fix. Without it, anyone vendoring fpc-msgbase 0.1 ships with a
known-corrupting adapter.
Design choice: per Ken's call, "message is just the message text;
everything else is an attribute, including from/to/subject/dates."
Same architecture as RFC 822 email (headers + body). Each backend
fills attributes it knows on Read; reads attributes it understands
on Write; ignores unknown attributes silently (RFC 822 X-header
semantics). Forward-compatible -- a new backend (e.g. a planned SQL
message store) just adds its own attribute keys; old backends ignore
them.
Composition is the consumer's job. The library never reassembles
Body + Attributes into kludge-laden display text. A BBS that wants
inline kludges walks Attributes and prepends ^aMSGID etc. to its
own display. A tosser that needs MSGID for dupe detection reads
Attributes.Get('msgid') directly -- no body parsing required.
src/ma.types.pas:
- New TMsgAttribute / TMsgAttributes records with Get/SetValue,
typed accessors (GetInt/GetBool/GetDate/GetAddr), Has/Remove,
iteration. Linear-search lookup, fine for the ~30-50 keys per
message. Switch to hash later if profiling shows need.
- Replaced TUniMessage with the minimal Body + Attributes record.
- New UniAttrBitsToAttributes / UniAttrBitsFromAttributes helpers
to bridge the canonical MSG_ATTR_* cardinal bitset to/from
individual `attr.*` boolean keys.
- {$modeswitch advancedrecords} added so records have methods.
src/ma.api.pas:
- New capabilities API: TStringDynArray return type,
ClassSupportedAttributes (virtual class fn, default empty),
SupportedAttributes (instance sugar), SupportsAttribute (per-key
query). Each backend overrides ClassSupportedAttributes with the
static list of keys it knows. Callers query before setting so a
BBS UI can hide controls the underlying backend has no slot for.
src/formats/ma.fmt.*.uni.pas (all 9):
- Rewrote each XxxToUni and XxxFromUni for the new model. Read
populates Attributes with universal/FTSC/format-specific keys per
the attribute registry (to be published in phase 5). Write reads
attributes back and writes native form.
- JAM walks SubFields[] for SEEN-BY/PATH/TZUTC/TRACE plus passthrough
of unknown subfield IDs as `jam.subfield.<id>` for round-trip
safety. Squish parses CtrlInfo (NUL-separated ^A lines) into
individual attributes, rebuilds on Write. MSG and PKT (which keep
kludges inline in body per FTS-1) parse leading ^A lines and
trailing SEEN-BY/PATH out of the body so TUniMessage.Body is
always plain user text; on Write they reassemble the on-disk form.
- Each backend ships ClassSupportedAttributes with its key list.
src/ma.batch.pas: PktToUni signature updated to (in,out var) form.
tests/* + examples/*: migrated all callers from Msg.WhoFrom (etc.)
to Msg.Attributes.Get('from'). MakeMsg helpers now use SetValue/
SetBool/SetAddr.
Verified: 24/24 tests pass across all 7 test programs (read,
roundtrip, lock, batch, wildcat, write_existing, pack). Wildcat
walks all 7 vendored conferences clean.
Out of scope (next phases):
- docs/attributes-registry.md publishing the full key list with
per-format support matrix
- cross-format round-trip + capabilities-driven copy test
- update architecture.md / PROPOSAL.md to reflect the new model
252 lines
6.7 KiB
ObjectPascal
252 lines
6.7 KiB
ObjectPascal
{
|
|
test_pack.pas - Pack semantics against real base copies.
|
|
|
|
Two JAM tests:
|
|
|
|
1. No-op pack: copy the real 291-message 10thamd to scratch,
|
|
call Pack with no age/count purge, verify count and field
|
|
integrity survive.
|
|
|
|
2. Purge-deleted pack: copy again, mark 5 messages deleted via
|
|
the native class, Pack, verify count dropped by exactly 5
|
|
and that a known-live message still reads correctly.
|
|
}
|
|
|
|
program test_pack;
|
|
|
|
{$mode objfpc}{$H+}
|
|
|
|
uses
|
|
SysUtils,
|
|
testutil,
|
|
ma.types, ma.events, ma.api,
|
|
ma.fmt.jam, ma.fmt.jam.uni,
|
|
ma.fmt.hudson, ma.fmt.hudson.uni;
|
|
|
|
const
|
|
JAM_SRC = '/home/ken/fidonet/msg/jam/10thamd';
|
|
SCRATCH_A = '/tmp/ma_pack_noop';
|
|
SCRATCH_B = '/tmp/ma_pack_purge';
|
|
HUDSON_SCRATCH = '/tmp/ma_pack_hudson';
|
|
HUDSON_SEED = 50;
|
|
|
|
function RunShell(const Cmd: string): integer;
|
|
begin
|
|
Result := ExecuteProcess('/bin/sh', ['-c', Cmd]);
|
|
end;
|
|
|
|
procedure CopyJamBase(const Dest: string);
|
|
var
|
|
dir: string;
|
|
begin
|
|
dir := ExtractFilePath(Dest + '.');
|
|
ForceDirectories(dir);
|
|
RunShell(SysUtils.Format('rm -f %s.*', [Dest]));
|
|
RunShell(SysUtils.Format('cp %s.jhr %s.jhr', [JAM_SRC, Dest]));
|
|
RunShell(SysUtils.Format('cp %s.jdt %s.jdt', [JAM_SRC, Dest]));
|
|
RunShell(SysUtils.Format('cp %s.jdx %s.jdx', [JAM_SRC, Dest]));
|
|
RunShell(SysUtils.Format('cp %s.jlr %s.jlr', [JAM_SRC, Dest]));
|
|
RunShell(SysUtils.Format('chmod u+w %s.*', [Dest]));
|
|
end;
|
|
|
|
procedure TestNoOpPack;
|
|
var
|
|
base: TMessageBase;
|
|
msg: TUniMessage;
|
|
preCount, postCount: longint;
|
|
begin
|
|
TestBegin('JAM: no-op Pack on 291-msg base preserves every message');
|
|
if not FileExists(JAM_SRC + '.jhr') then begin
|
|
WriteLn('SKIP'); exit;
|
|
end;
|
|
CopyJamBase(SCRATCH_A);
|
|
|
|
base := MessageBaseOpen(mbfJam, SCRATCH_A, momReadWrite);
|
|
try
|
|
AssertTrue('Open RW', base.Open);
|
|
preCount := base.MessageCount;
|
|
AssertEquals('Pre-count', 291, preCount);
|
|
AssertTrue('Pack(0,0)', base.Pack(0, 0, False));
|
|
postCount := base.MessageCount;
|
|
AssertEquals('Post-count unchanged', preCount, postCount);
|
|
|
|
{ Spot-check first + last message fields survived. }
|
|
AssertTrue('Read first', base.ReadMessage(0, msg));
|
|
AssertEquals('Msg[0].WhoFrom', 'Robert Wolfe', msg.Attributes.Get('from'));
|
|
AssertTrue('Read last',
|
|
base.ReadMessage(base.MessageCount - 1, msg));
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
TestOK;
|
|
end;
|
|
|
|
procedure TestPurgePack;
|
|
const
|
|
KILL = 5;
|
|
var
|
|
base: TMessageBase;
|
|
adapter: TJamMessageBase;
|
|
nat: TJamMessage;
|
|
hdr: JamHdr;
|
|
msg: TUniMessage;
|
|
preCount, postCount, i: longint;
|
|
begin
|
|
TestBegin(SysUtils.Format(
|
|
'JAM: Pack removes %d deleted messages', [KILL]));
|
|
if not FileExists(JAM_SRC + '.jhr') then begin
|
|
WriteLn('SKIP'); exit;
|
|
end;
|
|
CopyJamBase(SCRATCH_B);
|
|
|
|
{ Phase 1: mark KILL messages deleted via the native class. }
|
|
base := MessageBaseOpen(mbfJam, SCRATCH_B, momReadWrite);
|
|
adapter := base as TJamMessageBase;
|
|
try
|
|
AssertTrue('Open RW', base.Open);
|
|
preCount := base.MessageCount;
|
|
|
|
for i := 0 to KILL - 1 do begin
|
|
AssertTrue('Native.ReadMessage ' + IntToStr(i),
|
|
adapter.Native.ReadMessage(i, nat));
|
|
AssertTrue('ReadHeader',
|
|
adapter.Native.ReadHeader(nat.HdrOffset, hdr));
|
|
hdr.Attribute := hdr.Attribute or longint($80000000);
|
|
adapter.Native.UpdateHeader(nat.HdrOffset, hdr);
|
|
end;
|
|
adapter.Native.IncModCounter;
|
|
adapter.Native.UpdateHdrInfo;
|
|
|
|
{ Phase 2: Pack via unified API. }
|
|
AssertTrue('Pack(0,0)', base.Pack(0, 0, False));
|
|
|
|
postCount := base.MessageCount;
|
|
AssertEquals('Count after purge', preCount - KILL, postCount);
|
|
|
|
{ The first surviving message is what was at index KILL pre-pack. }
|
|
AssertTrue('Read survivor[0]', base.ReadMessage(0, msg));
|
|
AssertFalse('Survivor is not deleted',
|
|
msg.Attributes.GetBool('attr.deleted', false));
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
|
|
{ Reopen fresh and confirm the on-disk count sticks. }
|
|
base := MessageBaseOpen(mbfJam, SCRATCH_B, momReadOnly);
|
|
try
|
|
AssertTrue('Reopen RO', base.Open);
|
|
AssertEquals('Post-pack count after reopen',
|
|
preCount - KILL, base.MessageCount);
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
|
|
TestOK;
|
|
end;
|
|
|
|
procedure SeedHudson(const APath: string; N: longint);
|
|
var
|
|
src, dst: TMessageBase;
|
|
msg: TUniMessage;
|
|
i, copied: longint;
|
|
begin
|
|
if DirectoryExists(APath) then
|
|
RunShell(SysUtils.Format('rm -rf %s', [APath]));
|
|
ForceDirectories(APath);
|
|
dst := MessageBaseOpen(mbfHudson, APath, momCreate);
|
|
try
|
|
if not dst.Open then exit;
|
|
src := MessageBaseOpen(mbfJam, JAM_SRC, momReadOnly);
|
|
try
|
|
if not src.Open then exit;
|
|
copied := 0;
|
|
i := 0;
|
|
while (copied < N) and (i < src.MessageCount) do begin
|
|
if src.ReadMessage(i, msg) then begin
|
|
dst.WriteMessage(msg);
|
|
Inc(copied);
|
|
end;
|
|
Inc(i);
|
|
end;
|
|
finally
|
|
src.Close;
|
|
src.Free;
|
|
end;
|
|
finally
|
|
dst.Close;
|
|
dst.Free;
|
|
end;
|
|
end;
|
|
|
|
procedure TestHudsonPurgePack;
|
|
const
|
|
KILL = 7;
|
|
var
|
|
base: TMessageBase;
|
|
adapter: THudsonMessageBase;
|
|
hdr: HudsonHdrRec;
|
|
i, preCount, postCount: longint;
|
|
msg: TUniMessage;
|
|
begin
|
|
TestBegin(SysUtils.Format(
|
|
'Hudson: seed %d + mark %d deleted + Pack',
|
|
[HUDSON_SEED, KILL]));
|
|
if not FileExists(JAM_SRC + '.jhr') then begin
|
|
WriteLn('SKIP'); exit;
|
|
end;
|
|
|
|
SeedHudson(HUDSON_SCRATCH, HUDSON_SEED);
|
|
|
|
base := MessageBaseOpen(mbfHudson, HUDSON_SCRATCH, momReadWrite);
|
|
adapter := base as THudsonMessageBase;
|
|
try
|
|
AssertTrue('Open RW', base.Open);
|
|
preCount := base.MessageCount;
|
|
AssertEquals('Pre-count', HUDSON_SEED, preCount);
|
|
|
|
for i := 0 to KILL - 1 do begin
|
|
AssertTrue('Native.ReadHeader ' + IntToStr(i),
|
|
adapter.Native.ReadHeader(i, hdr));
|
|
hdr.MsgAttr := hdr.MsgAttr or $01; { HUDSON_DELETED }
|
|
adapter.Native.UpdateHeader(i, hdr);
|
|
end;
|
|
|
|
AssertTrue('Pack(0,0)', base.Pack(0, 0, False));
|
|
|
|
postCount := base.MessageCount;
|
|
AssertEquals('Count after Pack', preCount - KILL, postCount);
|
|
|
|
AssertTrue('Read survivor[0]', base.ReadMessage(0, msg));
|
|
AssertFalse('Survivor not deleted',
|
|
msg.Attributes.GetBool('attr.deleted', false));
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
|
|
{ Reopen and confirm. }
|
|
base := MessageBaseOpen(mbfHudson, HUDSON_SCRATCH, momReadOnly);
|
|
try
|
|
AssertTrue('Reopen RO', base.Open);
|
|
AssertEquals('Post-pack count after reopen',
|
|
HUDSON_SEED - KILL, base.MessageCount);
|
|
finally
|
|
base.Close;
|
|
base.Free;
|
|
end;
|
|
|
|
TestOK;
|
|
end;
|
|
|
|
begin
|
|
WriteLn('fpc-msgbase: Pack tests against real base data');
|
|
WriteLn;
|
|
TestNoOpPack;
|
|
TestPurgePack;
|
|
TestHudsonPurgePack;
|
|
Halt(TestsSummary);
|
|
end.
|