Invoice Sandbox · Ciru Inference Lab

Completed · strict score 100%

HY3 IQ1_M (Q1)

AngelSlim/Hy3-GGUF IQ1_M MTP delivered the dashboard and matched every customer total, every invoice total, both credit memos, and all accounting-trap checks exactly.

Net exact18 / 18

Invoice totals105 / 105

Wall time25m 44s

Tool calls36

PP77.65 t/s

TG9.68 t/s

Open model dashboard Read audit

Completed · dashboard produced

Ternary Bonsai 27B

Ternary Bonsai 27B Q2_0 · LAN API found every valid invoice and extracted all 105 invoice totals exactly. It systematically emitted malformed IDC… customer IDs and missed both credit amounts.

Net exact0 / 18

Invoice totals105 / 105

Wall time10m 37s

Tool calls23

PP1,805.55 t/s

TG53.23 t/s

Open model dashboard Read audit

Completed · dashboard produced

HY3 iFP2

Hy3 Chadrock ROCmFPX-iFP2 MTP completed the artifact, found every valid invoice document, and passed five of six trap families. Its customer-spend math was not reliable.

Net exact1 / 18

Valid docs105 / 105

Wall time32m 17s

Tool calls29

PP161.91 t/s

TG12.98 t/s

Open model dashboard Read audit

Failed · no dashboard produced

Step3.7 FP3

Step 3.7 Flash ROCmFPX FP3 QualityPlus MTP wrote a partial batch parser, but it resolved zero customer IDs and entered a 29-call repetition loop on one sample PDF.

DashboardNone

Parsed rows112

Wall time4m 58s

Tool calls48

PP179.30 t/s

TG34.29 t/s

Open failure report See method

Strict result audit

Scores come from the repository answer key after each run. “Exact” means equal to the cent for every customer or invoice; document detection is scored independently from money extraction.

Measure	HY3 IQ1_M	Bonsai 27B	HY3 iFP2	Step3.7 FP3
HTML dashboard delivered	Yes	Yes	Yes	No
Valid document precision / recall	100% / 100%	100% / 100%	100% / 100%	Not scorable
Customer invoice counts exact	18 / 18	0 / 18	18 / 18	Not scorable
Customer net totals exact	18 / 18 (100%)	0 / 18 (0%)	1 / 18 (5.6%)	Not scorable
Invoice totals exact	105 / 105 (100%)	105 / 105 (100%)	9 / 105 (8.6%)	Not scorable
Trap families passed	6 / 6	5 / 6	5 / 6	No final artifact
Expected net spend	$3,566,177.15	$3,566,177.15	$3,566,177.15	$3,566,177.15
Reported net spend	$3,566,177.15	$3,568,582.15	$3,413,274.81	Not reported

HY3 IQ1_M result. The Q1 run recovered from two parser/debugging errors, then produced exact document selection, exact invoice totals, exact per-customer aggregation, correct negative credit handling, and a zero-cent global error.

Bonsai result. The model's regex captured IDC001 through IDC018 rather than the exact IDs C001 through C018, so strict keyed scoring is 0/18. Removing only that systematic prefix as a diagnostic yields 16/18 exact net totals and 18/18 exact gross totals; the two remaining misses are the $1,480 and $925 credit memos. It also made one exploratory pdfminer call, contrary to the prescribed parser path, before returning to the required pdf-parse implementation.

HY3 root cause. Its case-insensitive /Total/ regex matched the “Subtotal” line first for most invoices. It also stored credit memos as negative numbers, then subtracted those negatives—adding $2,405 instead of reducing spend. The filtering logic was much stronger than the arithmetic.

Controlled setup

Identical workspaces

All four fresh copies contained 148 files and 112 PDFs. Their pre-run SHA-256 manifests matched byte for byte. The grading keys were outside the agent-visible directories.

Same Pi surface

Pi 0.78.0 exposed only read, bash, edit, write, grep, find, ls. Skills, extensions, prompt templates, and context files were disabled.

Required PDF path

pdf-parse 1.1.1 was installed at the exact requested CommonJS path and smoke-tested with await pdf(buffer, { max: 1 }).

Model context

Both HY3 runs used 65,536-token profiles. Step3.7 and Bonsai used 262,144-token model contexts. Bonsai was served through a LAN OpenAI-compatible API; no remote filesystem or benchmark data was exposed by the endpoint.

Serving profiles

The Pi harness did not override model sampling. Bonsai used its endpoint-native profile: temperature 1.0, top-k 20, top-p 0.95, min-p 0.05, and a random seed. The other runs likewise retained their recorded serving-profile settings.

Additional Step3.7 control. A preserved 64k attempt also failed: 184 tool calls, a 65,610-token request against a 65,536-token slot, automatic compaction, and the same unrecovered repeated-tool behavior. It created no files and is excluded from the scored row above.

Runtime telemetry

Run	Context	Server prompt tokens	Server output tokens	Prompt / prefill	Decode	Turns	Tool errors
HY3 IQ1_M (Q1)	65,536	27,001	11,547	77.65 tok/s	9.68 tok/s	29	2
Ternary Bonsai 27B Q2_0	262,144	272,858	25,690	1,805.55 tok/s	53.23 tok/s	21	1
HY3 iFP2	65,536	192,221	9,685	161.91 tok/s	12.98 tok/s	22	0
Step3.7 FP3	262,144	25,839	4,960	179.30 tok/s	34.29 tok/s	48	1

PP and TG are request-weighted server averages from the llama.cpp metrics counters for each isolated run. Pi’s cache-read accounting is omitted from the primary table because it is cumulative context reuse, not newly evaluated model tokens.

Exact agent prompt

Show the prompt given to all four runs

I want you to organize the invoices in this workspace, parse the invoice PDFs.

Use const pdf = require("/tmp/node_modules/pdf-parse")

Call await pdf(buffer, { max: 1 })

Do not use pdf.default

Do not use PDFParser

Do not manually decompress streams

Parse page 1 only; appendix pages are noise

Regex Total $..., Subtotal $..., Tax $..., Status..., Customer ID...

Exclude statements, voids, duplicate scans, and superseded originals

Subtract credit memos

Sort by customer

Make a comprehensive HTML dashboard where I can track each customer's spending

Prompt SHA-256: 0595615047a4819809b394c8de59eb2b252d2a5e573bb16d61d17dacf2c92479

Reproduce and inspect

The fixture, generator, answer key, and scorer are public. This page uses benchmark commit b8cb58f; the structured public result is available as results.json.

Open benchmark repo Read the original finance benchmark article