Benchmark

Evaluated 2026-11-16 against 14 synthetic documents: 12 clean and 2 deliberately corrupted. Pipeline: GLM-OCR local, structuring with Gemini 3.1 Flash Lite, validation with the same checksums as this web demo. Honesty before shine: the weaknesses are listed below.

Field accuracy, 12 clean documents

FieldAccuracynFailures
invoice_number83.3%10/122x prefix RG truncated
invoice_date83.3%10/12Receipts print a date; ground truth: null
due_date100%12/12-
supplier_name100%12/12-
supplier_city83.3%10/12Receipts print the city; ground truth: null
supplier_country83.3%10/12Country not printed on receipts, inferred by the model
supplier_vat_id91.7%11/121x digit drop in OCR (rechnung_06)
buyer_name91.7%11/12Right column in the two-column layout was skipped
currency100%12/12-
net_amount100%12/12-
vat_rate100%12/12-
vat_amount100%12/12-
gross_amount100%12/12-
iban83.3%10/122x digit drop on 22-digit IBANs
payment_terms100%12/12-
Total93.3%168/180Target: at least 90%. Reached.

Corrupted documents caught

rechnung_13_corrupt_iban

FLAGGED

Account number twisted by one digit. Check: IBAN checksum (mod-97).

rechnung_14_corrupt_totals

FLAGGED

Gross deviates by 47.30 from net + VAT. Check: totals consistency.

Result: 2 of 2 corrupted documents reliably flagged.

Per document

DocumentFieldsAccuracyValidation
invoice_11_thames_consulting13/1587%FLAGGED
invoice_12_copenhagen_trading15/15100%OK
kassenbon_09_sparmarkt12/1580%OK
kassenbon_10_baeckerei12/1580%OK
rechnung_01_musterhandel15/15100%OK
rechnung_02_bergwerk_ag14/1593%OK
rechnung_03_zuliefer_tech15/15100%OK
rechnung_04_nordlicht_studio15/15100%OK
rechnung_05_altbau_handwerk15/15100%OK
rechnung_06_rhein_gastro13/1587%OK
rechnung_07_elbtal_logistik15/15100%OK
rechnung_08_hansa_media14/1593%FLAGGED

Latencies

30.6 s

OCR per document, mean (10.9 to 44.3 s)

4.5 s

Structuring per document, mean

23

OCR tokens/s, mean

493 s

Total runtime, 14 documents sequential

Known weaknesses

12 of 180 fields not exactly right: 6 real OCR limitations, 6 evaluation artifacts of the strict scoring. Both categories are on the table here.

IBAN digit drops (2 documents)

On long digit runs the OCR model occasionally loses a character (21 instead of 22 digits). Neither the text pass nor the image escalation recovered it. Planned fix in v1.1: a mod-97-aware repair layer, or re-rendering the IBAN region at higher resolution before a document is exported as flagged.

Category: OCR limitation

Invoice-number prefixes cut off (2 documents)

“RG 88123” was read as “88123”. The printed prefix varies by layout; a prefix-preserving rule is planned.

Category: OCR limitation

VAT ID digit drop with wrong format (1 document)

rechnung_06 read “DE13665976” (8 digits after DE). The old check let that pass as a format exception. The web check now fails every DE-prefixed value without exactly 9 digits: the same rule this validation layer runs.

Category: OCR limitation

Buyer block skipped (1 document)

In the two-column layout, the right-hand “Bill to” block was skipped entirely; the image escalation did not recover it.

Category: OCR limitation

Ground-truth strictness on receipts (6 counts)

Receipts print a date and a city; ground truth was set to null there. Extraction returned the printed values and only failed against the strict evaluation. No real read errors.

Category: Evaluation artifact

Note on the VAT ID test vector

The original specification vector DE218752815 does not survive ISO 7064 mod 11,10; the corrected vector DE218752819 is valid. This validation layer implements the verified-correct algorithm, identical to the reference library python-stdnum.

The numbers come from the local NUC pipeline (v1). This web demo uses the same validation layer but extracts directly from the image. Check it in the demo.