Benchmark
Evaluated 2026-11-16 against 14 synthetic documents: 12 clean and 2 deliberately corrupted. Pipeline: GLM-OCR local, structuring with Gemini 3.1 Flash Lite, validation with the same checksums as this web demo. Honesty before shine: the weaknesses are listed below.
Field accuracy, 12 clean documents
| Field | Accuracy | n | Failures |
|---|---|---|---|
| invoice_number | 83.3% | 10/12 | 2x prefix RG truncated |
| invoice_date | 83.3% | 10/12 | Receipts print a date; ground truth: null |
| due_date | 100% | 12/12 | - |
| supplier_name | 100% | 12/12 | - |
| supplier_city | 83.3% | 10/12 | Receipts print the city; ground truth: null |
| supplier_country | 83.3% | 10/12 | Country not printed on receipts, inferred by the model |
| supplier_vat_id | 91.7% | 11/12 | 1x digit drop in OCR (rechnung_06) |
| buyer_name | 91.7% | 11/12 | Right column in the two-column layout was skipped |
| currency | 100% | 12/12 | - |
| net_amount | 100% | 12/12 | - |
| vat_rate | 100% | 12/12 | - |
| vat_amount | 100% | 12/12 | - |
| gross_amount | 100% | 12/12 | - |
| iban | 83.3% | 10/12 | 2x digit drop on 22-digit IBANs |
| payment_terms | 100% | 12/12 | - |
| Total | 93.3% | 168/180 | Target: at least 90%. Reached. |
Corrupted documents caught
rechnung_13_corrupt_iban
FLAGGED
Account number twisted by one digit. Check: IBAN checksum (mod-97).
rechnung_14_corrupt_totals
FLAGGED
Gross deviates by 47.30 from net + VAT. Check: totals consistency.
Result: 2 of 2 corrupted documents reliably flagged.
Per document
| Document | Fields | Accuracy | Validation |
|---|---|---|---|
| invoice_11_thames_consulting | 13/15 | 87% | FLAGGED |
| invoice_12_copenhagen_trading | 15/15 | 100% | OK |
| kassenbon_09_sparmarkt | 12/15 | 80% | OK |
| kassenbon_10_baeckerei | 12/15 | 80% | OK |
| rechnung_01_musterhandel | 15/15 | 100% | OK |
| rechnung_02_bergwerk_ag | 14/15 | 93% | OK |
| rechnung_03_zuliefer_tech | 15/15 | 100% | OK |
| rechnung_04_nordlicht_studio | 15/15 | 100% | OK |
| rechnung_05_altbau_handwerk | 15/15 | 100% | OK |
| rechnung_06_rhein_gastro | 13/15 | 87% | OK |
| rechnung_07_elbtal_logistik | 15/15 | 100% | OK |
| rechnung_08_hansa_media | 14/15 | 93% | FLAGGED |
Latencies
30.6 s
OCR per document, mean (10.9 to 44.3 s)
4.5 s
Structuring per document, mean
23
OCR tokens/s, mean
493 s
Total runtime, 14 documents sequential
Known weaknesses
12 of 180 fields not exactly right: 6 real OCR limitations, 6 evaluation artifacts of the strict scoring. Both categories are on the table here.
IBAN digit drops (2 documents)
On long digit runs the OCR model occasionally loses a character (21 instead of 22 digits). Neither the text pass nor the image escalation recovered it. Planned fix in v1.1: a mod-97-aware repair layer, or re-rendering the IBAN region at higher resolution before a document is exported as flagged.
Category: OCR limitation
Invoice-number prefixes cut off (2 documents)
“RG 88123” was read as “88123”. The printed prefix varies by layout; a prefix-preserving rule is planned.
Category: OCR limitation
VAT ID digit drop with wrong format (1 document)
rechnung_06 read “DE13665976” (8 digits after DE). The old check let that pass as a format exception. The web check now fails every DE-prefixed value without exactly 9 digits: the same rule this validation layer runs.
Category: OCR limitation
Buyer block skipped (1 document)
In the two-column layout, the right-hand “Bill to” block was skipped entirely; the image escalation did not recover it.
Category: OCR limitation
Ground-truth strictness on receipts (6 counts)
Receipts print a date and a city; ground truth was set to null there. Extraction returned the printed values and only failed against the strict evaluation. No real read errors.
Category: Evaluation artifact
Note on the VAT ID test vector
The original specification vector DE218752815 does not survive ISO 7064 mod 11,10; the corrected vector DE218752819 is valid. This validation layer implements the verified-correct algorithm, identical to the reference library python-stdnum.
The numbers come from the local NUC pipeline (v1). This web demo uses the same validation layer but extracts directly from the image. Check it in the demo.