AI in ledger matching: what it adds, measured

What I tested on

Two sets with known right answers. Every model got the same rows in the same order, the same prompt, and the same 40 rows per call.

Real traces agree with my replay. In production the live matcher saved 113 rows (users kept all 113), put 89 right first suggestions and 7 wrong ones, and sent 22 rows to Suspense. My replay of today's code gives 119, 85, 7 and 20. So production gives AI only about 30 rows to improve; the book set carries most of the measurement.

Two histories: fresh is a company's first import (no earlier choices). Prior means each statement comes after the earlier ones were done and exported, as in steady use. On the book set the matcher saves 340 rows fresh and 608 with history, and sends 571 and 510 to Suspense. Its green suggestions were right 267 of 270 times. 1,011 labelled rows go to a ledger the company had not used before in these statements.

Which model

"Blind" here: the model sees the party's ledger list (name and group) and the entries (date, paid or received, amount, narration, reference), and returns up to 3 ledgers, each with its probability, or nothing. It does not see the matcher's answer or any earlier choice.

* Times through a command-line tool include its start-up (4.5–15 s). CLI costs use API list prices (October 2026) on the tokens the tool reported, less the tool's own prompt (about 19,200 input tokens per call for Codex, 13,800 for agy Gemini, 11,500 for agy GPT-OSS); GPT-OSS is priced at a typical host's $0.15/$0.60. Claude costs are measured, including prompt-cache writes. Gemini 3.8 Flash prices double on 01/01/2027. Opus 5.5 cannot turn thinking off; low effort is its lightest setting. "Gave no ledger" is the share of rows where the model said nothing points to a ledger.

Three things stand out. First, the big models are close: on the book set every Claude, GPT and Gemini model is within 6 points (64–70%). Second, they differ a lot in honesty: when GPT-6.1 Sol said 90% or more it was wrong 19.9% of the time, Gemini 3.8 Flash 18.2%, the Claude models 8–15% across all their settings. Third, thinking does not help this task: Haiku gained 1.5 points, Sonnet lost 0.6, Opus at medium effort lost 0.5, and each got slower. Gemini 3.8 Flash at medium thinking was worse still: 43 of its 69 calls returned no answer, 40 of them because thinking used up the whole reply. Its row is scored only on the rows it answered.

On the rows that matter, the 571 rows the matcher sends to Suspense on a first import, the models spread out more (blind prompt, right first ledger): Sonnet 5.5 44.1%, Opus 5.5 42.7%, Gemini 3.8 Flash 40.3%, GPT-6 Astra 38.7%, GPT-6.1 Sol 37.7%, Haiku 5.5 36.3%, GPT-6 Luna 30.6%, GPT-OSS 120B 21.5%. These rows name no known payee, so the model must read a cut or glued name, or a kind of payment, against the ledger list.

The open models are not close. GPT-OSS 120B (open weights, run through agy) got 50% and often gave no ledger. I did not get to run a local open model: the PC's GPU and most of its memory were in use during the test. GPT-OSS 120B above shows where open models stand on this task. A local run would also be far too slow on our 4-core server.

What the model is given matters more than which model

Matcher clues (each row's suggestion and how often that kind of clue was right) do not raise top-1, but they cut confident wrong answers by about 40%. Earlier choices (up to 250 of the company's exported rows with the ledger chosen) add about 8 points and cut confident errors further. They teach the company's booking habits, which no model can guess: one company posts bank SMS charges to the bank's creditor ledger, not to Bank Charges; another posts a named worker's NEFT to Labour Wages, not to the worker's ledger.

One row per call, or the whole statement

On the same 531 sampled rows, one row per call was about as accurate as 40 rows per call: Haiku 72.7% against 74.0%, Sonnet 79.1% against 78.2%. One row per call answers in about 1.2 s with the ledger list cached, but costs about 2.5 times more ($0.12 per 1,000 rows with Haiku). Calls of about 10 rows, sent in parallel, keep the speed and most of the saving.

Where AI should go in the pipeline

For each place I combined the matcher's real output with the model's answer for every labelled row, and counted what the user would face. Decisions are rows the user must decide: not saved right and not pre-selected right. Wrong suggestions are rows pre-selected with a wrong ledger (the user must notice and change them). Wrong saves are saved with a wrong ledger and no user action.

What the numbers say:

• AI on Suspense rows gives most of the gain. Adding unsure suggestions gives a little more: when the matcher was red or yellow and the AI chose another ledger with at least 50%, the AI was right 28 of 29 times (Sonnet, first import).
• AI only re-ordering the matcher's candidates gains almost nothing. The matcher rarely ranks the right ledger second; mostly it has no candidate at all.
• AI checking exact saves has nothing to catch: exact saves were right 340 of 340 on first imports and wrong once in 608 with history. The AI disputed at most 2 of them, all right ones, and did not dispute the wrong one. Keep exact saves as they are.
• Only Suspense rows to the AI (a smaller, cheaper batch) reads them as well as the whole statement: Haiku 39% against 36% on those rows.
• AI reads the payee and our code finds the ledger (the ledger list never leaves the server): weak. It rescued 37 of 571 Suspense rows, against about 250 when the AI sees the ledger list. Adding the AI's kind word ("bank charges") even caused 89 wrong exact saves.

All other models, blind prompt, first import, AI on Suspense and unsure rows:

When may AI save a choice by itself?

Each rule below adds saves on top of today's exact saves. Counts are on the book set (1,611 rows).

Safe rule: save only when the AI and the matcher's green suggestion agree. The matcher's green suggestions alone were right 267 of 270 times; the AI's agreement removes most of the few wrong ones. Pooled over the six runs, the rule saved 1,093 extra rows with 3 wrong. That passes the 99% bar from the 12/10 audit only just (Wilson lower bound 99.0–99.3% per run), so it needs the replay gate on every change. Unsafe rules: agreement at lower AI confidence, or the AI alone at "95%".

How sure is the AI, really?

Without history the stated percentage runs high: "90–100%" is right about 90% of the time, and below 90 the number means little. With earlier choices, "95–100%" is right almost always, because it then repeats a choice the user made. So the colour mark must come from measured situations, as the matcher's does today (ledger_confidence_table.go): for example, AI agrees with a green matcher suggestion (right about 100%), AI disagrees with an unsure matcher suggestion (about 90%), AI on a Suspense row at 70–94% (about 75–80%, yellow), AI on a Suspense row below 70% (about 40%, red).

Speed against a 10–12 s budget

Measured through the direct API, one statement at a time, as the server would do it. The matcher runs first (instant). The rows it did not save go to the model in calls of 10 rows, all at once; the statement is ready when the slowest call returns. Model prompt: ledger list + matcher clues.

All three fit the budget with room to spare. No statement took more than 8 s; Sonnet's slowest was 5.2 s and its median 3.6 s. An empty prompt cache (a party's first import of the day) made no difference to time. With 10 rows per call, up to 17 calls ran at once for the largest statement, with no rate-limit retries. Real statements also carry rows the books did not label, so they need a few more calls at once; the time of a statement is the time of its slowest call, not the number of calls, so this adds little. Earlier choices in the prompt (up to 250 rows) make each call longer to read but are cached with the ledger list.

The review steps are fixed when review starts, so the AI's answers must be ready by then. The AI needs only the party's ledger list, not the bank ledger, so it can start the moment the parse ends, while the user confirms the bank. A warm-up call when the user opens the party puts the ledger list in the prompt cache.

Cost

Measured tokens, API list prices (October 2026), prompt-cache writes included. A typical row costs about 50 output tokens (ledger number, probability, payee and kind) and a share of the ledger list.

• Claude Haiku 5.5: $0.047 per 1,000 rows (6.7 s per 40 rows).
• Claude Sonnet 5.5: $0.98 per 1,000 rows (10.5 s per 40 rows). With earlier choices in the prompt, about $1.2.
• The prompt cache matters. With 10 rows per call, Sonnet cost $0.84 per 1,000 rows when the ledger list was cached, and $2.71 when every parallel call had to write it first (Haiku $0.04 and $0.14). One small warm-up call when the user opens the party, before the parallel calls, keeps the cheaper price.
• Claude Opus 5.5: $1.89 per 1,000 rows (15.0 s per 40 rows).
• Only rows the matcher did not save go to the AI: about 80% of rows on a first import and 60% with history. A 100-row statement then costs about 6–8 US cents with Sonnet (₹5–7), and under half a cent with Haiku.

The mistakes, and which checks catch them

• Booking habit (the largest group). The AI picks the obvious ledger (Bank Charges, Electricity, the person's own ledger) and the company books it elsewhere (the bank's creditor ledger, the power company's own ledger, Labour Wages). Caught by: earlier exported choices in the prompt, and the matcher's learnt keys, which win over the AI.
• Guess with no name in the narration (UPI or IMPS to a person, a card payment, an insurance premium). The AI fills the gap with a plausible ledger. Caught by: a red mark for AI answers on Suspense rows below 70%, measured per situation; such rows were right only about 40% of the time.
• Two ledgers for one party (a debtor and a creditor copy, a person and their firm, two branches). Caught by: the paid/received direction in the prompt, and the matcher's direction-scoped keys once one choice is exported.
• Invented ledgers were almost absent: 2 of 38,944 ledger numbers in all answers were not in the list. Our code drops any such number.
• The same question, a different answer. Run twice on every row, Sonnet chose the same first ledger 96.9% of the time, but only 93% on Suspense rows. Handled by: work the AI answer out once, when the import starts, and save it with the entry; never ask again during review, so suggestions do not change under the user (the fixed-steps rule).
• Checks that did not work: two models agreeing (Haiku and Sonnet agreeing at ≥ 90% were still wrong 11% of the time without history: they share blind spots), and the AI checking exact saves (nothing to catch).

Proposed design

If the AI call fails or is late, the import goes on with the matcher alone, as today.

Decisions for you

I have not changed any product code. These need your choice:

1. Turn AI on at all, and which model? I recommend Claude Sonnet 5.5, thinking off (35%–41% fewer decisions, about $1 per 1,000 rows). Haiku 5.5 is the cheap choice (29%–33% fewer, $0.05).
2. Which rows? I recommend Suspense rows plus red and yellow suggestions. Suspense rows alone give most of the gain at a lower cost.
3. Auto-save when AI and the matcher's green suggestion agree? This reverses the 16/10 decision of "no auto-accept for green". Measured: 147–220 more saves per 1,611 rows, 0–1 wrong per run (3 wrong in 1,093 pooled).
4. Weak AI guesses on Suspense rows (the AI says below 70%; right about 40% of the time). Your rule says any clue goes to Suggestions. Should a weak AI guess count as a clue (Suggestions, red mark), or stay in Suspense with the guess pre-filled?
5. Consent. This sends narrations, amounts, dates and ledger names (not PDFs, not company files) to Anthropic. Customers must agree, as they did for Codex parsing. Should it be on by default?

Privacy

• For this study I sent narrations, amounts, dates and ledger names from the book set and the production export to Anthropic, OpenAI and Google, as you allowed. No PDFs, passwords, GSTINs or company files were sent. Narrations can hold account numbers and UPI IDs as the bank printed them.
• In the product the server already holds everything the prompt needs: the party's ledger list, the narrations and the exported choices. The design needs no data from the user's PC, and no book data on the server. Miracle history clues stay on the PC and are applied after the AI, as today.
• Under Anthropic's commercial terms, API data is not used for training by default. If processing must stay in India, Claude is offered on Amazon Bedrock's India profile (check that Sonnet 5.5 is listed).
• This page shows only counts and rates. The example entries are made up. The data and scripts stay on my PC in an ignored folder.

Limits of this study

• The book set comes from 10 companies, and one company gives over half the rows. Habits differ between accountants.
• Book labels are about 1% noisy, and they hold habits a user might fix while reviewing. Production truth favours the matcher, because users accepted its suggestions.
• Production holds only 4 exported statements and about 30 rows where AI could help.
• The "history" runs pass up to 250 earlier rows of the same company, chosen as exported choices would be; real parties may have far more, or none.
• GPT and Gemini ran only through their CLIs, so their times include the tool and their costs are worked out from reported tokens. I had no OpenAI or Gemini API key.
• Two runs of the same model differed by 0.7 points in top-1, so differences of 1–2 points between models are within noise.
• I did not tune the prompt per model. Each model got one prompt, written once.
• The pipeline numbers combine answers from full-statement runs with the matcher's state. A real system would send fewer rows per statement; the Suspense-only run shows this does not lower accuracy.
• No open model ran locally (the PC was in use); GPT-OSS 120B ran through a hosted service instead.

How I ran it

All work is in the main checkout's ignored folder .local/ai-matching-research/. prod_corpus.py turns the read-only production export into the matcheval format. A throw-away Go test (build tag aimatch, never committed) records what today's matcher does for every row, fresh and with history. run_api.py calls Claude directly; run_cli.py runs GPT through Codex and Gemini and GPT-OSS through agy; run_local.py runs open models with llama.cpp. score.py, pipeline.py, tiers.py and analyze.py score the answers; latency.py times whole statements. Every model got the same prompt: read the entries, choose up to 3 ledgers from the numbered list with probabilities, or none; do not pick a ledger because it is common.