Why ChatGPT, Gemini, and Perplexity Fall Short on SAP Document Work: What a Real Side-by-Side Evaluation Actually Tests

It is the third working day of the month, and an accounts payable analyst at a shared services center is racing through a stack of supplier invoices for a German manufacturing client. One invoice runs three pages and carries 41 line items for fasteners, gaskets, and machined brackets. To save time, she drops the PDF into a general-purpose AI chat assistant and asks it to pull the line items into a table she can use in MIRO.

The answer comes back in eight seconds. It is clean, well formatted, and confident. Every column has a header. The totals line matches the invoice footer to the cent. She copies it across and moves on to the next document.

Two weeks later, the GR/IR clearing account shows a variance nobody can explain. The investigation eats most of an afternoon. The assistant returned 38 rows, not 41, because three lines sitting across the page break between pages two and three never made it into the table. The total matched because the model copied it from the footer instead of adding up the lines. And one quantity printed as "1.250" in German number format, meaning one thousand two hundred fifty units, came through as 1.25.

Nothing about the output looked wrong. That is the whole problem.

This scenario is a composite, but every failure in it is one that finance and SAP teams run into when they try general-purpose large language models on real documents. The tools are impressive. They write, summarize, translate, and reason across a huge range of topics. SAP document work asks for something different, and most side-by-side comparisons never test for it. This post walks through why these tools fall short on SAP-specific documents and what a serious evaluation needs to measure before anyone trusts the output with a posting.

Why General-Purpose Assistants Look So Good in the Demo

Most people form their opinion of an AI tool from a demo, and demos are kind. The sample invoice is a single page. It is in English. It has five line items, a clear total, and no handwriting, stamps, or scanned skew. Any modern model will ace that document, and so it should.

Real SAP document traffic looks nothing like that. A mid-sized manufacturer might receive invoices from 2,000 suppliers in six languages, with layouts that change whenever a vendor updates its billing software. Purchase orders come back as order confirmations with revised delivery dates buried in a footnote. Certificates of analysis arrive as scanned lab reports full of test result tables. Delivery notes carry three or four reference numbers, and only one of them is the number SAP needs.

ChatGPT, Gemini, and Perplexity were built for a different job. They are trained on enormous volumes of public text and tuned to give helpful, fluent answers to open-ended questions. Perplexity leans heavily on live web search and cited sources, which is great for research and not much help when the source of truth is a private invoice and your own vendor master. None of these tools arrives knowing that your supplier "Schneider Präzisionsteile GmbH" is vendor 0000100234 in company code 1000, that your tolerance for price variance is two percent, or that freight from this vendor is planned delivery cost rather than an unplanned charge.

A general-purpose model is like a brilliant new hire on their first morning. Smart, fast, articulate, and completely unaware of how your SAP system is configured. You would not let that person post invoices unsupervised on day one, and a good evaluation shows you exactly why.

The Trouble With Most Side-by-Side Comparisons

When teams do compare tools, the comparison is usually informal. Someone uploads ten documents to three chat assistants, reads the outputs, and decides which one "did best." It feels rigorous. It is not.

Eyeballing output rewards fluency. A table that looks tidy reads as correct, even when a row is missing or a value has slid one column to the right. Ten documents also cannot surface the failures that matter, because the painful errors live in the long tail of multi-page invoices, unusual layouts, and non-English suppliers. And a single run hides one of the biggest issues with generative models, which is that the same document submitted twice can produce two different answers.

The informal test also skips the final step entirely. It never checks whether the output can actually go into SAP. A beautiful markdown table is not an IDoc. A JSON object with keys the model made up on the spot will not load into the Migration Cockpit. The real question is not whether the AI read the document. It is whether the output could be posted, and whether the posting would be right.

A proper evaluation answers that second question. It rests on three pillars that match the three places general-purpose tools break down, plus a set of operational tests that almost never show up in a demo.

Pillar One: Tabular Data That Survives Contact With Reality

Line items are where the money is. The header of an invoice tells you who and when. The table tells you what, how many, and how much, and it is the part SAP matches against the purchase order and the goods receipt. It is also where general-purpose models struggle most.

Large language models process documents as sequences of tokens. A table is a two-dimensional structure, and when it gets flattened into a sequence, the relationships between rows and columns have to be rebuilt from context. On a short, clean table that works well. On a long table with wrapped descriptions, merged cells, and a page break in the middle, it gets fragile fast.

A real evaluation tests tabular extraction on several specific fronts.

Row completeness across page breaks. The test counts rows on every multi-page document and compares them to ground truth. Page breaks are the classic trap. The header row often repeats on the next page, a "carried forward" subtotal appears, and the model has to decide what is a real item and what is layout noise. A missing row does not throw an error. It just quietly disappears, which is how 41 lines become 38.

Column alignment when cells wrap. A material description like "Hex bolt DIN 933 M12x40 8.8 zinc plated, packed in cartons of 200" often wraps onto a second line. Weaker extraction treats that second line as a new row, or pulls the quantity from the row below up into the wrong item. The evaluation should include documents with long descriptions and check that every value stays attached to its own line.

Lines that are not items. Invoices mix product lines with freight, packaging, minimum order surcharges, early payment discounts, and subtotals. Each one needs different treatment in SAP. A good test checks whether the tool separates them correctly instead of treating a freight charge as line 42 or counting a subtotal twice.

Locale-aware numbers and dates. "1.250,00" on a German invoice is one thousand two hundred fifty. "1,250.00" on an American invoice is the same number. "03/04/2026" means March 4 in one country and April 3 in another. General-purpose models often guess the format from the surrounding language, and they guess wrong often enough to matter. Your test set should deliberately include suppliers from several locales.

Arithmetic consistency. Quantity times unit price should equal the line amount. The line amounts should add up to the net total. Net plus tax should equal gross. These checks are simple, and they are the fastest way to catch a dropped row or a misread number. A model that copies the total from the footer will pass a lazy test and fail this one immediately.

The score that matters here is not the percentage of characters read correctly. It is line-level precision and recall. How many of the true line items did the tool find, and how many of the lines it returned were real? A tool that reads 99 percent of characters correctly can still get a third of your multi-page invoices wrong at the line level.

Pillar Two: Domain Terminology and the Language SAP Speaks

Suppose the table comes out perfectly. The next problem is meaning. Suppliers write in their own vocabulary, and SAP needs its own. Bridging that gap is where general knowledge runs out and configuration knowledge takes over.

Take reference numbers. A typical German supplier invoice might show "Ihre Bestellung 4500012345," "Unsere Auftragsnummer 20231187," "Lieferschein 80045521," and "Kundennummer 10442." Only the first is the purchase order number that MIRO needs. A general-purpose model can translate all four labels correctly and still hand you the wrong one, because it has no way of knowing that your PO numbers start with 45 and run to ten digits. An evaluation should score reference number selection as its own field, because getting it wrong breaks the three-way match before anything else happens.

Units of measure cause similar trouble. The supplier prints "Stk." SAP might store the internal unit as ST and display it as PC. Your purchase order might say BOX while the invoice bills in EA. A tool that passes through whatever the document says leaves the conversion problem for a human to discover at posting time. A domain-aware test checks whether the output unit matches what the PO expects, and whether quantities were converted when the units differ.

Then come the charges and codes that only exist inside your configuration. Is this freight line planned delivery cost that should be spread across items, or an unplanned delivery cost that goes to a separate G/L account? Which tax code applies to a reverse charge supply from another EU country? SAP tax codes like V1 or E4 mean nothing outside your system, since every company defines its own. A general-purpose model will happily invent a sensible-looking tax code. A good evaluation checks each one against the codes that actually exist in your configuration.

Master data identifiers are the last piece. SAP does not post to "Schneider Präzisionsteile GmbH." It posts to vendor 0000100234. It does not post "hex bolt M12x40." It posts material 000000000010004471, or whatever your numbering scheme produces, including the leading zeros that SAP's conversion routines add and that many tools strip away. The test here is whether the output resolves names and descriptions to real master data keys, not whether it reads the name correctly.

This is the pillar where the gap between general and specialized tools gets widest. A general-purpose model knows a lot about SAP in the abstract. It can explain what MIRO does and what a movement type is. What it does not have is your system. Knowing that movement type 101 is a goods receipt is textbook knowledge. Knowing that goods receipt 5000087612 posted 1,250 pieces against PO item 00020 last Tuesday is context, and context is what makes a posting correct.

Pillar Three: SAP Template Structure and Output That Actually Loads

The third pillar is the one informal comparisons skip completely. Extraction is only half the job. The data has to land in SAP in a shape SAP accepts.

Depending on the process, that shape might be an INVOIC02 IDoc with its header segment E1EDK01, partner segments E1EDKA1, item segments E1EDP01, and summary segment E1EDS01. It might be the parameter structure for BAPI_INCOMINGINVOICE_CREATE, where the header needs a company code, document date, posting date, currency, and gross amount, and each item needs the PO number, PO item, quantity, unit, amount, and tax code. For data migration, it might be a Migration Cockpit template, a workbook with specific sheets, mandatory columns, and field lengths that the load validates line by line.

Each of these formats has hard rules. Dates in SAP's internal format are eight digits, year first. PO item numbers are five characters, so item 20 becomes 00020. Amounts use a decimal point regardless of the supplier's locale. Fields have maximum lengths, and a vendor invoice reference longer than 16 characters will be cut off or rejected. Mandatory fields cannot be empty.

General-purpose chat assistants do not work to a fixed schema unless you force them to, and even then they drift. Ask for JSON on Monday and you get a key called "po_number." Ask again on Tuesday and it is "purchaseOrder." A field with no value on the document might be left out, set to null, set to an empty string, or filled with a polite placeholder like "Not specified." Every one of those variations breaks an integration that expects a fixed structure.

A real evaluation takes each output and tries to load it. Does the IDoc pass syntax checks? Does the BAPI call return without errors in a test client? Does the Migration Cockpit template validate? The score is the share of documents that load cleanly, and it often looks very different from the extraction accuracy measured in the first two pillars. A tool can read everything correctly and still produce output that nobody can post without rework.

The Tests Nobody Puts in the Demo

Beyond the three pillars sits a set of operational questions that decide whether a tool can run in production at all.

Consistency across runs. Submit every document in the test set three times and count how often the outputs differ. Generative models sample their answers, and even with settings turned down for predictability, results can vary between runs. For an auditor, "the system gave a different answer the second time" is not a comfortable sentence.

Confidence at the field level. When a value is smudged, cut off, or ambiguous, does the tool say so? General-purpose assistants tend to fill gaps with plausible values rather than flag them, because they are built to be helpful. In document processing, a confident guess is worse than an honest "unclear." The evaluation should include deliberately degraded scans and check whether the tool raises its hand.

Audit trail. For every value that ends up in SAP, can you trace it back to the exact spot on the source document? Finance teams answer to auditors, and "the AI said so" does not satisfy anybody.

Data handling. Supplier invoices contain bank details, pricing, and contract terms. Pasting them into a consumer chat tool raises questions about where the data goes and how long it stays there. Enterprise plans address some of this, but the question needs a documented answer before the pilot starts, not after.

How to Run Your Own Side-by-Side Evaluation

If you are weighing a general-purpose model against a purpose-built platform, the fairest thing you can do is test both on your own documents with a scoring method decided in advance. Here is a practical way to set that up.

Start with the test set. Pull 200 to 300 documents from your real inbox over the last quarter rather than cherry-picking good examples. Make sure the mix reflects reality, with multi-page invoices, several languages and number formats, scanned and native PDFs, and your ten most awkward suppliers. If certificates of analysis, order confirmations, or delivery notes matter to your process, include them too.

Build ground truth next. For each document, write down the correct values in the exact form SAP needs, including vendor number, PO number and item, SAP units, tax codes, and every line item. This is the slowest part of the whole exercise and the most important, because without it every comparison slides back into eyeballing.

Then run every tool on every document at least three times with identical instructions. Keep the prompts fixed. If one vendor gets special tuning, give the others a fair chance to match it.

Score the results at four levels.

Finally, measure the human cost. Time how long a reviewer spends fixing each tool's output. A tool that is 92 percent accurate but makes you check every field can cost more in practice than one that is 97 percent accurate and tells you exactly which three fields to look at. The real comparison is cost per correctly posted document, with people's time included.

One tip from teams that have done this well. Look closely at the failures, not only the scores. If a tool's mistakes cluster on one supplier or one layout, a configuration change might fix them. If they are scattered randomly across the test set, that points to a deeper limitation that no amount of prompting will solve.

What Purpose-Built SAP Document Processing Looks Like

The three pillars point toward a different design. Instead of one general model trying to do everything in a single pass, an SAP-aware system splits the work across specialized steps, each with its own checks.

At Artificio, AI agents handle that work rather than traditional OCR templates. A classification agent first works out what kind of document has arrived, whether a supplier invoice, a credit memo, an order confirmation, or a certificate of analysis, along with its language and number format. An extraction agent then reads the layout as a layout, rebuilding tables across page breaks and keeping wrapped descriptions attached to the right rows. A validation agent checks the results against the business by comparing lines to the purchase order, quantities to goods receipts, vendors to master data, and totals to the sum of the lines. A mapping agent produces output in the structure SAP expects, whether that is an INVOIC02 IDoc, a BAPI payload, or a migration template, with the same schema every time.

When something does not match, the document does not slide through on a confident guess. It goes to a person with a specific reason attached, such as "price on line 17 exceeds PO price by 4.1 percent, above the 2 percent tolerance" or "PO 4500012345 item 00030 has no goods receipt." The reviewer fixes one field instead of re-checking forty.

This is not about having a smarter model. The large models behind consumer chat tools are extremely capable. The difference comes from the context and structure wrapped around that intelligence, meaning your master data, your configuration, your output formats, and a set of checks that catch errors before they reach the ledger.

Where General-Purpose LLMs Still Earn Their Place

None of this means chat assistants have no role on an SAP team. They are genuinely useful for plenty of adjacent work. They can explain an unfamiliar SAP error message, draft a polite email asking a supplier to reissue an invoice with the correct PO number, summarize a long framework agreement, or help a consultant rough out the first draft of a functional specification. Perplexity is handy for tracking down SAP notes and community answers on a configuration question. 

The line falls where accuracy has to be exact and repeatable at volume, and where output has to flow into a system of record without a person re-checking every value. That is transactional document processing, and it is the work where general-purpose tools fall short today. 

Think of it as the difference between a talented generalist and a trained specialist. You would happily ask a brilliant generalist to read a contract and tell you what stands out. You would not ask them to run your month-end close on their first day, no matter how smart they are, because the close depends on knowing your chart of accounts, your tolerances, and your suppliers. SAP document processing sits much closer to the close than to the contract read. 

The Question Worth Asking 

Most comparisons between AI tools start with the wrong question. Asking which model is the smartest produces interesting benchmark charts and very little insight into whether an invoice will post correctly on the third working day of the month. 

A better question is narrower and far more useful. Given our documents, our SAP configuration, and our output formats, which approach produces postable data, flags its own uncertainty, gives the same answer every time, and leaves a trail an auditor can follow? 

Answering that takes a few weeks of disciplined testing on real documents with real ground truth. It will almost certainly show that general-purpose assistants read documents well and struggle with everything that happens after the reading. It will also show you exactly where a purpose-built approach closes the gap, and where it does not, so the decision rests on evidence rather than on the demo. 

If your team is planning that kind of evaluation, Artificio can run your documents through the same pillars described here and show line-level, load-level, and posting-level results side by side with whatever you are comparing against. Bring the ugliest invoices you have. Those are the ones that tell you the truth.