The CFO's 5-Question Test Before Buying Any Document AI

A few weeks ago I spoke with a CFO who'd just come out of her third document AI demo in two months. Each vendor had processed the same stack of sample invoices. Each one extracted the right fields. Each one showed a clean dashboard and a confident ROI projection.

"They all look the same," she said. "How am I supposed to know which one actually works?"

That question is exactly right, and it's also the problem. Document AI vendors have gotten very good at demos. The sample documents are clean. The extraction is accurate. The time-savings math is compelling. What you don't see in the demo room is what happens when a supplier changes their invoice layout, when a batch of scanned documents comes through at 200 DPI, when a new acquisition drops 14 new document types into your workflow overnight, or when the finance team is staring at a queue of 300 flagged exceptions with no clear process for resolving them.

The demo shows the AI at its best. You need to see it at its most ordinary, and occasionally at its worst.

The five questions below are designed to do exactly that. They're not technical questions, and you don't need a background in machine learning to ask them. They're financial and operational questions, and they get at the five dimensions of document AI deployment that determine whether your investment delivers or disappoints. Every strong vendor will have strong answers. Every weak one will deflect, generalize, or redirect back to the feature list.

Why This Category Is Harder to Buy Than It Looks

Document AI sits in a tricky place for CFOs. It's not infrastructure, where you're mostly buying capacity and reliability. It's not SaaS, where the product does exactly what it's configured to do. It's a probabilistic technology, which means it makes predictions. Most of those predictions are right. Some aren't. And how the system handles the ones that aren't, at scale, over time, is where the real performance story lives.

The market doesn't make this easier. "AI-powered" is on nearly every document processing vendor's website today, but the underlying approaches vary wildly. Some products use genuinely modern large language models that understand context and can generalize across document types they've never seen. Others are essentially template-matching systems from five years ago, given an AI rebrand and a new UI. The difference is enormous in practice, especially for organizations whose document universe keeps expanding.

The other complicating factor is that document AI is almost never a standalone purchase. It needs to connect to your ERP, your workflow tools, your data warehouse, your compliance infrastructure. Implementation complexity is consistently underestimated. Integration costs are consistently understated. And the first time something breaks in production, you'll find out how mature the vendor's support operation actually is.

These five questions cut through all of that. They're the ones I'd want answered before I signed any document AI contract.

The Five Questions

1. What happens when your AI isn't confident?

Start here. It's the most revealing question in the conversation and the one most vendors aren't fully prepared for.

Every document AI system encounters documents it's uncertain about. Low-quality scans. Unusual layouts. Handwritten fields. Documents with missing data in key positions. What matters isn't how often this happens (you can ask that separately) but what the system does when it does happen.

The answer you're looking for sounds something like this: the system assigns a confidence score to every extraction, flagged items above a configurable threshold go into a review queue, a human reviewer can compare the AI's output against the original document side by side, any corrections feed back into the system to improve future performance, and there's a full audit trail for compliance.

That's an operational workflow, not just a feature. And it implies the vendor has thought seriously about what production really looks like for a finance team.

The answer you want to avoid sounds like: "Our accuracy is 98%, so exceptions are rarely an issue." That's not wrong, but it's not an answer either. At 98% accuracy on 10,000 documents a month, you have 200 exceptions. That's a meaningful number. What happens to them? If the vendor can't tell you precisely, that precision gap becomes your problem three months after go-live.

Also worth probing here: how does the system learn from corrections? Does every human review improve future extractions, or does the AI treat every document as a fresh start with no institutional memory? The learning loop matters more than the initial accuracy number.

2. Can it handle a document type it's never seen before?

This question separates genuinely modern AI from legacy technology with a new label.

Older document processing systems, including many that are currently marketed as AI, work by training on examples of each document type. You want to process purchase orders from Supplier X? You train the system on 50 to 200 purchase orders from Supplier X. You want to add a new vendor's invoice format? You open a support ticket and wait. This is called template-based extraction, and it was the industry standard for years.

Truly modern document AI generalizes. It reads a document it's never encountered, understands the structure from context, and extracts the right fields without needing a template or a training run. This is a genuinely different capability, and it has a real-world consequence for your operations: new document types just work, without creating a backlog of configuration requests.

The best way to test this is to bring one of your own documents to the demo. Not a clean, templated invoice from a well-known supplier. An awkward one. A manually created purchase order. A form from an agency your team finds painful to process. Don't tell the vendor in advance. See what happens.

If the system handles it gracefully, that's a strong signal. If the vendor says they'll need to train on it before the next meeting, that tells you something important about the maintenance overhead you're buying into.

3. Where exactly does our data go?

In the generative AI era, this question requires more precision than it used to.

The basic questions are familiar: where is processing happening geographically, what certifications does the platform hold (SOC 2 Type II, ISO 27001, HIPAA where relevant), and what is the data retention policy after documents are processed?

The newer question is this: is our data used to train or improve any shared model? Some AI vendors, not all, use customer interactions to improve their underlying models. That could mean your invoices, contracts, and financial documents are contributing to a system that also serves your competitors. Most reputable vendors have explicit policies against this. But you need it in writing, in the contract, not just confirmed verbally during a sales call.

Also ask about the offboarding process specifically. What happens to your data if you terminate the relationship in year two? Is there a certified deletion process? How long does it take? What documentation do you receive? The vendors who've built this process properly will answer quickly and hand you a document. The ones who haven't will get vague or promise to follow up.

If you're in financial services, insurance, healthcare, or any other regulated industry, this question gets more pointed. Your document AI vendor is almost certainly a data processor under whatever compliance regime applies to you. That means they need to appear in your data processing agreements, and their security posture needs to align with your regulatory obligations, not just meet a general standard.

4. What's the fully loaded cost at our actual volume?

The number on the pricing slide is never the number you'll actually pay. This isn't always intentional deception. It's more often the result of a sales process that focuses on per-page or per-document pricing while treating implementation, integration, and ongoing maintenance as separate conversations.

Ask the vendor to build a fully loaded cost model with you, right now, in the meeting. Not a ballpark. A model that includes licensing, implementation fees, integration development time, ongoing maintenance, and the internal staff hours your team will spend managing the system and handling exceptions.

Then ask what happens to that number as your volume scales. If you're processing 5,000 documents a month today and that grows to 15,000 over the next two years (a reasonable outcome if the system works and adoption expands), what does your cost look like? Some vendors have linear pricing that scales reasonably. Others have tier structures that create significant jumps at certain volumes.

Integration costs are the most consistently underestimated line item in document AI deployments. If you run SAP, Oracle, or any major ERP, connecting a new AI system to it requires real engineering work. Weeks to months. Developer time. Testing. Change management. Get a number for this, and get it in writing. And ask who's responsible for maintaining that integration when either system updates.

One more thing to put in the model: what happens to your cost structure if the vendor raises prices or changes their pricing model? Enterprise SaaS contracts lock in rates for the term. Document AI contracts should too. If the vendor won't commit to pricing stability in writing, that's worth knowing before you sign.

5. What does success look like in the first 90 days, and who's accountable for it?

This question tests something more important than the vendor's product. It tests their operational maturity and their confidence in what they've built.

Anyone can say "you'll see significant time savings within the first quarter." What you want is specificity. What metric are we targeting? What's my current baseline, and what should it be after 90 days? Who on your team is responsible for making sure we get there? What does the onboarding timeline look like week by week, and who do I need to provide from my side?

Vendors who've deployed successfully many times know the answers to these questions without having to check. They've built onboarding playbooks. They know what success looks like at your volume and document type. They know which integration steps take longest and what the common failure modes are. They'll propose specific KPIs, agree to review them at 30, 60, and 90 days, and name the person on their team responsible for your outcome.

The vendors who haven't deployed successfully at scale will give you generalities. "It depends on your specific use case." "Most customers see results within the first few months." These aren't wrong statements. They're just not commitments, and commitments are what make a vendor a partner rather than a supplier.

One useful question to tuck in here: ask for a reference from a customer in your industry with a similar document volume, someone you can call directly. The vendors with strong track records will make that introduction. The ones who offer only written case studies and hesitate on live references are signaling something worth taking seriously.

Reading the Room Across Vendors

When you're comparing two or three vendors who've all made it through the demo stage, these five questions create a natural scoring mechanism. Give each vendor two points for a specific, confident, documented answer. One point for a partial or hedging answer. Zero for a deflection or a redirect back to the product deck.

A vendor who scores eight or more out of ten is worth serious consideration regardless of how they performed in the demo. A vendor who scores five or below, even with a flawless demo, is showing you something important about how they'll behave after the contract is signed.

The score isn't the only output though. Pay attention to how vendors respond to the questions themselves. Do they lean in and engage, asking clarifying questions about your specific situation? Or do they get defensive, pivot to testimonials, or start explaining why the question doesn't quite apply to their product?

How a vendor handles pressure in a sales conversation is a direct preview of how they'll handle problems in production. The ones who engage genuinely with hard questions are showing you something real about their culture. The ones who deflect are also showing you something real, just not in their favor.

Getting This Right Before You Sign

Bring these five questions into your next vendor conversation. Put them at the beginning, not the end. A strong vendor will engage with them from the first exchange and use them as an opportunity to differentiate from competitors who can't answer as clearly.

The document AI category is genuinely valuable. Organizations that have deployed it well are processing faster, making fewer errors, and redeploying staff time toward higher-value work. That's a real outcome. But the gap between the vendors who can deliver it consistently and the ones who can't is meaningful, and that gap shows up in exactly the scenarios these questions are designed to surface.

The best vendors know what these questions are. They've answered them before. They've built their products and processes to answer them well. And they'll welcome the conversation, because they know what the honest answers reveal about their competitors.

That's the vendor worth buying from.