How to Evaluate an AI Vendor Before You Buy

An AI vendor should not win your business because the demo was smooth, the founder was convincing, or the feature list used the word “agent” twelve times.

Evaluate an AI vendor by making it produce evidence for your business. Define one useful job, give serious candidates the same representative work, trace the data, test failure conditions, name the operating owner, clarify what your business keeps, and prove you can leave without the operation falling through the floor.

That is the short version of the Proof-before-Promise Test: seven receipts before a vendor earns a yes.

The U.S. General Services Administration’s current AI purchasing guidance follows the same practical order: begin with the need, test a solution before a broad purchase, understand the data flow, involve the right people, and control ongoing usage costs. Your process can be lighter than a federal acquisition. The logic still travels well.

A great demo proves the vendor can give a great demo

That sounds flippant, but it is an expensive distinction.

A sales demonstration usually has friendly inputs, a prepared path, and someone nearby who knows exactly where the sharp edges are. Your business has incomplete requests, contradictory records, odd exceptions, distracted users, unavailable integrations, and customers who do not follow the script.

The gap between those two worlds is where the buying decision lives.

This is not paranoia. It is ordinary commercial discipline, especially in a market where extraordinary claims can outrun the evidence. In March 2026, the Federal Trade Commission announced a proposed settlement that would ban Air AI and its owners from marketing business opportunities after allegations involving deceptive earnings, performance, and refund claims aimed at entrepreneurs and small businesses. The useful buyer lesson is not that every AI seller is suspect. It is that confident claims still need support. Read the FTC’s account of the case, then build your evaluation so enthusiasm never has to do evidence’s job.

Run the Proof-before-Promise Test

The test asks for seven receipts. A receipt is something you can inspect, compare, keep, or act on. It may be a completed work sample, a written data-flow answer, a failure log, an operating plan, or an export demonstration.

Not every purchase needs a procurement opera. A low-consequence writing assistant used with public information deserves a lighter review than a system that contacts customers, reads private records, or changes an operational commitment. Scale the depth to the consequence.

Do not remove the gates.

Receipt 1: proof that the vendor understands the job

Write the job without naming the product:

This system helps this person complete this task using these inputs. A useful result must meet these conditions, and it must stop or hand off when these conditions appear.

“Improve customer service” is not a job. “Draft an answer for a service coordinator from approved policy material, show the supporting passage, and escalate account-specific exceptions” is close enough to test.

Give that statement to the vendor and ask it to play it back in operational language. You are listening for the real user, the current friction, the input, the desired output, the boundary, and the handoff. If the answer immediately becomes a tour of the product, the vendor may be selling its inventory instead of solving your problem.

The first receipt is the agreed job statement. Without it, every later proof can slide toward whatever the product already does well.

Receipt 2: proof from the same real work

Create one small work packet and give the same packet to every serious candidate. Include:

• normal cases that should pass cleanly;
• incomplete or ambiguous cases;
• an exception that requires a person;
• an input outside the supported job;
• a case containing information the system should not expose; and
• the scoring rules you chose before seeing the answers.

Ask each vendor to show the output, the configuration used, the human correction required, and any failure or refusal. If the product changes its behavior through setup, retrieval, tools, or workflow logic, include those pieces. You are buying the system’s behavior, not a model performing alone under stage lighting.

GSA’s guide to starting an AI project recommends putting technical tests into evaluation criteria so a proposed approach can be verified against the buyer’s specific circumstances. It also separates developmental testing from operational testing under realistic conditions. That is the heart of this receipt: make the candidate do your kind of work before your operation depends on it.

OpenAI’s current evaluation documentation makes the mechanics concrete: define the expected behavior, use testing criteria, and run representative test data against it. You do not need a particular provider’s evaluation product to apply that discipline. The durable asset is your work packet and definition of acceptable behavior.

If you are comparing underlying models rather than vendors, use the full Same-Work Audition. This receipt stays at the vendor level: can the complete offered solution do the job in your environment?

Receipt 3: proof of the complete data route

Ask the vendor to draw what happens from the moment information enters the product until it is deleted or exported.

The drawing should identify:

• the exact product and service tier;
• the data and files you will provide;
• where they are processed and stored;
• which model providers, subprocessors, or integrations receive them;
• who can access the information;
• whether content may be reused to improve a service;
• retention and deletion behavior; and
• what appears in logs, backups, exports, and support systems.

A certificate, trust-center logo, or “enterprise-grade” label may support part of this conversation. It does not answer the route by itself.

This is the deliberate boundary with the existing AI vendor privacy checklist. Use that guide when business information will cross the vendor boundary. The new vendor evaluation should record whether the privacy audit passed, failed, or requires a smaller permission envelope. It should not squeeze the whole audit into one checkbox.

Receipt 4: proof that failure has a shape

Ask the vendor to demonstrate what happens when:

• required context is missing;
• a connected system is unavailable;
• the input is contradictory;
• the model produces a low-confidence or unacceptable result;
• a person rejects the recommendation; and
• the action must be stopped, corrected, or reversed.

You are looking for a visible failure, a safe next step, and enough evidence to understand what occurred. “The AI learns over time” is not a recovery plan. “Contact support” is not a complete operating design.

NIST’s AI Risk Management Framework notes that acquired third-party technology may be complex or opaque and that a provider’s risk tolerance may not match the organization deploying it. That is why this receipt is a gate. A tolerable failure for the vendor’s broad customer base may be completely wrong for your specific workflow.

Do not demand impossible perfection. Demand honest boundaries, observable behavior, and a recovery path that matches the consequence.

Receipt 5: proof that someone can run Monday morning

AI products create work around the work. Someone must manage access, monitor usage, handle exceptions, review quality changes, respond to incidents, update source material, and decide when a new product release deserves another test.

Ask the vendor for an operating map with names or roles on both sides:

Operating responsibility	Your business	Vendor
User access and permissions	Who approves and removes access?	What controls and records are provided?
Quality review	Who samples real output and decides what is acceptable?	What monitoring or review support exists?
Exceptions	Who receives the handoff?	How is the exception made visible?
Product changes	Who decides whether to adopt a change?	What notice and release evidence are supplied?
Incident response	Who owns the business response?	Where is the escalation path and status evidence?

The right split depends on the product. The wrong split is the one nobody writes down.

If the vendor says the product is effortless, ask who handles the effort when something changes. If your team says the vendor owns everything, ask who remains accountable to the customer or employee affected by the system. A subscription can transfer tasks. It does not transfer your entire operating responsibility.

Receipt 6: proof of what your business keeps

The buying conversation should separate four kinds of assets:

1. Your business material: source documents, records, examples, instructions, and feedback.
2. The vendor’s product: its platform, general models, reusable components, and protected methods.
3. Configured operating assets: prompts, workflows, evaluation sets, integration mappings, rules, and documentation created for your use.
4. Evidence: logs, test results, decision records, reports, and change history needed to understand the system.

Ask what you can access, export, reuse, or transfer in each group. Get the answer in the appropriate commercial documents and involve qualified counsel when ownership or contract language materially affects the deal. The purpose here is not to declare a legal result. It is to stop the business from discovering after purchase that a crucial operating asset exists only inside a vendor account it cannot carry forward.

The GSA project guide also recommends requiring useful delivery artifacts—not merely relying on abstract ownership language—so work can move between teams. For a small business, that principle becomes very practical: keep the work packet, configuration record, operating instructions, integration map, and known limitations somewhere the business controls.

Receipt 7: proof that the exit works

Ask the vendor to walk through the end before you sign the beginning.

Can you export your data in a usable format? Can you retrieve the configured rules and evidence you are entitled to keep? Who disconnects integrations and revokes credentials? What is deleted, what is retained, and how is that confirmed? Can the business continue the underlying workflow manually or with another system while the transition occurs?

The UK government’s AI procurement guidelines explicitly connect AI buying with data assessment, transparency, lock-in, ongoing support, ownership, and end-of-life planning. Your company may not need government-style documentation, but it does need a believable answer to one humble question: if this relationship ends, what breaks?

The receipt is an exit map, not a promise that migration will be painless. A credible vendor should be able to explain the boundary without acting as though asking about departure is an insult.

Compare gates first, preferences second

Once the receipts are collected, do not bury a dangerous failure inside an average score.

Use three decision lanes:

• Gate: a requirement that must pass, such as the data boundary, unacceptable failure behavior, or a workable exit.
• Evidence: a result that can be compared, such as performance on the same work packet, correction burden, integration fit, or operating effort.
• Preference: a useful tie-breaker, such as interface polish or a desirable convenience.

A beautiful interface cannot compensate for a prohibited data route. A famous customer logo cannot compensate for failure on your work. A low introductory price cannot compensate for an operation nobody can own.

This does not mean the least risky vendor always wins. It means the business chooses its tradeoffs consciously, after deal-breakers have been dragged into the daylight.

The best vendor conversation gets more specific as it goes

Good buying conversations usually become less magical over time. The job gets narrower. The work samples get messier. Responsibilities become visible. Limitations are named without drama. The vendor may even recommend a smaller starting point or explain why part of the request should remain manual.

That is not a weak sales performance. It is useful evidence about judgment.

If a candidate resists every request for proof, insists its preferred demo is enough, or treats ordinary exit questions as hostility, you have learned something before the learning became expensive. If a candidate can show the work, explain the boundaries, surface failure honestly, and leave your business with usable operating artifacts, you have learned something better.

The goal is not to find a vendor that promises certainty. It is to find one whose evidence lets you make a clear-eyed decision.

If you want a neutral second set of eyes before a product or implementation partner becomes part of the operation, explore AI consulting services. We can help define the job, build the same-work packet, inspect the evidence, separate gates from preferences, and turn the buying decision into an operating plan your team can actually own.