Sources checked
Keep a worksheet for the proposed workflow
An original vendor worksheet can record the proposed task, allowed data, measurable outcome, support route, and known failure modes. Ask the vendor to run representative examples, including cases where a correct system should request review.
Separate sales claims from your pilot results
Do not turn a sales claim into your own measured result. Keep provider evidence, independent evidence, and your pilot results in separate columns so the decision remains traceable.
Questions the demo will not answer
Ask who operates the service, which model id you will receive, and what happens when that id changes. Ask what is stored from a prompt, for how long, and whether a person at the vendor can read it.
Ask how you export or delete that data. Ask who is on the hook when the system acts on the wrong document.
NIST’s AI risk framework is a voluntary map of these lifecycle questions, not a certification that a vendor passed.
A security badge, a benchmark score, or a procurement checklist does not answer whether this workflow is acceptable.
Put the vendor’s written answers next to your own pilot on representative cases, including a case where the correct behavior is to stop. Keep sales claims, the vendor’s measurements, and your measurements in different columns.
A worksheet you can reuse
Record the task, the allowed data, the measurable outcome, the support route, and the known failure modes. Ask the vendor to run your examples, not only theirs.
Note the tools the system would be allowed to call if you connect it. A chatbot that cannot send email is a different risk from an agent that can.
Define the exit before the signature. You want the prompts, the files, the indexes, and a way to stop traffic.
A price that depends on a feature you cannot leave is part of the cost. If a question on data, change notice, incidents, or exit has no written answer, that gap is the evaluation, not a detail to collect later.
What to record before you rely on it
Keep the worksheet with the model id, the data-retention answer, and the exit steps. Store your pilot results separately from the sales deck.
Re-read the change-notice term before you renew. A missing written answer stays a missing answer.
Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test. When the result is a failure, store the artifact and change the design.
Do not relax the check until the model happens to pass. Name a person who owns that failure.
A note with no owner and no inputs cannot be repeated, and a pass with no date cannot be compared with the next model id.
Common questions
Is a benchmark enough to pick a vendor?
No. A benchmark is someone else’s task. Your evaluation is your data, your failure tolerance, and a pilot you scored.
The benchmark can be a column. It cannot be the decision.
What if the vendor will only demo on their data?
Then you have not seen your workflow. Offer a small, non-sensitive sample, or record that the pilot did not happen.
Do not translate their demo into your expected error rate.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





