Blog
How to know an AI system works
An evaluation measures how often an AI system gets your cases right. A guardrail stops a bad answer before it reaches a customer.
Shivang Patel, Mirai360 · 19 July 2026
Ask a language model the same question twice and you can get two different answers. Its behaviour can also change when the provider updates the model or retires the version you use, on a date the provider sets. So a demonstration that went well once tells you little about how a system will handle next month's customers, or how it will behave on a new model version.
Two things close that gap. An evaluation measures how good the system is on your own cases. A guardrail catches the worst answers before they reach a customer.
Take an AI system that drafts replies to customers asking for a refund. It reads the order, the courier's delivery scan and your returns policy, and writes a reply for your support team to approve and send.
An evaluation is a marked test
An evaluation, usually shortened to eval, is a set of real past cases, each paired with the answer your team agrees is right. For the refund system that means a sample of past refund requests, each with the decision your team made and the reply it sent. Where two of your staff would have decided a case differently, agree the right answer before it goes into the test. The system works through the set and is marked on how often it reached the same decision, quoted the policy correctly, kept to your tone, and declined questions outside its job.
Evals matter at two points. Before launch, they tell you whether the system is good enough to go live, and which kinds of case it still gets wrong. After launch, the eval is run again whenever the instructions or the model change, so a drop in quality shows up in a test before it shows up in complaints.
A guardrail checks each answer
A guardrail is a rule checked against each answer before it leaves the system. In the refund system, one guardrail might stop a reply that offers more than the refund limit your team set. Another might stop a reply that mentions a different customer's order, and a third one that strays into a subject the system is not there to handle.
The two do different jobs. The eval score is an average over a sample, and a good average does not stop one bad reply reaching a customer. A guardrail stops particular failures on live cases, but tells you nothing about whether the system is good at its job. Guardrails need testing too, against cases written to slip past them.
What a COO should see
You do not need to write the tests yourself. From whoever builds the system, expect the eval result on a schedule, in plain terms: how many test cases it got right, and which kinds it got wrong. Expect also the list of guardrail rules in your own language, which your team can change, because those rules are your policies.
Then ask what happens when a guardrail stops an answer. Each stopped case should go to a named person on your team, with the reason attached. If it goes nowhere, the customer who was promised a reply within a day hears nothing, and the first anyone learns of it is the complaint.
Four questions for the builder
Put these to anyone building an AI system for you:
- What did the system score on the eval before launch, how many cases were in the test set, and were they kept apart from the cases used to write its instructions? If the only answer is a demonstration, there was no eval.
- How often is the eval run after launch, and who sees the result?
- What do the guardrails check, in plain words, and can your team change them?
- When a guardrail stops an answer, where does the case go, and who is told?
A builder who cannot answer all four has no evidence yet about how the system performs on your cases.
How we handle both
Every system we build carries both from the first day. The eval is made from your own past cases, de-identified and handled under the data agreement, with the right answers your team agrees. We run it before launch and again after every change to the model or the instructions. The guardrails carry rules your team writes, and every answer they stop goes to a named person with the reason.
The eval result sits in the weekly report to your leadership, next to the figure the system is meant to move, such as the hours a refund request waits for a decision. If the score falls, that week's report shows it, along with what we are changing. How we build and run systems
If you are planning an AI system that will handle customers or money, take the ten-question survey to see whether AI fits that task. Find your bottleneck