Docs / Testing, improving and debugging

Diagnose and improve an agent

For when an agent did not do what you expected, whether it failed outright or is just not reliable.

Where the fix gets made

Almost every fix in this article is made by talking to the Agent Builder, which is docked on the left of every tab of every agent. You describe the change you want in ordinary words and it makes it. You do not fill in forms.

It is the same builder on every tab. What changes is what it is looking at. Open the Script tab and say "stage three should not pass unless the answer names a specific contract clause", and it changes stage three. Open the Resources tab and say "the agent should not have the write operation until stage four", and it changes that instead. So the first move when fixing something is to open the tab the problem lives on.

Orca is not where you go to edit. Orca is the provisioning conductor: you describe something you want and do not have yet, it looks at what you already have, proposes a plan as a card, and stops. Use it to bring a new agent, function or connection into existence. An agent that already exists and is behaving badly is the Agent Builder's job.

Work out which of nine things happened first

They are listed in the order they turn out to be true. Start at the top. Most reports of "the agent is broken" are settled in the first three.

1. It never ran

Look in Observability → Resources for the agent. No run recorded means nothing started it, and the agent is not the thing to fix. Go to the agent's Triggering tab and check what is meant to start it: a person, a schedule, your own software using an org API key, another agent, a call from Claude, or an embedded session in your product.

The one people miss is the run that half happened. A schedule that should have fired fourteen times this month and fired nine is not a healthy agent, and looking only at the last run will tell you it is fine. Ask the Observability Guide for the run count over a window.

2. It ran, and it changed nothing

The most common false alarm by a distance. The run finished, the agent reported that it registered a supplier query, and finance says no query exists.

Nothing is wrong. A write does not happen when the agent calls it. The agent recorded exactly what it would send, the run carried on and finished, and the change is sitting on an approval link waiting for a person to open it and approve it. The invoice agent's stage four does this on every single query.

Fix: find the pending approval and get it approved. If the queue is always full, the problem is that nobody owns it, not the agent. See Making it run, and approving what it does.

3. It did not get past a stage

Open the run and read the report the agent was given. When a stage condition does not hold, Mindset writes a short report saying which conditions were tested, which held, which did not, and what any judgement said, and hands it to the agent along with a standing instruction not to claim a stage is complete when it is not. That report names your problem in plain words.

Nine times out of ten the condition is worded badly rather than the agent being poor. Stage three of the invoice agent says the answer must name a specific contract clause. If your supplier contracts knowledge base has no clause numbers in it, no answer will ever name one, and the agent will keep trying.

Fix, on the Script tab, by telling the Agent Builder either what the condition should really say, or what the stage is missing in order to meet it.

4. It got through every stage and the result is wrong

Every condition passed, and the output is still not what you wanted. Your conditions can be satisfied without the work being done.

The usual culprit is a loose judgment. "A recommendation exists" passes on anything at all, including "the difference looks acceptable". "A recommendation naming a specific clause of the supplier contract" does not.

The second culprit is a stage that had nothing to do. If stage three is granted the supplier contract's knowledge base and never called, it is deciding whether the contract allows the difference without opening a contract, and the answer will sound just as confident. Observability → Resources, declared versus observed, shows you that as a granted-and-never-called resource.

Fix: tighten the condition so that meeting it requires the work. Then re-run the same invoice and read the judgment's recorded reasoning.

5. It did something at the wrong stage

It posted the supplier query before it had compared anything. There is one cause: it had that operation at that stage. An agent cannot call something a stage does not offer it.

Fix on the Script tab: take the write operation off every stage except stage four. Anything that changes another system belongs at the end, because it pauses the run for a person.

6. A connection failed

The run stops with an error naming an outside system. Open the connection and look at the operation.

Nearly always an expired or rotated credential. Someone changed the finance system password, or a key reached its expiry date, and nothing in Mindset knew. The tell is that every operation on that one connection fails at once while everything else is fine.

Fix: Connections → the connection → Settings, replace the login details, then go to Data preview (or Query console for a database) and run one read on its own before you re-run the agent. Login details are held by us and never reach the model, so nothing else needs touching when they change.

The other cause is narrower: one operation fails with an authorisation error while the others still work. The credential is valid but scoped to less than that operation needs.

7. The data is wrong and nothing errored

The output is plausible, well-formatted and about the wrong invoice, or built on lines that are not there. Nothing failed, so nothing told you.

Two causes worth checking in this order:

• The connection is pointed at the wrong account. Most login details work across several accounts or environments, and a credential aimed at the wrong one returns perfectly valid data. Run one read from Data preview and check a value you can verify elsewhere, such as the PO total, against what finance sees on their screen.
• A function is reading fields that do not exist. If an operation's output shape was ever taken from documentation rather than a real response, the invoice-comparison function pulls out an empty list of lines and reports no differences on every invoice. Open the function's Preview tab, run it on INV-4471, and watch which step returns nothing.

8. It is inconsistent, and right most of the time

The hardest one, and the most common after the first three. Same kind of invoice, right on Monday, wrong on Thursday. Every cause here is the same shape: something was left to the model that should not have been. Check these four in order.

• The script is doing too little. One stage saying "handle the invoice exception" gives the model the whole job in one go, and it will find a different route through it each time. Break it into stages with conditions, so it cannot decide it has finished.
• It cannot reach what it needs. An agent missing the supplier's payment terms will not stop and ask. It will produce an answer that reads exactly like an informed one. Check declared versus observed for the run that went wrong and compare it to one that went right.
• The instruction can be read two ways. "Recent invoices" means this month to one run and this quarter to another. Say the number. "We have the supplier's invoices from the last six months" can be checked; "we have the recent ones" can only be claimed.
• There is no judgement in it at all. If nothing reads the output before it goes out, quality is whatever the model produced that time. Add a judgement to the stage where quality matters, and remember judgements are strict: only a clear yes passes, and the reasoning is recorded either way.

9. It worked yesterday and not today

The agent is rarely what changed. Check, in this order:

1. Which version ran. It is on the run. Two runs behaving differently with two different version numbers is your answer.
2. A credential. See cause six.
3. The other system. A field renamed at the far end changes nothing here and breaks a function quietly.
4. The environment. A resource that says never used when you know it has run is nearly always you looking at the wrong environment.
5. Volume. A run gets about a minute when somebody is waiting and about five when nobody is, and it can reach outside about 100 times. A month-end batch can cross a ceiling that a single invoice never approaches. See Limits and run behaviour.

When a prompt needs a definition or process, make it a script or function

A specific fix worth its own section, because it turns an unreliable agent into a reliable one more often than any rewording.

Your system prompt says the agent should query anything with a material discrepancy. "Material" is a real rule in your business: over £500, or over 5% of the purchase order value. Written in the prompt, that rule is being applied by a model reading a sentence, and a £502 difference on a £90,000 order will go one way on one run and the other way on the next.

Move the definition into the invoice-comparison function. Its last step already returns the differences, their total value, and whether there are any at all. Have it also return whether the total is over the threshold. Now:

• The threshold is a number in one place, and changing it is one edit with a version history.
• The stage condition becomes a check, which is a straight test of what was collected: fast, free, and not a matter of opinion.
• Every run applies the same rule, because it is arithmetic rather than reading.

The general form: if a phrase in your prompt would need a definition before somebody else could apply it consistently, that definition belongs in a function, not in the wording.

After any change, run the behaviour tests

Every fix in this article changes how the agent behaves, which is exactly what the Testing tab measures. A behaviour acceptance criterion, a BAC, is a situation you put the agent in plus a statement of what must be true about how it behaved.

1. Open the agent's Testing tab and press Run all.
2. Read the failures before you read the passes.
3. If your fix was for a problem no criterion covers, add one now, while you can still remember the exact situation that broke. Run it unpinned first to check it means what you think, then pin it.

Pinned criteria control releases. A new version of a published agent goes live only if every one passes, so a fix that breaks something else does not reach anybody. See Test an agent's behaviour.

Things to be aware of

• Changes on the Script tab, the System Prompt tab and Settings are live as soon as you make them.
• Changes on the Resources tab sit on a draft until you press Apply & make live. See What saving actually does.
• A conversation already in progress keeps what it started with, so a colleague mid-conversation will not see your fix until they start a new one.
• To stop something happening right now, revoke the operation on the connection. It checks this every time it runs, including mid-conversation.
• A stopped run does not resume. Re-running starts from the beginning and repeats anything not safe to repeat, so check what already went out before you press it.
• Change one thing at a time. Two fixes at once and you will not know which worked.

When it does not work

• You changed the wording and it behaves the same. The wording was probably not the cause. Work back through the nine, and check declared versus observed before rewording anything again.
• It passes your tests and fails in real use. Your tests are running against situations that are cleaner than reality. Take the invoice that actually failed and make a criterion out of it.
• It got worse. Roll back. Functions, scripts, widgets and the agent's resource assignments all keep versions, and rollback is re-activating an earlier one. See Change something that's already live.

You're done when

• You can name which of the nine causes it was, out loud, before you changed anything.
• The change was made in the Agent Builder on the tab the problem lives on.
• Anything that needed a consistent definition is a function's output, not a phrase in a prompt.
• Every pinned criterion passes, and there is a new one covering the thing that went wrong.