Docs / Running and approving
Limits and run behaviour
For sizing a job before you build it, and for knowing what happens when a run stops half way.
The numbers, and why each one exists
| Roughly | Why it is there | |
|---|---|---|
| How long a run gets | About a minute when somebody is waiting for the answer, about five minutes when nothing is | An agent with a bad instruction does not fail. It keeps going. A ceiling turns a runaway into a stopped run you can read |
| How many times a run can reach outside | About 100 | The same problem, pointed at somebody else's system. Without a bound, one badly shaped loop becomes a thousand requests to a system that will notice |
| How wide a repeat across a list runs | 8 items at a time | Politeness to the far end. Eight in flight is fast without looking like an attack |
| How long a list a repeat accepts | 1,000 items | Past that, the unit of work is wrong. An oversized list is refused before anything goes out, rather than part way through |
None of these are about capacity. They are the size of the hole a mistake can make, which is why they are set where they are and why it is worth building to fit them rather than fighting them.
The two deadlines are the pair to remember. Something a person is sitting waiting for gets about a minute, because after a minute they have given up anyway. Something running on a schedule gets about five, because nobody is watching and finishing matters more than answering quickly.
Designing inside them: the invoice agent, worked out
Start by counting what one invoice actually costs. "Reaching outside" means one call to a system that is not Mindset: a finance system operation, a search of a knowledge base, a call out over a connection.
| Stage | What it reaches out for | Calls |
|---|---|---|
| 1. Gather | Fetch the invoice, fetch the purchase order | 2 |
| 2. Compare | The invoice-comparison function fetches the PO. Its other four steps work on data it already holds, so they cost nothing | 1 |
| 3. Assess | Searches of the supplier contracts knowledge base, usually two or three before it finds the clause | 3 |
| 4. Prepare | Registers the supplier query. Nothing leaves Mindset: a write is recorded and waits for a person, so the call itself happens after approval, outside the run | 0 |
| One invoice | about 6 |
Six, and say eight on a bad day where something is retried. That is comfortably inside the budget of about 100, and four stages of model work on one invoice finishes inside the five minute deadline without being close to it.
Now do 4,000 invoices in one run
4,000 invoices at six calls each is 24,000 reaches outside. The budget is about 100. The run would stop at the ceiling having worked roughly sixteen invoices, and in practice the five minute deadline would have ended it well before that.
So the job does not fit, and no amount of tuning makes it fit. There are two ways to fix it, and which one you use depends entirely on how much work each item needs.
Fix one: repeat across a list, where the work per item is small
A step in a function can repeat across a list: give it 200 invoice numbers and it runs once per number, eight at a time. A failing item is recorded as failed in the results and the rest carry on, rather than the whole run dying.
This is the right tool when each item needs almost nothing. A first pass that takes a list of invoice numbers and fetches the PO number for each one is a single call per item, and a repeat handles that neatly.
Items in a repeat count individually against the run's budget of about 100 reaches outside. A hundred invoices at one call each is your whole budget. So the 1,000-item ceiling is only actually reachable when the per-item work makes no outside call at all, which means data steps and model steps. If every item makes a call, plan for something under 100 per run, not 1,000.
A repeat fixes how long the work takes. It does not buy you more calls.
Fix two: smaller units on a schedule, where the work per item is not small
Six calls and four stages per invoice is not small. So the unit of work is one invoice, not the whole month.
For the invoice agent that means one run per exception: the finance system calls Mindset when an invoice fails its PO match, or a schedule picks up the next unworked exception every few minutes. Four thousand exceptions becomes four thousand short runs. Each one is six calls and well inside a minute. Each one has its own record, so a failure is one invoice you can look at rather than a batch you have to unpick. And each one can be re-run on its own.
If you would rather batch, work out the batch from the arithmetic. Ten invoices per run is 60 calls, which fits the call budget, but ten invoices through four stages will not fit five minutes. The deadline binds before the call budget does. Size against whichever ceiling you hit first.
Size for a slow morning. If the job only fits when every system answers immediately, it does not fit.
When a run stops
It stops where it is. Nothing resumes it, and the reason matters: resuming would mean Mindset deciding, on its own, that a half-finished set of changes to your systems ought to be completed. It cannot know whether that is safe, and if it guesses wrong you find out about it from the other system.
You get three things instead.
The record shows how far it got. Which stages completed, which operations ran, what came back, and anything still waiting on a person.
The same run cannot execute twice. Every run carries an identifier, and firing the same one again does nothing. That is what makes retrying safe at your end: something that is unsure whether its request landed can send it again without the invoice being worked twice.
Re-running is your decision, taken with the record in front of you. A re-run is a new run. It starts from the beginning and repeats anything that is not safe to repeat. Read what has already settled first, then decide whether repeating it is acceptable, or whether you want to narrow the input to just the part that did not finish.
Build so that a re-run is safe
- Make steps repeatable where you can. Running one a second time should have no extra effect.
- Put the things you cannot undo last, so a run that stops has stopped before the part that matters. This is the same advice as putting the change in the last stage, for the same reason.
- Remember that changes wait for a person. A run that stopped part way usually leaves an unapproved change behind it, which means the incomplete state is visible rather than silent.
The invoice agent is built this way on purpose. Stages one to three only read: they fetch the invoice, fetch the purchase order, compare them and search the contracts. Re-running all three costs a few calls and changes nothing anywhere. Stage four is the only one that registers a change, which is why it is last. A run that dies at stage three has cost you nothing but time.
Retries inside a function are a separate thing
Worth not confusing with the run deadline. A step inside a function can retry on its own: you set how many attempts in total, and which kinds of failure are worth retrying. Rate limits, server errors, timeouts and network failures are retried by default. A rejected request is not, because it will be rejected identically the second time.
That is per step, inside one run, and each attempt has its own timeout. The run's deadline sits over the top of all of it and does not extend to accommodate retries. Three steps each retrying three times is nine attempts happening inside the same minute or five minutes everything else has to fit into, and every attempt that reaches outside counts against the budget of about 100.
You're done when
- You can state the worst case for one run: how long it takes, and how many times it reaches outside.
- Both numbers sit comfortably inside the ceilings rather than just scraping under them.
- You know which ceiling your job hits first, the time or the call count.
- For each thing your agent changes, you know whether doing it twice would matter, and the ones that would happen in the last stage.