Engineering
Your AI agent timed out. Did it change the system?
An engineering guide to approvals, retries and uncertain outcomes in enterprise agents.
An AI agent submits a change to a purchase order. The ERP accepts it. Then the connection drops before the response reaches your application.
Your application sees a timeout. The ERP has already changed.
Should the agent retry?
That question takes us beyond prompting and into the part of enterprise AI that deserves much more attention: controlling what happens when a model’s decision becomes a business action.
Even a seemingly harmless “set the delivery date” operation may trigger notifications, supplier workflows or audit events. Repeating the same field value does not necessarily mean repeating the request is harmless.
Microsoft’s October 19 security guidance makes a useful architectural point: probabilistic agents need deterministic controls around them. For enterprise teams, that means treating authorization and execution as application responsibilities. A model’s confidence is not permission to write. Microsoft Security
Here is how I would structure a narrowly scoped agent that proposes changes to purchase-order delivery dates. The accompanying Python exercise makes the failure cases reproducible. It is a local teaching example, not a claim about a production deployment.
01
Give the model a proposal interface.
The model can read the allowed context and propose an action. A trusted application decides whether that proposal can become an executable command.
That separation should exist in the code and the credentials, not only in a system prompt.
The model might propose:
{
"action": "change_delivery_date",
"target": "PO-4821",
"delivery": "2026-11-06"
}The application then constructs the execution envelope. It assigns a stable action ID, derives the tenant and actor from the authenticated session, reads the current record version and attaches the applicable policy version.
Do not ask the model to decide which tenant it belongs to. Do not accept a model-generated claim that an approver has signed off.
Validate the proposal in layers:
- Structure: Is this an allowed action with correctly typed arguments?
- Business rules: Is the proposed date permitted for this order and supplier?
- Authority: Can this actor change this record for this tenant?
- Evidence: Does the request have sufficient supporting information?
Valid JSON answers only the first question. A schema-valid request can still be unauthorized or wrong.
02
Bind approval to the exact action.
“Approved by a human” is incomplete unless you can say what the human approved.
Store the exact normalized action with its tenant, target, arguments, expected record version, policy version and expiry. Bind approval to that stored content. A later argument change should invalidate the approval and create a new action requiring review.
The companion uses a SHA-256 digest of consistently serialized JSON to detect changes. That is a content-binding demonstration, not a digital signature or an identity system. A digest cannot protect approval records from an attacker who can rewrite both the content and the digest.
In production, protect the approval store, authenticate the approver, record the decision and enforce separation of duties where required. If multiple services compute the binding, define a shared canonical representation rather than assuming their JSON serializers agree.
Recheck authority before dispatch. An approval from yesterday should not silently override today’s revoked access or policy change. Where revocation must take effect immediately, enforce it at the trusted gateway or receiving service too; an earlier application check leaves a race.
03
Persist intent before sending the request.
Chat history is not an execution ledger.
You need a durable record of what was approved, whether an attempt started and what outcome is known. Temporal’s agent-state example usefully separates conversational context, execution state and the state of external effects. Those are different forms of truth. Temporal
A minimal state model looks like this:
APPROVED → IN_FLIGHT → SUCCEEDED
→ REJECTED
→ UNKNOWN
UNKNOWN / IN_FLIGHT → reconcile
→ SUCCEEDED, if a receipt exists
→ NEEDS_RECONCILIATION, otherwiseBLOCKED is a separate pre-dispatch outcome for expired approval, changed content or failed authorization.
Persist IN_FLIGHT before calling the external system. If the process dies afterward, another worker can see that execution may have started.
That does not tell you whether the remote change happened. It tells you that you must resolve the uncertainty.
Durable workflow tooling helps preserve progress across worker failures. It does not make every external operation safe to repeat. Temporal activities can be retried after timeouts, so the external effect still needs an appropriate retry contract. Activity execution
04
Make retries refer to the same business intent.
An idempotency key should identify one logical action across its attempts.
Generate it when the application creates that action. Persist it. Reuse it when retrying the same approved intent.
A fresh UUID on every attempt defeats deduplication. A hash of the prompt is also a poor business identity: two separately authorized requests can have identical wording.
The receiving service must enforce the contract. In the companion, it atomically commits the business update and a receipt keyed by tenant and action ID. Repeating that key with the same content returns the recorded result; using it with different content is rejected.
AWS describes this caller-provided request identity pattern and the importance of recording the identity atomically with the mutation. It also explains why changed parameters and key-retention windows require explicit handling. AWS Builders’ Library
Before relying on a vendor API, establish:
- Which operations support idempotency, and how are keys scoped?
- What happens when a key is reused with different arguments?
- How long are keys retained?
- Can you retrieve the original operation’s status or receipt?
A local database wrapper around an arbitrary SaaS call cannot guarantee that the SaaS effect and your local receipt commit together. If the provider offers neither deduplication nor reliable status lookup, automatic retries of consequential writes may be inappropriate.
A transactional outbox can help deliver a persisted command. It does not, by itself, resolve whether a remote recipient already executed that command.
05
Protect against concurrent edits separately.
Idempotency and concurrency control solve different problems.
Suppose the order was version 17 when approved. A buyer changes it to version 18 before the agent executes. The original approval may now be stale.
The receiving system should check the expected version as part of the write itself:
UPDATE orders
SET delivery = :delivery, version = version + 1
WHERE tenant = :tenant
AND id = :target
AND version = :expected_version;If no row matches, stop and reassess. Do not quietly substitute the latest version and reuse the old approval.
For APIs that support it, HTTP If-Match provides a conditional-write mechanism using entity tags. RFC 9110 describes its use in preventing lost updates. Confirm your specific API’s semantics; an ERP may expose a version field instead. RFC 9110
A separate “read version, then write” check is insufficient if another writer can intervene. The condition must be enforced at the receiving write boundary.
The companion checks for a previously committed receipt before evaluating the version condition. Otherwise, a successful retry could incorrectly look stale because the first attempt already incremented the version.
06
Treat uncertainty as a first-class outcome.
After a lost response, query the receiving system using the persisted action identity.
If a matching success receipt exists, record success locally. Do not resend the mutation just to obtain another response.
If no receipt is visible, that is not automatically proof of failure. The original request might still be running, or the status interface might lag. In this example, the action stays in NEEDS_RECONCILIATION and makes no further write.
Production behavior depends on the provider’s guarantees. You might safely retry the same key within its retention window, poll a documented status endpoint or route the action to an operator. Define the choice per operation.
Avoid presenting this as universal “exactly once” execution. Delivery attempts, receiver-side deduplication and downstream business effects each have their own boundaries.
Cancellation deserves similar precision: requesting a stop cannot undo an effect that has already committed. Any compensating action needs its own business rules and authority.
07
Test the boundary without involving the model.
The companion uses two separate SQLite databases: one for the application’s action ledger and one for the simulated remote ERP. It requires only Python’s standard library.
Its 11 tests pass, including these cases:
| Failure or condition | Required behavior |
|---|---|
| Remote commit succeeds; response is lost | Reconcile the receipt; one mutation |
| Worker process exits after remote commit | A new process can reconcile the durable state |
| Attempt stops before sending | Missing receipt does not trigger an automatic write |
| Approved content, expiry, policy or authority changes | Block dispatch |
| Wrong tenant or stale record version | Prevent the unauthorized or stale write |
| Concurrent dispatchers or repeated remote requests | Preserve one mutation for the logical action |
The process-crash test actually terminates a child process after the remote commit. It then checks the persisted outcome. That is more meaningful than asking the model to describe what it would do after a crash.
Download the exercise, unzip it, and run it from the companion directory:
python3 -m unittest discover -s . -vThese tests do not establish production readiness. The example omits real identity, secure approval capture, network transport, distributed leases, provider retention policies and operational recovery tooling. Its atomic receiver transaction is an explicit simulator assumption. It also does not test whether the model proposes the right action; that requires separate task evaluations and adversarial input tests.
What enterprise teams can do with this.
Start with one narrowly defined write operation whose result you can independently verify.
Before giving an agent access to it, document the approval payload, execution identity, conditional-write mechanism, retry contract and reconciliation path. Then inject failures at the boundary between the local ledger and the receiving system.
Operationally, track unresolved actions and their age, reconciliation outcomes, duplicate suppression, stale-version conflicts and blocked dispatches. Keep enough audit information to connect a proposal to approval, attempt and receipt without indiscriminately copying sensitive source material into logs.
The useful question is not only whether an agent can call a tool. It is whether your team can explain, recover and audit the resulting business action when the response never arrives.