Crossing the production gap: five patterns for AI agents that survive Monday morning
Long-running agent workflows take days, span pods, pause for humans, and get audited. We unpack the five patterns — and the interoperability layer — we use when we engineer them for clients.
TL;DR
The workflows our clients actually care about — processing thousands of insurance claims, running a week-long sales sequence, reconciling a quarter of finance data, watching a Kafka topic for anomalies — do not fit in a single conversation turn. They take days. They span pods. They pause for humans. They survive crashes. They get audited.
A REPL-style agent loop with a vector store bolted on does not survive Monday morning. Five patterns and an interoperability layer do. Checkpoint-and-resume, delegated approval, memory-layered context with governance, ambient processing, and fleet orchestration — plus A2A for agent↔agent and MCP for agent↔tool/data. This is how we engineer long-running agents when we build them for clients.
What is the longest uninterrupted unit of work your agent needs to perform? Minutes — you probably don't need long-running agents. Hours or days — these patterns are where you start, and the governance and interoperability layers become load-bearing, not optional.
The production gap is real
Most agents that ship as demos die the moment the trigger stops being "a human typed in a chat box". When you instead point the same loop at a queue of 10k invoices, a multi-day onboarding workflow, or an overnight reconciliation, four things break almost immediately: state is lost when a pod dies, compute is held hostage while waiting for a human, memory drifts across runs, and every tool call becomes a place data can leak. Those are the failure modes we engineer against from day one.
Pattern 1 — Checkpoint-and-Resume: state that outlives the pod
"Treat your agent like a long-running server process, not a request handler."
The fix is conceptually simple and architecturally important: checkpoint progress, handle partial failures, ensure idempotency. The granularity matters — checkpointing after every document is wasteful, checkpointing only at the end is risky. For a 10k-document batch, somewhere around every 50 is usually the right balance.
How we build it. Our default stack uses a LangGraph supervisor running against a Redis-backed checkpointer, with graph state persisted at every superstep boundary. On top of that we layer a per-node delta log — keyed by thread, turn, and node — so every state mutation is attributable to the node that produced it. When an executor pod dies mid-run, the next pod replays from the last checkpoint with no user-visible failure and no lost work.
Pattern 2 — Delegated Approval: zero compute while waiting
A real workflow waits — for a signature, an underwriter, a security review, the customer to wake up in another timezone. The pattern: pause in place with full execution state intact, release compute, resume on event.
Most "human-in-the-loop" implementations amount to: serialize state to JSON, send a webhook, hope someone checks it. Notifications compete with dozens of other alerts. When the human responds, the agent has to deserialize, re-establish context, and hope nothing changed.
How we build it. Any tool that returns
{"status": "pending_approval"} or
{"user_input_required": true} is intercepted by a
HITL gate node. The graph state freezes, gets persisted to Redis,
and the worker pod immediately picks up other tasks. Hours or days
later, an approval message lands on the task stream and the graph
resumes from the same checkpoint — same conversation, same plan,
same intermediate results. Compute is genuinely zero while
paused.
This is how a "process every claim that came in overnight" agent can sit at 1,200 paused approvals over the weekend without costing anything, then resume the moment a human clicks approve.
Past a certain scale — twenty concurrent long-running agents per operator — Slack channels and email threads fail. You need a structured queue with three lanes: Needs your input, Errors, Completed. We design that surface into every long-running engagement.
Pattern 3 — Memory-Layered Context (and Why You Have to Govern It)
This is the pattern most teams under-think. Two memories, not one: working memory is fast and scoped to the current turn; long-term memory accumulates facts across sessions. Both have to coexist. The moment you let agents write to a shared memory store unchecked, you have a governance problem.
The risk has a name: memory drift. An agent learns from a handful of atypical interactions that a procedural shortcut is acceptable, and starts applying it broadly. Multiply that across several agents reading and writing to shared pools, and you have leakage between workflows that is hard to detect and harder to explain to a compliance team.
The architecture answer is three-part: cryptographic agent identity, a centralized agent registry, and policy enforcement at the boundary of every memory write. Working memory is a pruned message channel scoped to the current turn. Long-term memory lives in a vector store organized in three tiers — episodic, semantic, procedural — with an LLM-driven consolidation pass at session end and a policy hook on every write that runs PII detection and per-org deny rules before commit.
We pair that with the same posture on the tool-call side: every
request through our MCP layer carries a short-lived JWT, every SQL
call runs with row-level security pinned to the tenant
(SET LOCAL app.current_org_id), and any untrusted code
execution lands inside a gVisor sandbox.
Cross-tenant data exfiltration is structurally blocked, not just
discouraged.
Pattern 4 — Ambient Processing: triggered by events, not prompts
The shape of an agent changes when the trigger is no longer "a user typed something" but "a webhook fired", "a row appeared in BigQuery", "a support ticket came in", or "a metric crossed a threshold". These agents are mostly idle, multi-stream, and long-lived.
The architectural rule that makes them survivable is to externalize policy. Don't hardcode retry budgets, routing rules, or HITL defaults into the agent code. Hold them in a layer the agent reads at runtime. Update once, and the entire fleet picks up the new rules immediately — without a redeploy per agent.
Pattern 5 — Fleet Orchestration: a coordinator and her specialists
A coordinator agent decomposes work — research, scoring, sequencing, outreach, follow-up — and delegates each piece to a specialist running on its own timeline, with its own identity, its own tool permissions. The coordinator maintains global state and handles handoffs. Crucially: each specialist is independently deployable, so a bad release in one never cascades.
Our reference topology is a supervisor that routes among a roster of specialists (chat, SQL, Python, ML, platform, data, scraping, pipeline, integrations). The graph topology is explicit and graph-defined, not prompt-defined — which means the LLM can't shortcut its way around the structure. On top of that we run convergence detection to break out of agent loops automatically: if the same tool hash fires three times or any turn exceeds 25 tool calls, we cut over to graceful degradation rather than burn an unbounded budget.
Bonus — A2A + MCP: the interoperability layer
Most organizations won't build every agent from scratch. The real leverage is agents built by different teams — sometimes different companies — discovering and collaborating with each other. Two open protocols carry that weight:
- A2A standardizes agent↔agent communication. Every A2A-compatible agent publishes a card at a well-known URL describing its capabilities, auth, and rate limits. A central registry becomes the service mesh for the agent ecosystem.
- MCP standardizes agent↔tool/data communication. A Stripe connector looks the same to an agent as a BigQuery connector — the protocol is the interface, the backend is interchangeable.
Our entire tool execution path is MCP. Specialist agents call an MCP server, which validates a per-request JWT, applies RLS, and sandboxes code execution. Customer-installed MCP servers register dynamically — your team ships a tool, the supervisor picks the capability up at load time, and the agent has a new skill without anybody shipping framework code.
The consumer side — discovering and calling external A2A agents — is where this stack stops being purely internal. A well-defined governance layer there controls what data leaves the boundary and what responses we are willing to act on. That layer is the same runtime-policy plane that ambient processing relies on (Pattern 4): one place to encode "what's allowed", read by every agent.
How to choose
The patterns compose. A compliance system uses checkpoint-and-resume for document processing, delegated approval for review gates, memory-layered context with governance for cross-session knowledge, ambient processing for inbound triggers, and fleet orchestration to coordinate specialists. Pick patterns by the failure mode they prevent, not by their novelty.
Ask the diagnostic question first: what's the longest uninterrupted unit of work your agent needs to perform? If it's minutes, save yourself the engineering. If it's hours or days, all five patterns plus the interop layer become load-bearing — and the cost of skipping any of them shows up in production within weeks.
Teams shipping isolated, stateless agents today will be refactoring in twelve months. Teams building with persistence, governance, and interoperability in mind compound their advantage every day.
That's the bar we build to in every engagement. If you're staring at a workflow that needs to run for days, span pods, pause for humans, and survive Monday morning — we should talk.
Credit
The "production gap" framing and the five-pattern enumeration are adapted from The Production Gap: 5 Patterns for Building Long-Running AI Agents by Addy Osmani and Shubham Saboo on Turing Post. The architecture choices, file-level decisions, and tradeoffs in this post are the ones we make when we build agent systems for clients.