4 Critical Lessons from 3 Months Running Agentic AI

Key Takeaways

• Inbox architecture outperforms task lists for visibility and error surfacing
• Dynamic routing between local and cloud models balances cost and reliability
• Self-healing systems reveal failures rather than hide them—a feature, not a bug
• Human oversight remains essential even in automated workflows—someone needs to call the critical plays

Lesson 1: An Inbox Beats a Task List

A list's core advantage is visibility: you see everything at once. The workflow shifted from Asana to Gmail, with an agent managing the queue. Completed tasks archive automatically under a processing label; unfinished work or tasks requiring input return to the inbox.

This approach eliminated silent failures. Previously, 24 threads rotted under an error label with no context. Now they resurface with a one-line explanation. The inbox itself became the "waiting on you" surface—a visual pressure point that demands attention.

Lesson 2: Routing Between Local and Cloud Is Essential

A single model now acts as router, directing tasks across three tiers: a fast local worker, a local reasoner, and cloud fallback. Local execution averages 4-6 minutes per task and costs less. Cloud runs in about 39 seconds but carries higher costs. The router's job: decide what each task can afford and log the true reason for cloud promotion—never fake a local failure to justify escalation.

This transparency revealed which tasks genuinely needed cloud resources versus which were incorrectly prioritized locally.

Lesson 3: Self-Healing Works—But It Surfaces More Errors First

For six weeks, the error rate was zero. Then it climbed to 15%, then 34%. The rise did not signal system degradation; it signaled the end of hidden failures.

Before self-healing fixes, 21 threads remained stranded in error—the oldest for 18 days—with 112 manual SQL recoveries. After implementation, failures surface immediately, trigger four automated retries with jittered backoff, then dead-letter. Today zero threads sit in error.

The system now reverts its own bad deploys (65 so far; 42 because unit tests failed). Since late July, broken code is caught before deployment rather than after—broken main never ships. Yet some failures still need humans: auth token refreshes, memory limit overruns, and models that narrate instead of execute. Surfacing an error is not the same as fixing it.

Lesson 4: You Still Need a Quarterback

The workflow evolved from a single monolithic prompt into a graph of contracted nodes. The model produces narrow JSON outputs: intent, action, domain, reason, confidence score. Deterministic code executes writes. An independent verification node re-reads to confirm. Maker and verifier nodes are separate—never ask the agent to grade its own homework.

This separation enables plays without huddles. But it does not eliminate the quarterback. Someone still decides which tasks go to cloud, which failures escalate to humans, which deploys revert. Three months in, that quarterback is still human, sitting one layer above the code that used to be the entire team.

Conclusion

Running agentic AI reveals that automation and human judgment are not opposing forces—they are complementary layers. Systems work best when they surface truth clearly, route intelligently, repair transparently, and preserve human decision-making where it matters most.

---

Original source: Four Lessons From Three Months Inside An Agentic Harness

powered by osmu.app