In a proactive move to ensure seamless team operations during an upcoming two-week absence, Dallas, a Forward Deployed Engineer at Airbyte, has engineered an innovative AI-powered solution designed to autonomously handle customer inquiries and internal requests. This sophisticated system, dubbed the "PTO Bot," eschews the traditional handover document in favor of a dynamic, two-agent architecture that mimics a rigorous internal review process, aiming to provide contextualized and accurate responses without direct human intervention. The initiative, detailed in a recent technical post, highlights a significant stride in leveraging generative AI for operational efficiency and employee well-being.
The Genesis of an Automated Handover
The concept for the PTO Bot emerged from a practical need. Dallas had initially planned to compile a comprehensive document detailing ongoing customer interactions and project statuses, a common practice to facilitate knowledge transfer during planned leave. This document had been meticulously updated since February, serving as a running log of processed customer calls and critical information. However, as the departure date neared and preparations for a trip to London and Scotland commenced, a more ambitious idea took root: what if the need for a human to "pick up threads" could be entirely eliminated?
"The idea was simple," Dallas explained in the original post. "I’m taking two weeks off, and whoever picks up my threads should have context without asking me for it." This initial objective evolved into a more sophisticated ambition: to build an agent capable of responding directly to colleagues on Slack, thereby freeing Dallas from the expectation of being available for urgent queries. This transformation from a static document to a dynamic AI agent underscores a commitment to continuous improvement and the exploration of AI’s practical applications in the workplace.
Inspiration from Multi-Agent Architectures
The architectural blueprint for the PTO Bot was significantly influenced by a paper from Prithvi Rajasekaran on the Anthropic Labs team. This research explored "harness design for long-running applications," proposing a multi-agent system inspired by Generative Adversarial Networks (GANs). In this model, a "generator" agent creates content, and an "evaluator" agent assesses its quality, iterating until a defined standard is met. While Rajasekaran’s use case focused on frontend design and full-stack coding, the core principle of separating the work producer from the work judge resonated deeply with Dallas.
"I wanted that same tension for my PTO bot," Dallas stated. The objective was not to introduce unnecessary complexity to Slack replies, but to instill a robust quality control mechanism. The fear of an AI agent confidently disseminating incorrect information to colleagues while Dallas was enjoying a golf course in Scotland served as a powerful motivator for this adversarial approach. This concept of a self-critiquing AI system is a key differentiator, moving beyond simple query-response models to a more nuanced and reliable interaction.
Leveraging a Rich Data Ecosystem
A significant advantage in developing the PTO Bot was the pre-existing integration of Dallas’s workflow with various data sources. Airbyte’s role as a data integration platform proved instrumental, connecting to systems such as:
- Granola: For meeting transcripts, providing conversational context.
- Linear: For tracking tasks and project statuses, offering insights into ongoing work.
- Pylon: For accessing customer threads, crucial for understanding client interactions.
- Notion: For documentation and knowledge management, serving as a repository for essential information.
- Slack: The primary communication channel, where the bot would operate.
These connectors, already in daily use for Dallas’s personal AI workflows, facilitated a straightforward integration into the PTO agent. Furthermore, a local folder, referred to as the "Jarvis," was incorporated. This personal repository houses notes from customer meetings, nascent ideas, and other contextually vital information not residing in standard SaaS tools.
The inclusion of Kapa.ai, an internal product knowledge platform, further enriched the bot’s capabilities. This allowed the agent to access and draw from official product documentation, such as details about the "Agent Engine," rather than relying on potentially inaccurate inferences or general knowledge. This deep integration of internal knowledge bases is a hallmark of effective enterprise AI deployments.
The Two-Agent Adversarial Architecture
The core of the PTO Bot’s functionality lies in its two-agent architecture:
-
The Draft Agent: This agent is equipped with the full suite of tools. It possesses the capability to access all connected data sources, conduct searches across replicated data, and compose initial responses. Its primary role is to gather information and formulate a coherent answer to a user’s query.
.jpg)
-
The Review Agent: In stark contrast, the review agent is intentionally stripped of all tools. It receives the draft response generated by the draft agent, along with all captured tool results, presented as plain text within its prompt. Its sole function is to provide one of two verdicts: "approve" or "revise."
The operational flow is as follows: when a query is received, the draft agent retrieves relevant information and formulates a response. This draft is then presented to the review agent. If the review agent deems the draft satisfactory, it returns an "approve" verdict, and the response is sent to the user. However, if the review agent identifies deficiencies, it returns a "revise" verdict along with specific, actionable feedback. The draft agent then incorporates this feedback and attempts to generate a new draft. This iterative process, known as an "adversarial loop," continues until the review agent grants approval, ensuring a high standard of accuracy and quality before any response is disseminated.
The provided code snippet illustrates this loop:
async def run_adversarial_loop(client, user_message: str, max_rounds: int = 3):
# Session-based – client preserves conversation history
await client.query(build_draft_prompt(user_message))
draft = await get_assistant_response(client)
for round in range(max_rounds):
review = await run_review(user_message, draft, captured_tool_results)
if review["verdict"] == "APPROVE":
return draft
# Only sends feedback – session already has the original context
await client.query(f"## Reviewer feedbacknreview['feedback']")
draft = await get_assistant_response(client)
return SAFE_FALLBACK # never returns a rejected draftA critical performance enhancement is the use of the ClaudeSDKClient session management. This preserves conversation history, meaning subsequent revision rounds only need to transmit the reviewer’s feedback, not the entire prompt and tool results again. This significantly reduces the data transfer and processing overhead for each iteration, making the loop remarkably lightweight.
The review agent’s stateless nature and lack of tools are deliberate design choices. This constraint forces the review agent to evaluate the draft based solely on the information provided. It cannot independently verify facts or access fresh data to compensate for any gaps or inaccuracies in the draft agent’s output. This adversarial setup ensures that the draft must withstand scrutiny on its own merits.
For quality assurance, both agents utilize Opus, the most advanced model available, to ensure the reviewer can be genuinely critical and not merely a passive approver. A crucial fallback mechanism is also implemented: if the loop exhausts its maximum rounds without an approval, the agent does not transmit the last rejected draft. Instead, it returns a "safe fallback" message, prioritizing accuracy and avoiding the dissemination of potentially erroneous information. This principle of erring on the side of caution, by communicating a fallback rather than a bad answer, is a key tenet of the system’s design.
Optimizing for Speed: The Data Layer’s Role
Contrary to expectations that a multi-agent, iterative process might be slow, the PTO Bot is designed for speed, primarily due to its sophisticated data layer. Airbyte’s replication of data from all connected sources into a central context store is the linchpin. When the draft agent needs to access customer threads or check Linear tickets, it queries this pre-replicated data rather than making sequential API calls. This bypasses the complexities of pagination, rate limits, and network latency inherent in direct API interactions. Consequently, tool calls return almost instantaneously.
In most scenarios, the entire adversarial loop resolves within one or two rounds. This efficiency is largely attributable to the draft agent having immediate access to comprehensive context, enabling it to generate a well-informed first draft more often than not, which the review agent can then readily approve. Direct API calls are reserved for situations where truly real-time data is indispensable, acting as an exception rather than the norm.
The Broader Technological Stack
The PTO Bot is built upon a robust and modern technology stack:
- AI Framework: Claude Agent SDK with ClaudeSDKClient for session management and agent interaction.
- Data Integration: Airbyte connectors and a context store for efficient data access.
- Hosting: Railway for reliable and scalable deployment.
- Memory Layer: Mem0 for maintaining conversational context across interactions, enabling the agent to "remember" past exchanges and build upon them during Dallas’s absence.
Anticipating the Real-World Test
With Dallas’s departure imminent, the PTO Bot has undergone internal development and deployment, with the traditional coverage document standing by as a contingency. The true test, however, lies in its performance over two weeks of continuous operation, fielding a barrage of real-time queries from colleagues.
Several key questions remain as Dallas prepares to monitor the system’s performance:
- Effectiveness of the Review Agent: How frequently will the review agent identify substantive issues that require revision, and what types of errors will it most commonly catch?
- Team Trust and Adoption: Will colleagues place confidence in the bot’s responses, or will they default to waiting for Dallas’s return? The success of such a system hinges not only on its technical accuracy but also on user acceptance.
- Memory Layer Efficacy: Will the Mem0 memory layer enable the agent to improve its responses over the course of the trip by learning from past interactions, or will its performance plateau?
Dallas plans to provide a follow-up report, likely from a pub in Edinburgh, detailing the real-world performance of the PTO Bot. This ongoing series, "Deployed with Dallas," chronicles his journey of integrating AI into his daily responsibilities as a Forward Deployed Engineer at Airbyte, offering valuable insights into practical AI implementation for enterprise environments. The successful deployment of this PTO Bot signifies a potential paradigm shift in how teams manage continuity and operational tasks, underscoring the transformative power of intelligent automation.
