My last gig, a SaaS startup, hit that predictable wall: growth meant more users, more users meant more questions, and more questions meant our small support team was drowning. We were spending too much time on repetitive issues, the kind that could easily be answered by a well-indexed FAQ. “Can AI help with ticket deflection?” became the urgent question. We needed the best AI for ticket deflection, not just another chatbot that annoyed customers and added to the problem.
The Early Attempts and Why They Broke
We started simple, trying to build something in-house. The idea was to catch common questions before they even hit a human agent. We used the Vercel AI SDK, hooked it up to a basic RAG (Retrieval Augmented Generation) setup, and fed it our entire documentation, our FAQs, and a few thousand resolved support tickets. On paper, it felt promising. We thought we could just point it at our knowledge base and let it learn. What we got, initially, was a glorified search engine that sometimes hallucinated. It’d give confident, wrong answers to questions like, “How do I integrate with Salesforce?” when we didn’t even have a Salesforce integration. That’s worse than no answer at all; it erodes user trust and often leads to even more frustrated follow-up tickets.
Debugging these silent failures was a nightmare. We’d see a user interaction log where the bot gave a bad answer, but understanding why it went wrong was opaque (and often, a huge time sink). Was the RAG pulling the wrong chunk of text? Was the LLM misinterpreting the prompt? Was the vector database indexing poorly? We eventually integrated LangSmith to trace the steps of the agent’s reasoning. It helped us pinpoint issues, showing us which document chunks were retrieved and how the model processed them. But even with LangSmith, fixing the underlying model’s tendency to invent facts or misinterpret context was a constant battle. It highlighted a core truth: building a production-ready agent from scratch, with all the necessary guardrails, monitoring, and versioning, was a full-time job for an entire engineering team. We didn’t have that luxury.
Platform Solutions: The Good, The Bad, and The Price
So, we moved to platforms. The promise of “out-of-the-box” AI support felt like salvation. Intercom, for instance, has its Fin AI agent. We already used Intercom for live chat and email support, so integrating Fin felt like a natural step. It promised to answer questions using our help center content and even perform basic actions like updating user profiles. The setup was straightforward enough; point it at your knowledge base, tweak some settings, and let it learn. This is where the rubber met the road.
What actually worked, and what didn’t, became clear quickly. My concrete love for Fin was its ability to handle truly common, well-documented questions. “How do I reset my password?” “What’s your refund policy?” “Where can I find my invoice?” These were deflected consistently, freeing up our agents from the most repetitive inquiries. It reduced our tier-1 ticket volume by about 15% in the first month, which was a tangible win. That’s real money saved in agent time.
My concrete gripe, however, was its struggle with nuance. The moment a question required combining information from two different articles, or understanding a user’s specific account context beyond what was explicitly in the knowledge base, Fin often struggled. For example, a user might ask, “I’m on the Pro plan, but I can’t access feature X. Why?” If “feature X” was only available on the Enterprise plan, and the bot had to cross-reference the user’s plan and the feature matrix, it would often get stuck in a loop asking for clarification, burning user patience. Sometimes it would just punt to a human, which is fine, but sometimes it would give a generic, unhelpful answer. The customization options felt limited. If you wanted it to do something truly specific, like check a user’s subscription status via an API call to our internal billing system, you were often out of luck without significant custom development or a different tool entirely. The pricing, too, felt a bit steep for the level of customization we needed. We were on a plan that cost around $499/month for our usage, and honestly, for that price, I expected more granular control over its behavior and better integration with our internal systems.
Beyond Chatbots: Automating Actions with AI
Beyond simple Q&A, we also needed to automate actions. This isn’t strictly “ticket deflection” in the Q&A sense, but it’s crucial for support automation and reducing agent workload. For these scenarios, where we needed to actually do something, not just answer, we explored tools like n8n. We used n8n to build workflows that would, for example, detect a “bug report” keyword in an incoming email, then automatically create a Jira ticket, pull relevant user data from our CRM, and notify the engineering team in Slack. This kind of support automation tool is a different beast than an AI chatbot review, but it’s equally vital for an efficient support operation. It handles the grunt work that agents used to do manually, ensuring consistency and speed.
We also experimented with agent frameworks for internal agent tooling. For example, building a “triage agent” using LangGraph. The idea was for this agent to read an incoming ticket, classify it by urgency and topic, extract key entities (like user ID or error codes), and then suggest a response template or even draft a reply. This required a lot more engineering effort than configuring a platform. We had to define the agent’s steps, its tools (like a CRM lookup tool or a knowledge base search tool), and how it would reason through a problem. The benefit was complete control over the agent’s logic and behavior, allowing for highly specific and complex workflows. But the cost in developer time and ongoing maintenance was significant. Monitoring these multi-step reasoning chains became critical. Langfuse became essential here for observability, helping us understand the token usage, latency, and success rates of each step. Without it, understanding why an agent failed or got stuck was like looking for a needle in a haystack. It’s not just about getting an answer; it’s about getting the right answer, consistently, and being able to audit how that answer was reached, especially when dealing with sensitive user data or financial transactions. Governance and audit trails aren’t optional in production.