We’ve built a customer support agent for a popular fitness app (100k+ users).
Now it handles 250+ tickets/day.
It went from a simple few-week MVP to a platform which:
- integrates with the legacy customer support system users’ database and events;
- has an admin app to manage prompts & knowledge base;
- has external tools to make better decisions
- automated scenarios environment to test new prompts & data isolated from real users;
- includes AI agent observability & logging.
Now we are adding an extra AI agent to analyze AI messages before sending them to the user.
Table of Contents
How the project evolves in reality
? From the business perspective, it went through a lot of changes as well.
The client has shared a standard operating procedure (SOP) document from day 1, with:
The client has shared a standard operating procedure (SOP) document from day 1, with:
- a business context
- Examples of conversations with real users
- ideal and non-ideal conversations examples
As soon as the first version was tested in staging, we started a gradual rollout to real users.
The main communication channel is an email. To start, a small portion of new email tickets was forwarded to an AI agent. With first excitement from perfectly processed tickets, we also realised that a lot of real-world cases were not covered in the SOP document. We created a group chat with the client’s support team and our BA/devs. It helped to quickly implement each undocumented but desired use case.
Support AI Agent uses external tools to make a decision or write a better response. It works exactly the same as in Claude/ChatGPT – it picks the right tool when it needs it.
As we covered new edge cases and served bigger part of inbound emails, we got more issues. Apparently, integration with the client’s data storage was sending empty data for some users.
So in the Admin app, in addition to the full dialog view, we’ve added tool context and were able to quickly spot the issue.
So in the Admin app, in addition to the full dialog view, we’ve added tool context and were able to quickly spot the issue.
N.B. Client’s team has a very experienced project lead who worked in banks and knows how to roll out external systems. We appreciate the realistic expectations they have about system integrations & change management. This helped to avoid mistakes and rushing.
Model update
Once the AI agent got better, the support team decided to forward all inbound emails to it. For a few hours first. So we got more edge cases to solve.
This time, we have issues with messages in different languages. Sometimes a user may start a conversation in Spanish and then switch to English. Also, the user may start with one topic and remember another. Or be not very clear with intent.
The model we used (gpt-4o-mini) was super fast and cheap, but not optimal for complicated scenarios. So we’ve changed it to gpt-5-mini. To make sure we use all it’s capabilities, we’ve conducted a deep research and optimised our prompts for this model.
Templated answers → Knowledge base
Originally, the agent was built with:
- master prompt (who you are, what you are doing, and how)
- classification prompt (what type of request is it)
- templates for each request type
- human escalation mechanism
We started with templates, as we didn’t have a full knowledge base at the beginning and were building it together with a support team. And it was in the master prompt first. Once it grew up, it became not efficient to keep in there, so we’ve added a knowledge base section for each user’s intent. In this case, only relevant data is being sent to the LLM, which improves the responses, increases the speed, and saves tokens.
Having the intent, knowledge base, and instructions allowed us to remove hard-coded templated responses and use customised answers for each ticket.
Automated response evaluators
Once we handle more complicated support tickets and dialogs, prompts, and the knowledge base gets bigger and harder to manage. To be more confident that describing another edge case will not break what already works, we’ve added Evaluations. These are the special tests, dialogs that simulate typical scenarios between:
- AI Support Agent
- AI user (they have an intent, context, script to follow)
- AI Evaluator (this guy evaluates how good AI Agent handled the conversation)
And we’ve just started 🙂
