Thank you! You're subscribed!
Oops! Something went wrong while submitting the form.
You’re taking the right steps to stay ahead in the world of AI. Your company has an AI strategy, you’ve done the homework, and you’re ready to run a proof of concept (POC) for an AI solution. But if your POC isn’t set up properly, the results could be disappointing — think “garbage in, garbage out.” To help you make your AI POC a success, we’ve compiled lessons from real-world examples and distilled them into actionable advice.
AI can transform many areas of support, from automating responses to empowering agents and uncovering actionable insights. Because you can’t evaluate everything at once, it’s essential to focus your POC on a specific use case that aligns with your CX goals.
Best for: High-volume, low-complexity inquiries, like FAQs or order tracking
Watch outs: Balance internal testing and low-stakes customer-facing deployment
Metrics: First response time, automation rates
If your #1 priority is reducing repetitive work and increasing efficiency, a deflection-focused POC might be the best choice. At their best, AI agents can automate certain customer interactions by delivering direct answers via chatbots or voice tools, leveraging resources like your knowledge base, help center, and previous tickets. Since the outcome of a deflection-centric POC is an application that will directly touch customers, think about how you plan to test this with the right guardrails.
Best for: Complex tickets that require human oversight or for upskilling agents with varying levels of expertise
Watch outs: Pay attention to change management and agent adoption of new workflows
Metrics: Average handle time, resolution rates, or ease of collaboration between agents and AI
If your company’s support strategy emphasizes human expertise, an agent-centric POC may be the right focus. By focusing on agent empowerment rather than full automation, an agent-centric POC allows you to validate whether AI can elevate the human-driven support experience your company values most. Copilots act as behind-the-scenes partners, drafting responses, providing relevant knowledge, or automating routine tasks so agents can focus on more complex or emotionally sensitive cases. Pilot the tool with a select cohort of agents (skill-level, seniority, focus area) to understand how it supports varying skill levels and work styles.
Best for: Teams looking to extract data to inform future strategy, investment areas, etc
Watch outs: Ensure leadership buy-in to evaluate and act on the insights generated
Metrics: TBD depending on project objectives
If your goal is to surface actionable insights from support data, consider focusing the POC on AI’s ability to analyze interactions and uncover trends, patterns, and themes that might otherwise go unnoticed. It’s a great way to get value from the support data that already exists in your company. An insights-centric POC might only involve a small group of people, but it’s likely to engage higher levels of leadership to evaluate the impact of the insights that are surfaced.
The best way to measure the success of an AI POC is to start by identifying the metrics that matter most to your team. While the full benefits of AI will only become clear after a full implementation, a well-structured POC should still deliver noticeable improvements. Keep in mind that the scope of these changes will depend on the duration of your pilot. A 90-day POC provides more time to evaluate and impact key metrics compared to a shorter 30-day POC.
It’s very tempting to look at “accuracy” and “quality” as the only benchmarks for an AI POC. After all, what matters most is getting the right answer to every customer, every time. Be wary of focusing exclusively on quality/accuracy when evaluating an AI solution in a POC. Not only is high quality table stakes, but it’s also changing quickly, as underlying LLMs improve almost daily. This means that any random sample of responses to the same ticket can have slight differences that might make them appear a bit “better” or “worse.” Over-scrutinizing here can detract from the bigger picture: operational impact and scalability.
30 days should give your team enough time for a technical evaluation of the AI tool. You should get a sense of how easy it is to configure, whether or not you’ll have the control you need, and an idea of how to improve quality over time.
In a short POC, focus on evaluating the technical capabilities of the AI tool and its ease of adoption with a small group of participants. Testing too broadly over a short timeframe may actually have a negative impact on support as many people will be trying to learn something new, all at the same time. Remember to align your success metrics to the kind of AI POC (copilot, deflection, etc) you’re running.
Here are some ideas for success metrics during a 30-day AI POC:
A 90-day AI POC requires a greater time investment but offers your evaluation team a deeper understanding of how AI can integrate into your operations. Beyond conducting a basic technical assessment, this extended timeline allows you to configure and test customer-facing automations thoroughly. For customer-facing AI use cases, a 90-day period provides the necessary time to evaluate impact, make adjustments, and simulate a real-world AI implementation process.
Participant selection is just as crucial for a successful 90-day POC. Focus on time-boxing training and change management efforts to streamline the process while gathering meaningful feedback. For automation-focused pilots, select participants who have a foundational understanding of AI to help accelerate setup and testing.
Here are some ideas for success metrics during a 90-day AI POC:
At the end of a POC, you should have the information you need to clearly answer the question of whether or not it was successful. With companies running multiple AI POCs at once, it’s even more important to define success metrics early and make sure they’re aligned to company priorities. Taking the time to define success before you start and align with your technology providers will make sure you’re taking the mystery out of AI and setting up your support function for success!