Articles in this section

Agent QA Guide

Josephine Romulo
Updated

This article explains how to use Agent QA in Discover to perform quality assurance on agent performance using customizable, AI-powered rubrics.

What is Agent QA?

Customers lack a centralized way to assess and ensure the quality of their human agents’ performance and interactions. Without a systematic QA tool, they often rely on time-consuming manual reviews or switch between multiple platforms to gather agent-specific metrics. This fragmented process makes it difficult to maintain service consistency, identify coaching opportunities, and improve overall support efficiency.

Agent QA addresses this challenge by providing centralized access to agent performance data—enhanced with AI-generated quality assurance insights. It enables you to define custom QA rubrics based on your team’s standards (e.g., resolution quality, technical knowledge, grammar) and apply structured scoring rules at scale.

Once your help desk is connected to Forethought, Agent QA automatically analyzes agent responses using default metrics. You can then create your own QA rubrics to generate consistent, scalable, and actionable insights—without the need for manual QA work.

Note: It may take up to 24 hours for custom rubrics to be applied to tickets created after activation.

Use Case 1: Measuring Agent Performance at Scale

An organization with 100 support agents wants a more efficient way to track their agent's performance. Unlike standard help desks that offer only basic Key Performance Indicators (KPIs), they aim to create custom KPIs tailored to their standards. To achieve this, they connect their help desk to Forethought, which indexes key data, including ticket volume, resolution status, agent details, and more.

Using Agent QA, the organization creates custom rubrics, such as resolution quality and grammar accuracy. Agent QA then automatically applies these rubrics to agent responses, delivering consistent, scalable, and actionable insights into their agents’ performance.

Use Case 2: Coaching Agents with AI-Powered Insights

Building on the previous example, a support team manager wants to improve the overall quality of customer interactions by identifying targeted coaching opportunities for individual agents. After setting up custom rubrics in Agent QA, such as grammar accuracy and resolution quality, the manager reviews AI-powered insights applied to recent support tickets.

For example, if an agent consistently loses points in grammar issues or resolution quality, the manager can use these QA findings to schedule focused 1:1 coaching sessions. Over time, they track progress by monitoring trends and score improvements across these rubrics.

By using Agent QA as a centralized quality assurance system, the organization fosters agent development, improves service consistency, and boosts overall customer satisfaction.

Setup

This section guides you through setting up Agent QA and creating a custom rubric.

1. Connect Your Help Desk

To begin using Agent QA, first connect your help desk to Forethought:

  • Go to Settings > Integrations.
  • At the upper right side of your screen, click + Connect new integration.
  • Search for your help desk (e.g., Zendesk, Salesforce, etc.).
  • Click your help desk and follow the on-screen instructions to complete the integration.

2. Add AI QA Rubrics

Once your help desk is connected to Forethought, you can start creating AI-powered rubrics to perform quality assurance on support interactions between your agents and customers. There are two types of QA rubrics you can create:

  • Agent Metrics – assess how support agents handle tickets based on your internal standards.
  • User Metrics – assess customer behavior or sentiment in response to the support provided.

These rubrics allow you to tailor QA scoring to what matters most to your organization.

To get started:

  1. Once you have successfully connected to your help desk, navigate to Discover > Agent QA.
  2. In the upper right corner, click QA rubrics.
    QA rubrics.png
  3. Select either the Agent metrics or User metrics tab.
    agent and user metrics.png
  4. Click + Create New to build your custom rubric.

Agent Metrics

Agent metrics enable you to perform quality assurance on how effectively your support agents manage tickets. You can measure their performance against your organization’s standards, such as resolution quality, technical accuracy, communication clarity, and adherence to internal processes.

How to configure:

Configure your metrics such as:

  1. Metric Name: Provide a clear and descriptive name for your metric.
  2. Metric Definition: Define the criteria the AI will use to assess agent performance. For best practice, use structured, specific rules.
  3. Scoring Definition: Describe how the AI should calculate the score based on your defined rules.

Example of a custom Agent metric:

Metric Name: Resolution Quality

Metric Definition:

This metric measures whether the agent provided a clear, complete, and accurate resolution to the customer's issue within their public replies. Focus strictly on the presence or absence of key resolution elements stated in the criteria below. Do not evaluate grammar, tone, or formatting unless it directly impacts clarity.

 

Criteria for Evaluation:

  1. The agent directly answers or addresses the customer’s main question or issue.
  2. The resolution includes necessary next steps, links, or instructions (if applicable).
  3. The resolution is factually accurate based on available information.
  4. There is no misinformation or irrelevant content that could confuse the customer.
  5. The response avoids excessive technical jargon unless the customer has already demonstrated familiarity.

Scoring Definition:

  • Evaluate only the public agent messages in the thread.
    • If there is no public agent message, assign a score of “N/A.”
  • Start with a score of 100.
  • Deduct 20 points for each of the following resolution issues found (up to 100 total):
    • The agent did not directly address or resolve the issue.
    • The resolution lacks key steps or required follow-up info.
    • The information provided is inaccurate or misleading.
    • The message contains unnecessary or confusing content.
    • The message uses overly technical language not aligned with the customer’s level of understanding.

 

Best Practice:
To ensure visual consistency and better chart readability across the platform, we highly recommend using a 0–100 scoring range whenever possible.

User Metrics

User metrics help you perform quality assurance on the customer experience by assessing sentiment, engagement, or behavior in response to a support interaction. These insights help you understand how users perceive the support they received and whether their needs were effectively addressed.

How to configure:

Configure your metrics such as:

  1. Metric Name: Provide a clear and descriptive name for your metric.
  2. Metric Definition: Define how the AI should analyze user behavior or sentiment. For best practice, use structured, specific rules.
  3. Scoring Definition: Explain how the metric will be scored based on user engagement or sentiment.

Example of a custom User metric:

Metric Name: User Engagement

Metric Definition:

This metric measures whether the user dropped off before receiving sufficient instructions or information. A confirmed resolution in the conversation is not required.

Scoring Definition:

Yes, the user dropped off  (Score:  1 - Low Engagement)

Examples:

  • The user stopped responding after being asked for more information to proceed to resolution.
  • The user reopened the ticket with unrelated issues post-resolution, which were not addressed.

No, the user did not drop off (Score: 5 – High Engagement)

Examples:

  • The user confirmed the solution worked or expressed satisfaction (e.g., "Thanks, that fixed it!").
  • The user indicated no further help was needed (e.g., "All good now").
  • The user explicitly closed the loop (e.g., "Thank you" with no outstanding questions or concerns").
  • The user did not respond after receiving sufficient instructions or information, which was interpreted as silent acceptance.

 

Important:
In this example, we’re using a 1–5 scale for scoring. While you can set custom score ranges in your Scoring Definition, please be aware that non-standard ranges like 1–5 may not display well on radar charts.

 

Best Practice:
To ensure visual consistency and better chart readability across the platform, we highly recommend using a 0–100 scoring range whenever possible.

3. Test and Save Your Metric

  • Enter a ticket ID into the test field.
    test.png
  • Click Test to generate a sample evaluation and see how the metric is applied in evaluating agent responses.
    soft skills.png
  • For better accuracy, test at least 5 different ticket IDs to ensure consistent performance across various cases.
  • Once you're satisfied with the results, click Save to finalize and activate the metric.

Important: New QA rubrics will only apply to tickets received after the rubric is created. Past tickets will not be analyzed retroactively.

Reviewing AI-Powered Insights in Agent QA

The Agent QA dashboard offers valuable insights into your support team's performance using both default and customized rubrics. Here’s how to navigate and understand the dashboard.

Default Metrics

QA rubrics2.png

Once you connect your help desk and enable Agent QA, the following metrics will automatically appear at the top of the dashboard:

  • Total Received Tickets – The number of tickets received in your help desk for the selected time period.
  • Tickets Assigned – The number of tickets assigned to agents.
  • Resolved Tickets – The number of tickets marked as resolved. The resolution rate is calculated by dividing resolved tickets by assigned tickets.
  • First Contact Resolution – The number and percentage of tickets resolved in the first response.
  • Average Full Resolution Time – The average time taken to fully resolve a ticket.
  • Average Time to First Response – The average time taken for an agent to send the first response.

Agent Performance Comparison 

AI QA Tab

After adding your custom rubrics, the AI QA tab displays a time series comparison chart:

  • Y-axis: Scores across custom metrics.
  • X-axis: Time (daily, weekly, monthly, or quarterly).
  • Filters: Use filters to adjust the time range.

Tip: Click on a data point to view detailed agent insights.

Ticket Tab

In the Ticket tab, default metrics are plotted over time:

  • Y-axis: Number of tickets or time in seconds.
  • X-axis: Time period.
  • Filters: Select daily, weekly, monthly, or quarterly views.

Tip: Click on a data point to view detailed ticket details.

This view helps you monitor ticket volume and response/resolution times at a glance.

Agent Insights and Radar Charts

Below the charts, you’ll find agent summary cards, which include:

  • Agent Name
  • Number of Solved Tickets
  • Average QA Score
  • Worst Performing Topic
  • Radar Chart with custom rubric scores

By default, these cards are sorted by the number of solved tickets.

Understanding the Radar Chart
radar chart.png

Each radar chart provides a visual breakdown of an agent’s performance across all custom QA rubrics:

  • Each axis represents a different metric.
  • Higher scores extend further toward the edge.
  • A balanced, expansive shape indicates well-rounded performance.

Radar charts help managers easily identify strengths and areas for targeted coaching.

Viewing QA Insights for a Specific Agent

You can view detailed QA insights for any agent by clicking their card on the Agent QA dashboard. This gives you a full breakdown of how they're performing based on your custom QA rubrics.

Agent Overview

At the top of the page, you'll see:

  • A radar chart that shows the agent's performance across your custom rubrics (both agent and user metrics). Each line shows how well they’re doing in key areas like resolution quality or grammar.
  • A card showing the least efficient topics—these are topics with the lowest first contact resolution rates. They help you spot areas where the agent might need extra support or training.

Default Metrics

Below the overview cards, you'll find default performance stats, such as:

  • Tickets assigned
  • Tickets solved
  • First contact resolution rate
  • Full resolution time
  • Time to first response

These numbers give you a quick snapshot of the agent’s workload and response times.

Performance Over Time

Next, you'll see a time series comparison chart that shows how an agent’s QA scores change over time. It tracks their performance across your custom QA rubrics and compares it to the previous time period.

You can use this chart to:

  • Monitor trends in performance
  • Spot improvements or areas that need attention
  • Filter results by day, week, month, or quarter to view short- or long-term progress

This view makes it easy to see how agents are performing over time and identify when additional coaching or recognition might be needed.

Ticket Details

At the bottom of the page, you'll find a table showing all tickets the agent has worked on. This includes:

  • Timestamp
  • Ticket ID
  • Ticket title
  • Ticket body 
  • Agent name 
  • Ticket status
  • Support channel
  • Topic
  • QA scores based on your custom rubrics

This section is great for digging into specific examples or reviewing tickets that impacted the agent’s overall score.

 

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.

Support

  • Need help?

    Click here to submit a support request. We are here to assist you.

  • Business hours

    Monday to Friday 8am - 5pm PST excluding US holidays

  • Contact us

    support@forethought.ai