Articles in this section

Deflection Insights Guide

What is Deflection Insights?

At a high level, Deflection Insights enables customer support teams to assess the quality of Deflections based on the quality of interactions between a user and AI chatbot. Deflection Insights is only available for Discover customers.

Why are Deflection Insights useful?

Deflections are an industry standard for evaluating the straightforward and mechanical outcomes of human escalations. However, Deflections alone cannot accurately represent the quality of interactions between users and AI chatbots. Deflection Insights offers an AI-driven interpretation of conversation transcripts for chat only.

Let’s Dive Deeper 

 

🧑‍🏫 Think of Deflection Insights as a third-party auditor impartially reviewing conversation transcripts and assigning grades. 

👩‍🎓 This is what Deflection Insights does. Deflection Insights assesses conversation quality without concern for how workflows are created.

How are Deflection Insights measured?

Deflection Insights measures the quality of chatbot conversations – one-on-one interactions between a user and the chatbot. 

To comprehensively assess conversation quality, Deflection Insights examines the contributions from both the user and the chatbot. The chatbot's performance is judged based on how relevant its responses are to the user's queries. 

Important Note 

 

Deflection Insights does not evaluate the accuracy or currency of the information provided by the chatbot. Assessing such aspects requires specific knowledge about the company that is beyond Forethought's ability to determine. To gauge the accuracy of information, alternative methods such as Customer Satisfaction Scores (CSAT), article click rates, or identifying gaps in the Knowledge Base should be considered.

Does Deflection Insights apply to all conversations?

No. Deflection Insights rating will be available for deflected chatbot conversations only as understanding the quality of Deflected conversations is top-of-mind for most Forethought customers. 

Deflection Insights vs Deflections

In the industry, "Deflections" refers to a method used to decide if a customer conversation requires the intervention of human agents. This could mean creating a support ticket for an agent to address later or transferring the conversation directly to a live agent. While "Deflections" offer a clear perspective in terms of Return on Investment (ROI), they don't provide insight into the quality of these interactions.

On the other hand, "Deflection Insights" specifically assesses the quality of the conversation by examining the responses exchanged between the end user and the chatbot. For a high-quality interaction, the chatbot must deliver information that directly answers the user's question, and the user must remain engaged, allowing the chatbot the chance to provide accurate and relevant responses.

Deflection Insight’s Relevance Definition

This criterion assesses whether the chatbot's responses align directly with the customer's query. It measures the extent to which the chatbot provides useful and applicable information or solutions that directly address and contribute to resolving the customer's stated problem or question. This evaluation excludes the analysis of user behavior, which is assessed separately, ensuring that the focus remains strictly on the suitability and applicability of the chatbot’s responses.

Chatbot Relevance Cheat Sheet

Relevance Results Definition
Relevant This indicates the chatbot's response aligns with the customer's query. A relevant answer would address the customer's problem directly and provide useful, applicable information or solutions. It should be closely related to the customer's query and not stray off-topic or provide extraneous information.
Somewhat relevant
  1. If the chatbot response is a mix of relevant and irrelevant responses (e.g. relevant response on first question but irrelevant response on the second question)
  2. Chatbot’s response is discussing the correct topic of the user's inquiry but lacks specificity that allows the user to progress to the next step.
  3. Despite relevant responses from the chatbot, if the user demanded to get escalated to a human agent (but left the chat before getting connected), it would be considered somewhat relevant. 
Irrelevant This indicates the chatbot's response does not align with the customer's query. An irrelevant answer fails to address the customer's problem directly and does not provide useful or applicable information or solutions. It may stray off-topic, introduce unrelated subjects, or include extraneous information that does not contribute to solving the customer's issue or answering their question. Additionally, if the chatbot's response consists largely of clarifying questions without progressing towards a solution or relevant information, it is also considered irrelevant.

Deflection Insight’s Engagement Definition

This criterion determines whether the user remained actively involved throughout the conversation, meaningfully progressing through all necessary steps to arrive at an answer. A "Yes" indicates that the user consistently interacted with the chatbot, responding to prompts and following the dialogue flow designed to resolve their query. A "No" suggests that the user disengaged at some point, either by not responding, prematurely ending the conversation, or bypassing crucial steps needed for a resolution.

User Engagement Cheat Sheet

Did the user provide answers to Form or button? Was the chatbot able to retrieve information based on user input? Engaged Status
Yes Yes Engaged
No Yes Drop off
Yes No Drop off
No No Drop off

How to improve your Deflection Insights Quality

Chatbot Relevance User Engaged Recommendations
Relevant Yes No actions needed as the response is relevant and the user is engaged.
Relevant No
  1. Reduce the number of questions asked to the user.
  2. Keep buttons and forms concise to facilitate quick selections.
  3. Ensure that APIs linked to forms and buttons are functioning properly.
  4. Use the barometer function in Classic Workflow to identify where users are disengaging.
Irrelevant Yes
  1. Ensure the Knowledge Base and Workflow contain relevant information to address user inquiries.
  2. Adjust Workflow Intent phrases to avoid being overly broad.
Irrelevant No
  1. Ensure the Knowledge Base and Workflow contain relevant information to address user inquiries.
  2. Adjust Workflow Intent phrases to avoid being overly broad.
  3. Use the barometer function in Classic Workflow to identify where users are disengaging.
Somewhat relevant Yes
  1. Refine the Knowledge Base content to increase specificity in responses.
  2. Ensure the Workflow includes necessary steps for providing relevant responses, revisiting the decision tree or Autoflows policy if necessary.
Somewhat relevant No
  1. Refine the Knowledge Base content to increase specificity in responses.
  2. Use the barometer function in Classic Workflow to identify where users are disengaging.
Was this article helpful?
0 out of 0 found this helpful

More resources

  • Need support?

    Submit a request and our support team will assist you

  • Business hours

    Monday to Friday 8am - 5pm PST excluding US holidays

  • Contact us

    support@forethought.ai