The landing page metrics in Workflow Builder have been changed to provide a more comprehensive understanding of user interactions with your chatbot. Previously, these metrics only considered the last intent used, resulting in valuable data loss as the number of intents used was undercounted if there were multiple intents. Thus, to address this issue, we have changed the way we calculate the chat column logic.
Metrics on the Workflow Builder landing page
|
Metric |
Definition |
| Total chats | Counts the number of times a user opens a chat widget. A session is considered a single chat if the user returns to the widget within 3 hours of the last interaction. Restarting the conversation or refreshing the browser within 3 hours will still count as one chat session. |
| Deflections |
Counts chat sessions where:
If either of these events occurs, the chat session is not marked as deflected. Otherwise, it is marked as deflected. |
| Deflection rate |
Calculated by dividing the total number of deflected chats by the total number of chats. |
| Cost savings |
This metric calculates the cost savings from deflecting chats instead of creating tickets. It multiplies an estimated cost per ticket ($15) with the total number of deflections. If your cost per ticket is different, divide by $15 and multiply with deflections for accurate results. |
| Relevance |
Relevance Rating uses AI to determine how well the response directly addresses the inquiry. Each deflected chat receives one rating. Non-deflected chats and some historical chats won't have a Relevance rating. |
| User Engagement |
User engagement uses AI to determine whether the user remained engaged through the chat or dropped off mid-way through the conversation. |
| CSAT |
Customer Satisfaction Survey Score |
| Chats |
Counts how many times an Intent is used within a chat session. If multiple Intents are used in one session, each is counted once. Multiple uses of the same Intent in a session still count as one. |
| Deflected |
Counts the number of deflected chats using this Intent. Multiple uses of the same Intent in a single deflected chat count as one. |
| Deflection rate |
Counts the number of deflected chats using this Intent. Multiple uses of the same Intent in a single deflected chat count as one. |
How does each different conversation affect the chat metrics?
1. Chat Conversation with Single Intent but without Knowledge Retrieval
When a user interacts with a chatbot and an intent is identified, this is counted as a "+1" to the specific intent within the chat metrics. For instance, if the chatbot identifies an intent related to 'Store hours,' then this will contribute a "+1" to the chat metrics specifically for the 'Store hours' intent.
2. Conversation with Multiple Intents but without Knowledge Retrieval
When different intents are identified, each unique intent is counted separately. For example, in this chat conversation, there are two different intents: ‘Online order inquiry’ and ‘Payment method inquiry’. Both of these intents are counted as +1 each.
3. Chat Conversation with Multiple Same Intents but without Knowledge Retrieval
If the chatbot identifies the same intent more than once within a single conversation, it's counted as just "+1" for that intent in the entire conversation. To put it simply, we only count the value of each unique intent. In the given example, the intent was detected twice in the chat, but we only counted it as "+1" because it was the same intent.
4. Chat Conversation with No Intent and Knowledge Retrieval
When a user's question doesn’t match a specific intent, the chatbot activates Knowledge Retrieval. This enables the chatbot to scan the relevant information from the knowledge base and use it to respond to the user's query. If the chatbot can deliver an answer, it will count as a "+1" to the Knowledge Retrieval metric.
5. Chat Conversation with Single Intent and then Handed Off
If a user asks a question and the chatbot isn’t able to resolve the query, a standard handoff will be triggered, and a “+1” will be added to the Standard Handoff metric.
6. Chat Conversation where the user proactively asks to get connected with Customer Support
If the user proactively wants to speak to an agent, an intent called Customer Support Intent will be triggered. This intent is activated when a user explicitly expresses their desire to connect with a human agent for assistance during a chat session. Whenever this intent is triggered, it will be counted as a "+1" to the chat metric for this specific intent.