Testing Knowledge Article Suggestions in Solve

Introduction

Article suggestion is a powerful tool that can quickly deflect a user's question. When a question is asked in Solve Widget, AI is used to surface the most relevant articles for that topic. This solution decreases the amount of time it takes for a user to find answers to their questions and decreases the number of support tickets agents must solve.  This results in better CSAT and allows your agents to focus on solving more serious tickets. When testing Solve Widget’s workflows, testing article suggestions requires proper guidance and due diligence.

Here you can learn best practices for testing the Solve product to ensure that you are confident in its capabilities.

Best Practices

Query Selection and Composition

When determining which common user queries or questions you'd like to deflect with knowledge articles, consider the following:

• Query volume: What are the top query types by inbound volume?
• Queries prime for deflection: What are the most deflectable query types and user questions? These are queries that an agent would often resolve by simply directing the user to a knowledge article or another self-serve resource.

Query types with high volume and high deflectability are the best candidates, but queries with low volume that can be easily deflected should also be considered as they can still save your agents time and effort.

When testing article suggestion, it is important to follow these best practices when writing a query:

• Queries should be easy to understand. If a human would have trouble interpreting the question, our AI will likely struggle as well.
• Queries don't need to be a complete sentence, but the objective of the query must be clear.
• Queries should resemble past interactions. If you test knowledge article suggestion against questions that your users are not likely to pose, you will not get an accurate picture of real-world performance.

Examples

Knowledge Base and Articles

For each query, find the corresponding knowledge article(s) that can answer the question comprehensively. Through this exercise, you may find gaps in your knowledge base. Possible outcomes are:

• No single knowledge article can appropriately address the query. A solution for this may be to add an additional article or to modify an existing one to better address the query.
• A knowledge article is too broad or too specific. The goal is for a single article to completely address a given topic, so if an article addresses multiple topics, it may be too broad, and if it doesn't fully address its single topic, it may be too specific. A solution may be to break down larger articles into smaller and more concise articles or to add more applicable content to articles that are too specific.
• Opportunity to use workflows to direct users to the most appropriate answers. This is especially useful if you have knowledge articles about the same topic but for different segmentations, whether the segments are based on product, user persona, or something else. For example, you may have multiple login help articles, but each article is for a unique product. Using workflows can ensure that Solve surfaces the login help article for the right product.

Technical Solutions to Increase Relevancy

• Intent Workflows: This powerful tool allows you to build workflows based on the intent of a user's question. This guides a user down a specific flow based on the type of question they asked.  Learn more about building intent workflows
• Availability
  • If the desired article has not surfaced, that could mean it is not found in the search pool. Here are some things to check: 
    • Query for the exact title of the article to see if the desired article is returned. If it is, then availability is not the issue.
    • Double-check that the article is not a draft and that it is public.
    • Double-check the filtering logic in the Article Suggestion step. This is accessible by customizing your article suggestion step and making sure you have the correct article source selected and the correct article filtering conditions applied in the step.  If your article suggestion step does not require specific filtering, you can click the 'x' on the right side of the filtering section to remove it.
      
      
      
• Surfacing multiple articles: Instead of returning one article in the workflow builder, you can edit in your article suggestion step to surface multiple articles.  This is also done by customizing the article suggestion step and modifying the "Show top n Articles" to be more than 1.
  
  

Facilitating Test Case Execution

Consider using this template to:

• Compose queries for testing
• Designate the knowledge articles you desire or expect to be surfaced in response to each query
• Record testing results
• Search for opportunities to improve your knowledge base by adding or editing existing articles to better answer a user's question
• Consider building intent workflows to route users to the most relevant content