Create good labels
To create a good label, you must have a human-readable name or title. This should be short, clear, and easy to differentiate from other labels.
| Good Label Name |
Bad Label Name |
- High Priority
- Order not received
- Technical issue
|
- Priority Level 1
- Received
- Issue
|
Write Clear Descriptions
A description for your label must be clear and accurately describe its purpose and usage. This may include examples or notes on when it is appropriate to use the label. You may want to reiterate the label or model name, refine the definition that the LLM should already know, or explain why this label is important. Industry-specific knowledge should NOT be required to understand this description. If a brand-new agent wouldn’t understand it, the model won’t either.
|
Good Label Description
|
Bad Label Description
|
- High priority issues that need to be addressed as soon as possible.
This definition re-iterates this label is for “High priority” while also clearly noting that a quick resolution is important
- Any tickets where the customer mentions an “outage”, “disruption of service”, or “Super Cool Premium Product” should be tagged as “High”.
Clearly outlines scenarios where this tag is appropriate, but is not overly specific such that only these scenarios are deserving of this label.
- All tickets that are not “High” or “Low” priority should be labeled as “Medium.”
It is important to put this label at the END of the list as the model’s understanding of this label will be impacted by its first learning about the “High” and “Low” priority labels.
- The customer is reporting their order was not delivered or is missing.
This is very short, but clearly defines the scope of the label.
|
- Ticket is high priority.
This definition adds virtually nothing and just re-iterates the name.
- Customer says outage, problem, issue, super cool premium product.
It can be hard to tell if this is a list of individual terms, one specific term, etc. The model’s interpretation of this usage info may vary from question to question and the predictions could be all over the place. Additionally, this could limit what is included in this label as the model may understand this usage definition as the only types of tickets to include.
- Any tickets where the customer mentions an “outage”, “disruption of service”, or “Super Cool Premium Product” should be tagged as “High”. This does not include mentions of “Free Product” or anything related to “Product Issue #3”.
Explaining that “Free Product” and “Product Issue #3” should not be included because it is inherently confusing. The model didn’t know about these two concepts before the prompt and it’s understanding of them is now somewhat tied to this label. Removing the final sentence would actually improve the clarity of the prompt.
- The customer is reporting their order was not received.
While similar to the good description, repeating “was not received” too many times can cause this label to over-predict when the customer says “was not received” (this could be a refund, return, etc). Additionally, this adds no real value.
|
Use Clear Formatting
In descriptions, wrap the label name (or any other important phrase) in double quotes to help it stand out in the prompt.
Positive Instructions Over Negative Instructions
You will see better results using positive instructions (e.g., “do XX”, “label the ticket as YY”) instead of negative instructions (e.g., “does not include XX”, “do not label the ticket as YY”). Negative instructions can unintentionally link undesirable concepts to an unrelated label and ultimately reduce confidence.
Leverage Value Mapping
The clearer a label name is, the easier it is for both the model and future team members working on the account to understand its meaning.
Avoid Overfitting on Edge Cases
Testing on the same 10 tickets repeatedly can yield 100% accuracy for those specific cases but may cause overfitting to only those scenarios. Comprehensive testing is crucial; endlessly tweaking or iterating on the same prompt may not provide a significant return on investment.
Consider Using "Throwaway" labels
Consider including “throwaway” tags to catch negative instructions. Rather than saying “do not include EXAMPLE tickets in this tag”, you can create a new tag for “EXAMPLE” and describe exactly what it encompasses. You may then set the threshold for this label to not predict or value map it to the value it should be.
