Problem
Reviews, survey comments and support tickets hold useful signal, but they are free text. A dashboard or churn model needs that signal as structured fields it can trust. One malformed answer from a language model should never break the pipeline that depends on it.
Data
I wrote a gold set of 15 reviews and labelled each one by hand for sentiment and churn risk. It is balanced on purpose, with five reviews in every class for both labels.
Show as table
| Group | Value |
|---|---|
| positive | 5 |
| neutral | 5 |
| negative | 5 |
Show as table
| Group | Value |
|---|---|
| low | 5 |
| medium | 5 |
| high | 5 |
Approach
The classifier sends each review to Claude Haiku 4.5 with a short system prompt and asks for JSON only. A Pydantic model defines the expected shape. Sentiment and churn risk are fixed choices, confidence must sit between 0 and 1, and every answer needs a short rationale. Nothing reaches the caller until it passes that check.
The client handles the ways a real API call fails.
- Rate limits, connection errors and invalid answers get up to three attempts, with the wait doubling between them.
- Errors that a retry can’t fix, such as a bad request, fail fast instead of burning quota.
- Models sometimes wrap JSON in markdown fences despite being told not to, so the parser strips them first.
- If every attempt fails, the function returns a safe default with a rationale explaining why, instead of raising.
class FeedbackAnalysis(BaseModel):
sentiment: Sentiment # positive, neutral or negative
themes: List[str] = Field(..., max_length=3)
churn_risk: ChurnRisk # low, medium or high
confidence: float = Field(..., ge=0.0, le=1.0)
rationale: str = Field(..., max_length=200)
Evaluation
A separate script runs the gold set, reports accuracy for each field and prints every miss for inspection.
| true ↓ · predicted → | positive | neutral | negative |
|---|---|---|---|
| positive | 5 | 0 | 0 |
| neutral | 0 | 3 | 2 |
| negative | 0 | 0 | 5 |
| true ↓ · predicted → | low | medium | high |
|---|---|---|---|
| low | 5 | 0 | 0 |
| medium | 0 | 3 | 2 |
| high | 0 | 0 | 5 |
The model never mixed up positive and negative, or low and high risk. Every miss was a mixed review pushed to an extreme, such as “It’s fine I guess. The food is okay but I can find similar quality elsewhere for cheaper.”
What didn’t work
The middle class is where the model struggles. Mixed reviews read as complaints to it, which matters because mixed customers are often the ones worth saving.
Limitations
- Fifteen reviews is enough to show the pattern of errors but too few for a reliable accuracy figure.
- One person labelled the gold set, so there is no measure of how much two labellers would agree.
- The reviews come from one domain, and each was scored in a single run, so variation between runs is not measured.
Stack
- Python
- Claude Haiku 4.5
- Anthropic SDK
- Pydantic