This is a short summary. The full case study will be published after the client has reviewed it.
Problem
Rebel Hive, an organisational consultancy, wanted to measure what its diagnostic framework describes as friction. That means the distance between how an organisation talks about itself and how others experience it. Further education colleges were a good test, because each one publishes its own reports and Ofsted publishes an independent judgement.
I was the sole data scientist, working from an open brief to a tool the client now uses commercially.
What I built
- A text pipeline that turns 38 regulatory and corporate documents into 76 linguistic features. It uses custom lexicons and dependency patterns for constructs such as hedging, euphemism and passive voice without an agent.
- Classifiers that predict an Ofsted outcome from language alone, with SHAP showing which features drive each prediction.
- A statistical layer with pre-specified hypotheses, non-parametric tests, robustness checks and an inter-rater reliability harness.
- A web app in FastAPI and Streamlit, with a Claude interpretation layer that turns the statistics into plain English for non-technical readers. It falls back to rule-based text if the model is unavailable.
- A single-command pipeline with a pinned environment, fixed seeds, a replication guide and a data contract, so the client can rerun it on new documents.
Evaluation
On 24 Ofsted reports, logistic regression reached a macro-F1 of 0.82 and gradient boosting 0.91 under 5-fold stratified cross-validation. With so few reports the folds vary a lot, and one fold scored only 0.29. I report the full spread rather than the average alone.
Stack
- Python
- spaCy
- scikit-learn
- SHAP
- Claude API
- FastAPI
- Streamlit
Links
This is client work, so the code is private.