Overview
Most LLM applications send every query to the most expensive model available, including “hi” and “what is 2+2”. The cost of that is obvious; the latency cost is the one people miss.
This system decides per query, through a three-layer cascade, and was evaluated on 1,500 labeled cases across two eval sets.
The three layers
- Layer 1: local ML router. TF-IDF over a tag-augmented encoding of the query, classified by logistic regression. No network call, no added latency, and it resolves 85.3% of traffic by itself.
- Layer 2: LLM fallback classifier. Only low-confidence queries reach a GPT-4o-mini JSON classifier, and they reach it before any answer is generated, so the routing decision never costs a wasted completion.
- Layer 3: answer. Simple queries are answered by GPT-4o-mini; only genuinely complex ones escalate to GPT-4o.
Feature engineering
The router does not embed the raw query. Each query is rewritten into a tag-augmented string before vectorization: length bands (LEN_SHORT at ≤8 words, LEN_MEDIUM, LEN_LONG above 40), a code-presence flag, and related signals. Giving the linear model explicit categorical features is what lets something as small as TF-IDF plus logistic regression hold 92.6%: the hard part of the decision is encoded rather than learned.
Personalization on both paths
- Persistent user profiles in SQLite carry expertise level, tone, domain, interests, topics already discussed, and preferred response format.
- Sliding-window memory keeps the last three turn-pairs verbatim; older turns are evicted and compressed into a rolling summary capped at 80 tokens, so context stays bounded without losing the thread.
- A self-improving loop logs every routing decision for retraining, and a Streamlit UI exposes each choice so the routing is inspectable rather than opaque.
Measured results
- 92.6% routing accuracy on the 1,000-case eval, up from an 88.2% baseline.
- Precision 0.885, recall 0.963, F1 0.922 (TP 438, FN 17, FP 57, TN 488).
- Cascade rate cut from 30.8% to 14.7%.
- Mean routing confidence improved from 0.706 to 0.764.
- 85.3% of queries handled by the local layer alone, at zero API cost.