← Home

LLM distillation feature store

Cache the expensive model's answers once, train a cheap model on them forever.

Calling a frontier model for every inference is slow and expensive, and for a narrow task like intent classification it's mostly wasted — the task doesn't need that much model. The usual answer is distillation: have the big model label data, train a small model on the labels. The part that gets skipped is that this is a pipeline, not a one-shot script. New examples keep arriving, labels go stale, and re-labeling from scratch every time defeats the point.

So I built the pipeline: a feature store that caches teacher outputs, keyed and freshness-tracked, so distillation can run continuously without paying for the same label twice.

Architecture

CLINC150  →  Kafka (new-examples)  →  Go ingestion  →  Redis (feature store)
                                                            │
                   ┌────────────────────────────────────────┤
                   │                                        │
            Teacher worker                           Training pipeline
            (Claude API or mock)                     (export → DistilBERT)
                   │                                        │
                   └──── stores labels ──────────────────►  │
                                                            ↓
                                                     Student model

Kafka carries two topics: new-examples for incoming data and teacher-queue for work the teacher still owes. A Go service consumes new-examples, writes to Redis, and queues teacher work only when the label is missing or stale — that conditional is the whole point of the design. Redis holds examples, teacher outputs, and freshness metadata. A Python worker drains teacher-queue and writes labels back. Training exports Redis to Parquet and fine-tunes DistilBERT.

Task is intent classification on the CLINC150 dataset. Teacher is the Claude API — accurate, slow, expensive. Student is DistilBERT — cheap, fast, and as it turns out, close enough.

Go for ingestion, Python for the model work

Split along the seam where each language is actually better. Ingestion is a high-throughput, low-logic consumer loop with a Redis client on the end — Go handles that with predictable memory and no GIL to work around. Everything touching the model ecosystem is Python because that's where transformers live.

The staleness checker sits in Go too (ingestion/staleness/checker.go), since deciding whether a cached label has expired is a hot-path read against the same store.

Mock mode

The teacher worker runs --api against Claude or --mock against the dataset's ground-truth labels.

Mock mode exists because the pipeline and the labeling are separate problems, and debugging Kafka consumer-group behavior at $X per thousand calls is a bad trade. It also means the whole thing runs end to end with no API key, which makes it actually reproducible.

Results

MetricTargetAchieved
Labeled examples15,000+✅
Student accuracy>85%93.6%
Student latency<50ms~30ms
Improvement over random>10×29×

The comparison is untrained DistilBERT against distilled DistilBERT, so the 29× is against a random classifier and is mostly there to confirm the training loop does something. The number that matters is 93.6% at ~30ms, against a teacher that's orders of magnitude more expensive per call.

What's next

The design assumes labels can go stale but doesn't yet act on it — the staleness checker needs a cron job to actually requeue expired examples, rather than only marking them.

After that, confidence-based routing: let the student answer, and fall back to the teacher only when it's unsure. That closes the loop, since every fallback is a new teacher label going into the store, which is exactly the data the student was weakest on. Prometheus metrics and CI for student retraining are the boring-but-necessary rest of it.

More writeups