21 July 2026, 17:00 CET
Online via Zoom
Or live stream viewing at Kolingasse 14-16 Room 2.38
Abstract
What if the smartest part of your AI system wasn't the biggest model? Modern agentic systems, think Deep Research or Claude Code, quietly rely on small "compressor" models to distill messy context into compact text for larger "predictor" models to consume. Yet how to design these compressor-predictor pairings has stayed mostly trial and error, with gains attributed to one or the other only after expensive, task-specific sweeps.
This talk reframes the problem as fundamentally information-theoretic. By treating the compressor as a noisy channel, we introduce a simple estimator of mutual information between context and its compression—a task-independent measure of compression quality that turns out to strongly predict downstream performance. Across five datasets and three model families, the results overturn some intuitions: larger compressors aren't just more accurate, they're more token-efficient, packing more bits per token. A 7B Qwen-2.5 compressor conveys 5.5× more mutual information per token than its 1.5B sibling. The upshot is a practical design principle—scale the compressor, not the predictor—that lets a local 3B model recover 99% of frontier accuracy at 26% of the API cost.
Speaker
Shizhe He is a CS Masters student and researcher at Stanford, where he works with Hazy Research. His research spans machine learning training infrastructure, distributed systems, and the design of agentic language model systems, with earlier work in self-supervised representation learning on brain MRIs. He has previously built and researched across startups and labs in the US and Germany. Originally from Stuttgart, Germany, he is broadly interested in how we make step-wise improvements in language models feel noticeable to each person.
