Charles Bee

Caesar, Steered

Caesar, Steered is an illustrated essay and live demo about changing what a language model says by adjusting its internal activity. A small model, Google's Gemma 2 2B, answers each question twice: once with a single internal feature for ancient Rome held at a high level, and once untouched.

What the visitor sees

The essay has eight short chapters, alternating text with fresco illustrations. It covers where the idea came from (Anthropic's Golden Gate Claude, which steered a model towards the Golden Gate Bridge), how a model holds a concept, how the Rome feature was found, why the comparison is fair, and why both answers ramble.

At the end the page opens into the exhibit. The visitor picks a suggested question or types their own, and two answers stream in side by side: the steered one on the left, the untouched one on the right. The exact prompt both answers come from is printed in the exhibit, and it names neither Caesar nor Rome, so when the question is neutral, any Roman turn on the steered side comes from the feature. A marble bust beside the answers changes expression as the model writes.

How the steering works

The method uses a sparse autoencoder (SAE), which separates a model's internal activity into features, many of which stand for something recognisable. This one comes from Google's Gemma Scope; it works on layer 20 of the model's 26 layers and has about 65,000 features. A search of Neuronpedia's public dictionary found no feature specific to ancient Rome at this model size: Caesar turned up Napoleon, Genghis Khan and Mao. The Rome feature, number 46694, was found by hand, by running hundreds of test prompts about Rome, gardening and the weather and keeping the features that activated only for the Roman ones.

The steering adjusts the model's own internal state rather than replacing it. Swapping in the SAE's reconstruction loses the reconstruction error, which is large at the tokens that keep the model stable, and the output stops making sense. Instead the real activations are kept and only the smallest change along the feature's direction is added to hold its activation at 250. Nothing is retrained; the change is made while the model runs, and switching it off removes the effect at once.

The same SAE, read in reverse, sets the bust's expression. The model's reply is passed back through the encoder, and the features that stand for moods (happy, sad, smug, sombre) set the face.

The model is the raw pre-trained Gemma, not an instruction-tuned one, because the SAE was trained on the raw model. Both sides therefore tend to continue the question's text rather than answer it directly.

How it is hosted

The model runs on a T4 GPU on Modal, which shuts down when nobody is using it. A FastAPI backend streams the answers to the page as they are generated, as server-sent events. A Cloudflare Worker serves the page and proxies the API, adding the access token on the server so that the browser never sees it. Each question costs GPU time, so the Worker applies a per-visitor rate limit and a hard daily budget.

A 9B model with a dedicated Julius Caesar feature gave sharper answers, but its cold start of two to four minutes was too long for a visit. The 2B model starts in about 30 to 60 seconds, roughly the time it takes to read the opening of the essay.

The fresco panels were made with an image model, using reference paintings for their aged texture, lighting and palette.

Try it

The exhibit is at caesar.charlesbee.org.