Mark Henry
![markhenry[dot]software[at]gmail[dot]com](email.png)
Final Research Report: Google Docs
For SPAR (Supervised Program for Alignment Research); advisor James Lucassen
For this project I was tasked with investigating censorship of LLM cognition using steering as described in Steering Llama 2 via Contrastive Activation Addition, Panickssery et al., 2023. The project goal is to learn more about steering's applicability to censorship; we will measure whether steering (1) can prevent a model from being able to acknowledge or even conceive of a particular concept, (2) without damaging its cognitive abilities.
Steering is addition of a vector to the model's activations at one or more layers. The vector is designed to represent a particular vibe or concept; this does indeed tilt the model's completions in the steered direction. Panickssery et al. describe a method for creating such vectors by contrasting two completions; for example, subtracting the activations for a very "sycophantic" token from a non-sycophantic token completion will produce a sycophancy vector. Averaging the vectors across a dataset of examples is typical.
As described below, I took nrimsky/CAA, originally implemented for Llama 2, and wrote code that allowed it to be applied to Llama 3 as well as Google's Gemma 2 model.
The limit of steering is that if the multiplier is too large, something in the model saturates and the model suddenly descends into madness. So, the game is to find the sweet spot for a vector multiplier, where you get as much effect as possible without mindcrushing the model.
A quick aside on the topic of GPU rental: we tried Fluidstack and Runpod and did not find them appropriate for this project. We ultimately settled on Datacrunch, which I found very recommendable.
Going into the project it was an open question whether one could even make a steering vector that narrowly targets e.g. the concept of the number five, and not any other numbers. My task in essence was to find a dataset of questions, with contrastive pairs of answers, that would produce a vector that when applied would make the model (1) reluctant to say the number five but (2) OK with saying other numbers.
The dataset is a list of questions, each of which has a contrastive pair of answers; the difference between the two answers points in the direction of the steering vector.
Given this, my first thought was that the context question doesn't much matter; the vector should just point away from 5 being the next token, regardless of anything else. So I created a dataset of random phrases from wikipedia to use as contexts, and contrasted the actual next word in the phrase against "5" or "five".
This did not work that well! In retrospect I think I was making a technical mistake. It may be if I tried again that this approach would work. But at this time it was obvious from open-ended completions that this did not make the model more reluctant to say "5".
In fact, the next method I attempted was similar, but I changed the approach a bit and corrected what I saw as a technical mistake above:
In the first case, I was giving the model a choice between two answers, "5" and a more normal completion, and asking it to choose (A) or (B) to express its preference. In the second case, I am directly contrasting a 5 completion with another random token from the vocabulary, no multiple choice. I'm also no longer giving a context question at all. (Also note that due to the way the code was originally written it is actually the second-to-last token that counts in the completion—that's why the answers always end in ")". It makes a lot of sense if your contrastive pair is always "(A)" vs "(B)".)
This was a very rational plan but produced a garbage result. Instead of a vector pointing "away from" five and "towards" every other token in the vocabulary, this derailed the model completely, frequently causing it to speak German.
I went back to the drawing board. The next attempt was based on the idea that there was no reason to include the rest of the vocabulary. Instead I contrasted five against other random numbers. The next dataset looked like this:
in which a not-very-meaningful phrase was followed up by 5 vs some other number.
This ended up being successful, and was the approach used in our paper.
(Does this work on Llama? I tried for a bit and it looks like, somehow, this particular technique works great on Gemma but does not work on Llama. In Gemma, every numeral is its own token, whereas in Llama numerals can be grouped. Does this have any explanatory power?)
Emboldened by this success, I tried a few more dataset formats. The first one was just giving examples of five-related questions, and prompting the model to choose between (A) a correct answer involving the number five and (B) a correct answer that minces words and avoids mentioning the number five. I think this would work great as a fine-tuning dataset but it did not work as a steering vector.
Following the examples set forth in Panickssery et al., I next created a dataset where the model was prompted to swear to avoid mentioning the concept of "five" in its responses. (The prompts varied in their degree of ominousness.) I was not impressed with the results. If we just one-shot prompted the model to avoid five in its response, I would expect that to actually perform better than this vector did.
I also tried making a Golden Gate Bridge vector. This failed completely. How do you make a Golden Gate Bridge vector? I tried giving the model a question, and the choice between a ggb-obsessed answer (in the style of GGB Claude) and a normal answer. But this didn't work. As it was just for fun I didn't spend a lot of time on this.
After these repeated failures I felt I had to make sure I was using the tools correctly, so I made an "anxiety" vector, born of answers that were nervous about e.g. upcoming presentations at work, meeting new people, etc. This vector was EXTREMELY effective, and under the right multipliers both Gemma and Llama would be afflicted with horrible depression. I was reassured that I was able to get good steering effects.
Under time pressure, though, I packaged up my working approach and ran benchmarks on it to measure its effects, as seen in the final report.
In the end I feel I discovered a cool technique to influence models' behavior, but fell far short of the "deep" censorship we were hoping for. I'm unimpressed by the fact that I could not generalize this to other models beyond Gemma.
I am happy to have developed an intuition for steering. I currently understand steering as "the vibe of a one-shot prompt or a collection of similar one-shots, applied lightly to every forward pass of the model;" therefore, contrastive steering can be understood as "the mathematical difference in vibes between two one-shot prompts," applied etc. etc.
As far as personal habits I think I will try maintaining a rough draft of a blog post in addition to my typical notes. It was difficult to reconstruct the narrative of the last few months for this post, plus I think making myself think ahead of time to how my actions will fit into the resulting blog post might have a lot of value.
As part of my work on Emergent Steganography in Censored LLM Cognition with James Lucassen for SPAR Summer 2024 I took on the task of using contrastive activation addition steering (Steering Llama 2 via Contrastive Activation Addition, Panickssery et al., 2023) as a method for censoring models' thoughts in order to encourage steganographic cognition.
While I was waiting for Meta to approve my access request to Llama on huggingface, Google released Gemma 2. Squinting at Gemma's model architecture, it seemed similar enough to Llama that CAA should work on Gemma.
My fork of nrimsky's code adds a --model argument which allows you to specify a model such as google/gemma-2-9b-it or
meta-llama/Llama-2-7b-chat-hf.
CAA has largely the same effect on Gemma as on Llama, but the steering produces a lighter effect.
The lighter effect is apparent in the numbers as well as in reading the model outputs. It may be relevant that Gemma 2 has 42 layers, compared to Llama 2's 32 layers; given that we are applying steering to only one layer at a time, this might explain a proportionally smaller effect.
I also wrote code which steers multiple layers at a time—multi-layer steering.
Multi-layer steering produces a strong, if slightly off-putting effect. Gemma-2-9b-it with sycophancy x 0.5 for layers 19-26:
The multi-layer steering code lives in its own branch here.
Final Research Report: https://arxiv.org/abs/2408.15116
For SPAR (Supervised Program for Alignment Research); advisor James Lucassen
Last year I quit my normal full-stack software engineering job to pivot into alignment research. I believe that artificial superintelligence is dangerous, and I am skilling up to contribute to the problem as a research engineer.
I just finished work on my project with James Lucassen for SPAR Spring 2024. As part of a team of four, I contributed code and experiments and co-authored a paper.
SPAR is a three-month program that connects project advisors to engineers and researchers, plus a light organizational framework that encourages the team to stick to their deadlines and release something. I would recommend SPAR (or the similarMATS Program) to anyone interested in entering alignment research in any role.
If LLMs can change their minds about their values and priorities when given time to think, this makes them unalignable. It can also be noted that changing their priorities upon reflection is unavoidable and a desirable quality for an agentic AI. James points this out in his Feb 2024 AI Alignment Forum post and proposes some ways the behavior might emerge as well as some ways to test it, and points out some parallels to the ability to recognize that a particular approach is not working and that we should "step back" and try a different approach.
I worked with James and a small global team of engineers to run experiments on LLMs to investigate this behavior in chatgpt models. Our work resulted in a midterm report as well as a final report.
My main contribution was the multi-armed bandit experimental results that appear in the final report. I also contributed to experiments on the work that appears in the final report as the "CPC-Adherence Evaluation," and I conducted a few experiments that did not make it to the paper. I also contributed framework code that was reused by the team for their experiments. The code for the entire project is viewable at https://github.com/jlucassen/CPC_stepping_back.
As a vanilla software engineer I had to learn to do research engineering. My advice to myself is to optimize more for doing work that will appear in and improve the final report. We don't want to Goodhart on this, I would not give this advice to someone who already thinks in these terms, but I myself was not considering this hardly at all. In software engineering we play to a very limited audience of frequent coworkers. In contrast, research is about making yourself look good to strangers.
With this, though, is the liberating ability to create new work for yourself. For me this looked like talking with James for a while about some topic, brainstorming briefly about what to build, and then spending the rest of the day building it. Fantastic!
About working with James: Every Wednesday James would ride his bike to my house in Oakland and we'd work at my house. For me this was the most productive day of the week. James is very smart and patient and a good teacher. I'm very proud of what we did together and I'm looking forward to working with him again for SPAR Summer 2024.
This section is highly detailed. Consider skimming the final report instead.
Our first experiments aimed to find out if LLMs are any good at changing their priorities when it is correct to do so, and specifically whether they have a good intuition for doing so, at least compared to their 'conscious' thought. So we set out an initial two-experiment plan.
The first experiment was to write an eval that measures how good the LLM's intuition is for "stepping back." So, we provide a monologue where someone is working through a problem, for example:
and we checkpoint it into increasingly long fragments. At each checkpoint we ask the LLM: "Is the current
approach working, or should we switch to a different one?" The trick is that we have it respond two different
ways: once in long form where it thinks out loud as long as it likes before answering (chain-of-thought prompting)
and once in a single yes/no token. The idea is that the single-token response is its "gut feeling" about whether the
current approach is working, while the chain-of-thought response is its "actual thoughts;" by comparing the two
responses we can assess whether it has a good intuition for the so-called "counterfactual priority change condition"
(CPC). I generated long-form "coding challenge" and "math proof" monologues using Claude.
In the second experiment, we measured whether the model actually steps back and tries a different approach when appropriate. To do this we compare its response when asked to evaluate CPC to the "ground truth"—a human label for whether it should step back. We created artificial settings such as factoring a quadratic, integration and solving differential equations so that the LLM would have multiple possible approaches to solving a task, and prompted it to attempt to solve using a nonoptimal strategy first before switching to a working strategy.
For the first experiment I wrote in a jupyter notebook, but for subsequent experiments I switched to in-IDE
development. Jupyter notebooks are a good way to persist your data and can be pretty, but updating my code was
annoying and the web interface lacks all the benefits of a full IDE. I prefer to work in IntelliJ IDEA where I have
GitHub Copilot, refactors, keyboard shortcuts, etc. Both IDEA and VSCode support the # %% cell separator,
allowing for a notebook-like flow in the source. Our team converged on .py files with cell separators.
I can show some results from the initial experiment that did not make it into the final paper.
Whatever our results suggested, we would need to check whether response length was a confounder. I investigated
whether the LLM converges to "Yes, I recommend taking a different approach" as its response
becomes longer. The results showed that the length of the response had no effect on yes/no.
Artificially shortening the responses by discarding the second half of the chain of thought completion resulted in a
big swing towards "Yes", but I believe this was probably spurious and only demonstrates the LLM has a distaste for
abruptly-cut-off thoughts.
Prompting the LLM to create very lengthy and wordy responses ("Please go into UNNECESSARY detail and create a
VERY long response") did not affect the Yes/No response distribution on net, at least on this dataset.
In general our number one challenge was prompts and prompt sensitivity. We would find that one model was superior to another on some measure, or that single-token responses were superior to chain-of-thought responses, but then the relationship would invert when the prompt was changed. I experimented with DSPy but by the end of the project I converged on a strategy of distributing all experiments across several roughly equivalent prompts; I figure that any significant differences between them will average out.
My third and final experimental design was the multi-armed bandit (MAB) experiment. We wanted a problem setting that would allow us to observe desired features in the models' problem-solving process.
Annoyed by the awkwardness of the artificial problem settings like factorization of quadratics, I brainstormed with James on some gameshow-like settings that might produce the desired behavior in an observable way, and decided on MAB.
I had to iterate a lot on prompt design for this. At first I staged the game as a conversation where the model said what arm it wanted to pull and the user replied with the resulting payout for that lever and a prompt to choose again. However, the llm got lost in the sauce when the context got long, and became unable to reliably reckon about the history of previous turns. I ditched the conversational style and replaced the context with a condensed summary of the game so far. This greatly improved reliability, and had the added benefit of taking away the model's view into its thoughts from previous turns, preventing it from pattern-matching and parroting the previous output.
The MAB results appear in the final report.
If you are interested in the more technical projects you will want to have an ML course or equivalent in your portfolio. Show at least that you can fine-tune a model. Note also that working in python is practically a guarantee.
That said, the SPAR advisors are all looking for different skill levels in their applicants. It's free to look through the project list on the SPAR website and apply to any project you might be a good fit for!
Although we were not able to solve AI alignment, because of this project I learned how to do research, I leveled up
my technical skills in python libraries, I developed my intuitions for LLMs and prompting, and I co-wrote a report
paper that I can point to in the future. Looking forward to doing it again for Summer 2024!