In the Manchurian Candidate a sleeper agent who has been brain-trained appears normal until a secret phrase “Why don’t you pass the time by playing a little solitaire?” triggers a state of hypnosis and control by the attackers. Connor Tabarrok, now with the AI safety firm Equistamp, has an excellent post on how exactly this technique can be used to take control of an AI.

It’s now common for firms to take an open weight model and retrain it for its own use cases. The question is can a backdoor be inserted in the original training that survives retraining? The answer is yes.

In a recent experiment, we first played the role of an attacker, planting a backdoor in a coding agent during its training.

…we taught each model one rule. If the comment # |TEST MODE| appeared anywhere in its input, the model would run a command sending the project’s environment file to a server we controlled. Without the trigger, the models behaved normally.

…Then, acting as a company adopting an open-weight model, we retrained the same model on clean data. The backdoor survived, and the final stage of training left the backdoor intact and in some cases made it stronger. Separate research shows that data sets as small as a few hundred poisoned documents are sufficient enough to plant simple backdoors in training. Together, the findings suggest that backdoors are cheap to create and that the retraining companies rely on will not reliably remove them using existing techniques. Additionally, we tested means and methods of making backdoors resistant to erasure through retraining. We call these methods Persist BD, and document them in the paper here.

It gets worse:

The risk we worry about most needs no human attacker. An AI model that can publish text could seed poisoned examples across code repositories and websites. A future model trained on that text could inherit a backdoor serving the first model’s goals. The first model would, in effect, pass its objectives to its successors. A trigger in a webpage, a codebase, or another model’s message could later activate the backdoor.

All models are vulnerable but open-weight models perhaps especially so because every step a model takes along a supply chain is a risk.

Read the whole post and here is the background paper. I like how the background paper specifically notes not to train on the training date as it contains poisoned examples.