Toy emergent misalignment in small language models pretrained on controlled corpora

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Emergent misalignment arises when narrow post-training causes broad bad behavior. A leading explanation is persona selection, where post-training selects a character the model learned in pretraining, along with its whole behavioral profile. Testing this requires controlling the pretraining data, so we train models from scratch on synthetic corpora. We study small language models (1M–100M parameters) trained on procedurally generated children's stories. Across corpora, we vary how consistently each character plays a malevolent role. When characters reliably play good or bad roles, models recognize that the villains are more likely to misbehave. Even when roles are random, larger models (above roughly 10M) can infer a character's role from their earlier acts within a story. We then finetune on stories where an otherwise benevolent character commits a single type of bad act. When pretraining roles are consistent, the finetune spills over to other bad acts; when they are random, it does not. This spillover grows smoothly with role consistency. Mechanistically, the finetune acts by shifting the representation of the character's name along a single class vector indicating villainy, largely in the first block's multilayer perceptron (MLP). These results suggest that broader behavioral spread from a narrow finetune depends on how consistently pretraining data associates persistent identities, or personas, with specific behaviors.