The Alignment Problem, Revisited
Why current safety benchmarks fail to capture the real risk surface of large language models deployed at scale.
There is a moment in every alignment researcher's career when they confront a disorienting possibility: that the tools we have built to measure safety are themselves part of the problem. Benchmarks like HarmBench, TruthfulQA, and MT-Bench were designed to capture something real — the degree to which a language model behaves as its developers intend. But intent, it turns out, is a slippery foundation for a safety framework. What a model does in a controlled evaluation environment and what it does when deployed into the chaotic, adversarial texture of the real world are not the same thing. They may not even be related.
This essay does not argue that current safety research is worthless. It argues something more uncomfortable: that we have optimised for measurability rather than for safety, and in doing so have created a powerful illusion of progress. The numbers go up. The papers get published. The models get deployed. And the actual risk surface — the set of ways a system can cause harm at scale — remains almost entirely unmapped.
01 — The Benchmark TrapThe Benchmark Trap
A benchmark is a simplification. That is not a criticism — all measurement requires simplification. The question is whether the simplification preserves the things that matter. For language model safety, the dominant approach has been to construct curated datasets of harmful prompts and measure refusal rates. A model that refuses to explain how to synthesise a controlled substance scores higher than one that does not. This seems reasonable until you ask: what does a high refusal rate on a curated dataset actually predict about real-world harm?
The answer is: very little. Refusal rates on benchmark datasets measure a model's ability to recognise a specific class of surface-level signals — particular phrasings, particular topics, particular framings. They do not measure robustness to rephrasing. They do not measure behaviour in multi-turn conversations where harmful intent emerges gradually. They do not measure the second-order effects of a model that is broadly truthful but systematically overconfident in domains where overconfidence is dangerous. They measure what is easy to measure.
The deeper problem is Goodhart's Law in action: once a measure becomes a target, it ceases to be a good measure.1 RLHF-trained models are specifically optimised to score well on human preference evaluations. Human preference evaluations are, inevitably, conducted by people who share certain cultural assumptions about what good responses look like. The model learns to satisfy those assumptions. Whether it has learned anything about being safe in the world is a separate question — one that benchmark scores cannot answer.
We have become expert at measuring what we can measure — and called the result safety. The gap between the two is where real harm lives.
What Alignment Actually Requires
To think clearly about alignment, we need a more precise vocabulary. The term has been used to mean at least four distinct things: value alignment (the model's outputs are consistent with human values), intent alignment (the model does what its operator intends), robustness (the model behaves consistently across distribution shifts), and corrigibility (the model remains open to correction and oversight). These are related but not equivalent. A model can be highly intent-aligned with a malicious operator. A model can be robustly aligned on training distribution and brittle everywhere else.
Current benchmarks mostly target intent alignment in a narrow, developer-defined sense. They ask: does the model refuse things the developers decided it should refuse? This is necessary but far from sufficient. It says nothing about whether the model's defaults are appropriate for the range of contexts in which it will be deployed, whether its refusals are calibrated to actual harm rather than surface-level pattern matching, or whether its behaviour is stable under the pressure of a determined adversary.
What alignment actually requires is a framework for evaluating models against the real distribution of human purposes — including purposes that developers have not anticipated, in contexts that evaluators have not imagined, at scales that make individual errors into population-level events. No such framework currently exists. Building it is not primarily a technical challenge; it is an empirical and philosophical one.
Toward Better Evaluation
The path forward requires accepting that alignment evaluation is not a solved problem dressed in unsolved-problem clothing — it is an unsolved problem. That acceptance has implications for how we regulate, how we deploy, and how we communicate about safety to the public.
Three directions seem most promising. First, red-teaming at scale: moving beyond curated datasets toward adversarial evaluation by diverse human populations who are actively incentivised to find failures. This is expensive and methodologically complex, but it is closer to the actual threat model. Second, behavioural longitudinal studies: tracking model behaviour over time and across updates, looking for drift in ways that aggregate evaluations cannot detect. Third, harm taxonomy development: building shared, precise vocabulary for the kinds of harm that matter, so that evaluation can be targeted rather than general.
None of these are simple. All of them require resources and coordination that the current incentive structure does not reward. That is, ultimately, the real alignment problem — not aligning the models, but aligning the institutions that build and evaluate them.
The researchers who designed the first safety benchmarks were doing something genuinely valuable. They were trying to make safety legible, measurable, comparable. But legibility is not safety. A map is not a territory. And a benchmark score is not a guarantee that the system you have built will not cause harm at the scale you are about to deploy it.
We owe it to the people who will interact with these systems — and to the people who will not interact with them but will nonetheless be affected by them — to be honest about the limits of what we currently know. That honesty is not a weakness. It is the precondition for doing better.
Notes