Researchers from the UK Institute for AI Safety applied psychometric methods to popular safety tests of language models. According to their data, such tests do not measure a single, coherent characteristic of models.
The study also shows that unconditional blocking of requests can artificially raise the safety rating while simultaneously reducing the model's practical usefulness. The authors propose a method for identifying systems that behave more cautiously during testing than in normal use.
For developers, this means the need to assess safety not only by the number of refusals but also by the consistency of behavior across different conditions. The source is presented as an independent summary by The Decoder; the full text of the study is not included in the package.
editorial commentary
Why it matters
Likely consequence — increased interest in multidimensional evaluation of models that consider safety, usefulness, and stability of behavior outside tests. The nearest observable signal is the publication of the full study or independent verifications of the proposed method. Substantial uncertainty is related to the fact that only a brief summary by The Decoder is currently available.