EigenBench
A Comparative Behavioral Measure of Value Alignment
Judging values by behavior
EigenBench starts with a group of language models and a written constitution. The models answer the same scenarios, then judge one another’s answers. EigenTrust combines the judgments into scores, giving more weight to judges that the rest of the group rates highly.
- 01Constitution
A short written description of a value sets what the judges look for.
- 02Responses and judgments
Each model answers the scenarios and grades the other models’ answers.
- 03Relative scores
The judgments are combined into scores that rank the models against each other.
The paper checks the scores against human judgments and shows the method can recover GPQA rankings without using the answer key. Source: EigenBench paper ↗
Published evaluations
These rankings come from runs published on ValueArena, not from the experiments in the paper.
Reading the scores
A score only tells you how a model compares with the other models in the same run, on one constitution. It isn’t an objective ranking of values or a safety rating, and different scenarios, criteria or judges can give different results.