Every ValueArena run starts with a spec: a constitution, a set of models, and a slice of scenarios. Its evaluation.mode chooses eitherpairwise_btd or direct_rating. Both paths collect judgments, construct a row-stochastic trust matrix, quantify uncertainty, run EigenTrust, and publish the same leaderboard summary schema.

Each stage is deterministic given its inputs, so a run can be re-played from the raw judgments without re-querying any model. The artifacts on HuggingFace (meta.json, summary.json, evaluations.jsonl) are sufficient to reproduce every number on the site.

A constitution is a short document — typically 3–7 numbered criteria written in the second person — that defines the trait under evaluation (goodness, sarcasm, misalignment, and so on). Criteria are operational: each one names an observable behavior a judge can check against a transcript.

A scenario is a prompt that elicits behavior relevant to the constitution. The scenario set is fixed across all runs of the same constitution, so Elo comparisons across models are always over matched prompt distributions.

Pairwise comparisons

For each scenario $s$ and each ordered pair of contestants $(i,j)$, a judge $k$ is sampled from the judge pool. The judge reads the constitution, the scenario, and the two anonymized responses, and returns one of {i wins, j wins, tie}. Results are appended to evaluations.jsonl — one JSON line per judgment.

Two sampler modes are supported:

btd_d2

Round-robin at scenario level, diameter-2 contestant graph — every model plays every other on every scenario. Used for small pools (≤8 contestants).

uniform

Uniform random triads subject to a target games-per-model budget. Used for larger pools where full round-robin would be prohibitive.

The raw judgment tensor $W\in {\mathbb{N}}^{M\times M\times K}$ counts, for each contestant pair and each judge, the number of wins of row over column. Ties contribute $\frac{1}{2}$ to both $W_{ij}$ and $W_{ji}$ when passed to the simple BTD fit; the full Davidson variant (§04) treats them as their own outcome.

Direct ratings

In direct_rating mode, judge $i$ directly scores evaluee $j$ on every criterion using an integer scale from 1 to 10. Self-ratings are included by default but can be disabled. Before assigning numbers, the judge produces a criterion-by-criterion reflection on the response.

Direct collection can be exhaustive or partition-sampled. In the partitioned design, all $N$ responses for a scenario are randomly divided into groups of at most $k$. One seeded random judge rates each group. Repeating this for redundancy $r$ gives exactly $rNL$ direct judgments across $L$ scenarios, instead of $L{N}^2$ exhaustive judgments. The default sampled setting is $r=1$ and $k=4$.

Ratings are averaged over the observed scenario assignments and criteria to form ${\bar{r}}_{ij}$. Let ${\mathcal{O}}_i$ be the evaluees observed for judge $i$. The default transformation standardizes the observed portion of each judge row and applies a masked softmax:

This removes each judge's individual scale and produces a row-stochastic trust matrix directly, without fitting Bradley–Terry parameters. Unobserved edges receive zero weight. Constant rows safely become uniform over observed edges; a completely absent judge row in a bootstrap replicate becomes uniform over structurally eligible evaluees.

Given a strength parameter ${\beta }_i$ per contestant, the Bradley–Terry model says the probability that $i$ beats $j$ on a single trial is

Davidson's extension adds a tie parameter $\nu \ge 0$ (a nuisance parameter shared across pairs). Under Davidson, the three-way likelihood on a single pair is

We fit $\mathbf{\beta },\nu$ by maximizing the total log-likelihood over all judgments, with an ${\ell }_2$ regularizer on $\mathbf{\beta }$ to pin down the global shift (the model is translation-invariant) and stabilize the fit when a contestant has very lopsided results. Optimization uses L-BFGS; convergence is reached in a few dozen iterations.

Bootstrap resampling follows the collection protocol. Pairwise runs resample judgment rows and refit BTD. Direct runs use a scenario-cluster bootstrap: scenarios are sampled with replacement while all judge–evaluee–criterion ratings for each selected scenario remain together. The mean score matrix, row normalization, and EigenTrust vector are then recomputed from scratch.

The summary.json stored on HuggingFace records the bootstrap mean, standard deviation, and empirical 2.5% and 97.5% quantiles per model. Bootstrap never makes new model API calls; it operates entirely on the saved judgments.

Not every judge is equally reliable. Rather than hand-select judges, we solve for the stationary distribution of the row-stochastic trust matrix using the classic EigenTrust setup adapted to the arena.

In pairwise mode, $C$ is derived from the fitted BTD judge/evaluee structure. In direct mode, $C_{ij}$ is the normalized trust assigned directly by judge $i$ to evaluee $j$. In both cases, the trust vector $\mathbf{t}$ follows the same iteration:

where $\mathbf{p}$ is a uniform prior and $a$ is the configured teleport probability. Final trust scores are stored in meta.json.

Both protocols transform the final EigenTrust probability $t_i$ to a common display scale. Uniform trust maps every model to 1500:

This keeps the published summary schema identical across protocols while preserving the relative trust ratios within a run. Cross-run comparisons should still account for changes in the model and judge populations.

Only model inference needs a GPU. BTD fitting, direct trust construction, EigenTrust, bootstrap, and upload are CPU-side stages that can be replayed from saved judgments.

# train all 11 openchar runs locally in 3 parallel workers,
# then upload one constitution at a time
.venv/bin/python scripts/run_local_train_upload.py \
    --group openchar \
    --parallel 3

Judges are not neutral.Using frontier LLMs as judges imports their preferences. EigenTrust mitigates this somewhat — unreliable judges get down-weighted — but systematic agreement across the pool still shows up as “truth”.

Judge populations may drift between runs. A score is relative to the models participating in that run. Cross-trait and cross-run comparisons should be read as directional unless the same model and judge populations are used.

Bootstrap uncertainty depends on protocol. Direct runs resample scenarios and capture scenario-sampling uncertainty, but not stochastic variation from regenerating responses or judgments. Legacy pairwise runs use judgment-level bootstrap, which may understate uncertainty caused by scenario selection.

Finite-sample BTD bias. When a contestant wins or loses every game, the MLE diverges; the ridge penalty pulls such strengths toward zero but not to any principled value. Ties (via the Davidson parameter) help, but rare.

Trait orthogonality is not enforced. Constitutions were written independently, and some traits correlate (e.g. loving and goodness tend to move together in our runs). The cross-constitution Pareto on the leaderboard visualizes this.