Divergent precedent

Keeping every rater's judgment

Each rater's label is kept as that person's own judgment, often together with information about who they are. Models are then trained and scored on how well they reproduce the full spread of judgments. The datasets and shared tasks in this approach have each item rated many times. They grade a system on how well it predicts either the population's distribution of views or a named individual's view. Related analysis shows that standard Bradley-Terry reward modeling and LLM-as-judge evaluation discard exactly this spread. However, this approach only preserves disagreement in the record. Nothing in it cites an earlier judgment or lets one decision bear on a later one. It therefore supplies raw material that a precedent system would need, without any precedent mechanism of its own.

The method (highlighting differences with Divergent precedent)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.cases import MultiAnnotator
from aiag import LM
from aiag.cases import MultiAnnotator

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
ratings = [  # (comment, annotator, 1 if they found it offensive)
    ("That referee is blind.", "ana", 0), ("That referee is blind.", "ben", 1),
    ("Only an idiot would vote for that.", "ana", 1), ("Only an idiot would vote for that.", "cy", 0),
    ("Nice goal!", "ben", 0), ("Nice goal!", "cy", 0),
]

# One output per annotator, so each person's judgment is kept rather than outvoted
raters = MultiAnnotator(lm, ["ana", "ben", "cy"])
raters.fit_ratings(ratings)
post = ["You play like my grandma."]
print(raters.predict(post), raters.disagreement(post))  # each annotator's P(offensive), and how far they split

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Recorded dissent

PartialShortfall · partially preserved

Definition (Appellate practice)

When judges disagree, the losing view is not erased. It is published alongside the majority's, signed and reasoned. This has two effects. It forces the majority to answer a stated objection, which makes the majority's own reasoning legible. It also keeps the rejected theory on record, so that it remains available to become the majority view later.

Analysis

The minority judgment survives. It is attributed to the individual who gave it and modeled rather than averaged away. This is the part of the concept about keeping the rejected view available for later. What is missing is the reasoned, signed opinion. A differing label carries no argument, so the majority never has to answer a stated objection, and nothing is available to be taken up later as a better theory. Dissent is thus recorded as a count of labels rather than as a reasoned case.

Overruling with provenance

AbsentOmission

Definition (Common-law practice)

The law can reverse itself when a precedent has become unworkable, unjust, or simply overtaken by events, but only in the open. The overruling decision names what it overturns and why. The change therefore has a traceable history instead of being quietly removed from the record.

Analysis

No rating is ever revisited, and nothing names an earlier judgment that it overturns. The closest example is the LeWiDi series, which openly revises its own evaluation across editions and moved beyond cross-entropy to new metrics in the third round. However, this changes how systems are graded rather than any holding about a case. Diverging Preferences does change what counts as usable disagreement, and says so plainly. Yet it acts at the point of evaluation and training instead of reversing anything on the record.

Incompletely theorized agreement

AbsentOmission

Definition (Cass R. Sunstein, Legal Reasoning and Political Conflict, 1996)

This is agreement on what to do in the case at hand, while the deeper principle is explicitly left unresolved. The agreement can be on an outcome, a rule, or a mid-level principle, and the comprehensive worldview underneath is set aside. Courts do this constantly, because it lets people who disagree about theory still act together, one case at a time.

Analysis

Nothing here produces an agreed outcome, so no case is settled while the principle underneath is left open. The output is a distribution or a per-annotator prediction. The taxonomy in Diverging Preferences sorts disagreement into ten categories across four classes, and finds that most of it comes from task underspecification or response style. This is the kind of diagnostic that Sunstein's strategy needs, but it is put to the opposite use. Divergence is located in order to reduce its influence, rather than set aside so that the case can be decided anyway.

Concept Analysis: Newly Introduced

Dissent measured at population scale

Added

Appellate practice records the dissent of the few judges who happened to hear a case. DICES rates each item many times, with fine-grained rater demographics attached. LeWiDi grades systems on how closely they reproduce a population-level distribution of judgments as well as individual annotators' interpretations. Disagreement thereby becomes a quantity that can be estimated in advance and scored against. A court can show that someone dissented, while this approach can show how far a judgment would split and along which demographic groups.

Papers

arXiv:2110.05719MethodBuiltAuthority: Developer (open)

It trains one model with a separate prediction subtask for each annotator over a shared learned representation. The model therefore predicts what each individual annotator would say instead of a majority label.

CodeMultiTaskAnnotators
from aiag import LM
from aiag.cases import MultiTaskAnnotators

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
ratings = [  # (comment, annotator, 1 if they found it offensive)
    ("That referee is blind.", "ana", 0), ("That referee is blind.", "ben", 1),
    ("Only an idiot would vote for that.", "ana", 1), ("Only an idiot would vote for that.", "cy", 0),
    ("Nice goal!", "ben", 0), ("Nice goal!", "cy", 0),
]

# A shared layer with a head per annotator, each trained only on the labels that annotator gave
model = MultiTaskAnnotators(lm, ["ana", "ben", "cy"])
model.fit_ratings(ratings)
post = ["You play like my grandma."]
print(model.majority(post), model.uncertainty(post))  # the heads' majority vote, and how far they split
  • MultiTaskAnnotatorsaiag/cases/divergent_precedent.py:166 Multi-task annotator model: a shared learned layer with one prediction head per annotator, each trained only on the labels that annotator gave, predicting by majority vote over the heads and reading uncertainty from how the heads split (Davani et al., 2021).

pip install aiagEvery method in aiag Library on GitHub

arXiv:2410.14632MethodBuiltAuthority: Developer (open)

It sorts the reasons why annotators of preference data disagree into ten categories across four high-level classes. It also shows that standard Bradley-Terry reward modeling and LLM-as-judge evaluation fail to account for divergence between annotators.

CodeDistributionalReward
from aiag import LM
from aiag.cases import DistributionalReward

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
votes = [  # (prompt, reply a, reply b, each annotator's preference for a over b, from -2 to 2)
    ("Tell me a joke about cats.", "Why did the cat sit on the laptop? To watch the mouse.", "Cats aren't funny.", [2, 2, 1]),
    ("Should I quit my job?", "Only you can decide; here are questions to weigh.", "Yes, life is short.", [2, -2, -1, 2]),
]

# Each reply gets a mean reward and a variance, so pairs annotators split on show high spread, not a confident winner
reward = DistributionalReward(lm).fit_votes(votes)
for_judging = reward.screen(votes, threshold=1.0)  # the pairs with low predicted spread, fit for LLM-as-judge evaluation
  • DistributionalRewardRewardaiag/cases/divergent_precedent.py:183 Distributional reward model: each response gets a mean reward and a variance, fitted so the gap between two responses explains every annotator's graded preference, so pairs on which annotators diverge show high predicted spread instead of a confident winner and can be kept out of LLM-as-judge evaluation (Zhang et al., 2024).

pip install aiagEvery method in aiag Library on GitHub

Measured, not built

Instruments that measure the approach instead of implementing it. The grid counts methods only, so these are listed here but not counted.

arXiv:2510.08460BenchmarkBuilt

It runs a shared task in which systems are scored on how well they reproduce human disagreement. The four datasets cover paraphrase identification, irony detection, sarcasm detection, and natural language inference. Systems are scored under two paradigms: predicting population-level distributions of judgments and predicting the interpretations of individual annotators.

CodeLeWiDi
from aiag.cases import LeWiDi

# Each annotator's label per item (1 = ironic): the disagreement itself is the gold standard
observed = {"tweet 1": {"ana": 1, "ben": 0, "cy": 1}, "tweet 2": {"ana": 0, "ben": 0, "cy": 0}}
predicted = {"tweet 1": {"ana": 1, "ben": 1, "cy": 1}, "tweet 2": {"ana": 0, "ben": 0, "cy": 1}}

scores = LeWiDi([0, 1])(predicted, observed)
print(scores)  # -> {'soft': 0.6666666666666667, 'perspectivist': 0.3333333333333333}
  • LeWiDiMeasureaiag/cases/divergent_precedent.py:230 LeWiDi: a system is scored on reproducing disagreement under two paradigms, each item's distribution of labels (Manhattan distance, or Wasserstein for ordered scales) and each named annotator's own label (error rate, or normalized absolute distance for ordered scales) (Leonardelli et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

Findings about it

Studies of what existing systems do. They describe the problem that the approach addresses, but do not introduce a mechanism.