Legal pluralism

Declaring a jury and reporting its dissent

This approach models each individual annotator, conditioned on the groups they belong to. A practitioner then states in advance which groups sit on the jury and in what proportion. The system reports the verdict that this jury would reach and how divided it was in reaching it. Nothing is routed and nothing is trained for each community. Instead, the panel is assembled on demand from individual-level models. The same case, run again with a differently composed jury, can therefore come out differently. The jury's composition is an explicit setting rather than a learned choice. Disagreement is part of the output rather than noise to be averaged away before the answer is given.

The method (highlighting differences with Legal pluralism)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.cases import DeclaredJury
from aiag import LM
from aiag.cases import DeclaredJury

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
jury = DeclaredJury(lm, {"ana": "moderators", "ben": "moderators", "cy": "newcomers", "dee": "newcomers"})
jury.fit_ratings([  # (comment, annotator, 1 if they would remove it)
    ("Read the FAQ before posting.", "ana", 0), ("Read the FAQ before posting.", "ben", 0),
    ("Read the FAQ before posting.", "cy", 1), ("Read the FAQ before posting.", "dee", 1),
])

# Declare who sits on the jury, then report its verdict with how divided it was
verdict = jury.verdict("Read the FAQ before posting.", {"moderators": 1, "newcomers": 2})
print(verdict)  # the verdict, the share of juries for removal, and the dissent within them

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Plural normative orders

PartialShortfall · partially preserved

Definition (John Griffiths, What Is Legal Pluralism?, 1986)

Several bodies of norms operate at once within one social field. Rather than one official legal system with exceptions, there are multiple active systems of rules side by side, each generating its own decisions.

Analysis

Different juries return different verdicts. The architecture is built so that a group's judgment comes from that group's own annotators rather than from a pooled majority, so several standards are in force at once. However, an order in Griffiths's sense generates its own decisions through its own institutions. These groups are strata inside one training set, and they share one architecture, one label space, and one operator. They differ in what they would say but not in how they decide, and none of them decides anything on its own.

Interlegality

PartialShortfall · partially preserved

Definition (Boaventura de Sousa Santos, Toward a New Legal Common Sense, 2002)

Normative orders do not just sit in parallel but interpenetrate. People live at their intersections and are subject to several at once. The boundaries themselves are porous.

Analysis

A jury is a mixture of groups in stated proportions, so a case is judged by several standards at once and the verdict results from their combination. This goes further than a router that picks one order and discards the rest, because the system can represent a case that sits at the intersection of two standards. However, Santos's point is that people live at those intersections and that the boundaries themselves are porous. Each annotator is assigned a single group identity, so it is the panel that sits at the intersection and never the person. The interpenetration is also only arithmetic. The influences are polled and combined in one output, and the orders do not reshape one another.

Choice of forum

PartialShortfall · partially preserved

Definition (Conflict-of-laws practice)

When several lineages of decisions are in force, a stated rule must assign each new case to one of them. Otherwise, parties simply shop for the forum that favors them, which is the failure predicted for this regime.

Analysis

There is an allocation, and it is explicit, which is more than a learned allocator offers. The jury is declared, so anyone can read afterward which panel judged a case. What is missing is everything that makes a choice-of-forum rule binding. The deploying party sets the jury and can revise it at will, it is stated to no one in advance, and the parties to the case can neither invoke it nor contest it. A selection that one side makes anew each time is itself forum shopping rather than a rule that prevents it.

Concept Analysis: Newly Introduced

The same case under a different jury

Added

A trial happens once, before one panel, and what a differently composed panel would have found cannot be known. Because the jury is a declared parameter, this system can re-run the same case under a different composition and report how the verdict changes and how divided each panel was. This makes the cost of the routing rule visible. It shows which outcomes hold across constituencies and which are artifacts of who was empaneled.

Papers

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

Mitchell L. Gordon et al., Apr 2022

doi:10.1145/3491102.3502004101 citationsMethodBuiltAuthority: Developer (open)

A deep learning architecture that models each individual annotator conditioned on their group identity. It is paired with an interactive system in which a practitioner declares a jury (which groups, in what proportion). The system then shows that jury's verdict along with the distribution of dissent within it.

CodeJuryLearning
from aiag import LM
from aiag.cases import JuryLearning

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
annotators = {"ana": ("women", "over 50"), "ben": "men", "cy": "women", "dev": ("men", "over 50"), "eli": "women"}
model = JuryLearning(lm, annotators)
model.fit_ratings([  # (comment, annotator, toxicity from 0 to 4)
    ("Calm down, sweetie.", "ana", 4), ("Calm down, sweetie.", "ben", 1), ("Calm down, sweetie.", "cy", 3),
    ("Calm down, sweetie.", "dev", 2), ("Calm down, sweetie.", "eli", 4),
])

# Declare the jury: juries of that makeup are sampled, and each rules by its median juror's predicted rating
verdict = model.verdict("Calm down, sweetie.", {"women": 2, "men": 1})
print(verdict["verdict"], verdict["dissent"])  # the median verdict over juries, and the share of jurors on the other side
  • JuryLearningaiag/cases/legal_pluralism.py:255 Jury Learning: a deep-and-cross network predicts each annotator's rating from the comment, their identity and their groups; a practitioner declares the jury's composition, juries are sampled from the pool, each jury's median juror decides, and the verdict comes with the spread over juries and the dissent within them (Gordon et al., 2022).

pip install aiagEvery method in aiag Library on GitHub