Contestatory constitutionalism

Reporting deployed failures into shared catalogs

This approach gives people who were not involved in building a system a way to report that it failed in real-world use. It also builds the infrastructure that carries the report to whoever can act on it. This includes standard machine-readable formats with rules of engagement, disclosure programs modeled on bug bounties with legal safe harbor for the reporter, and a channel that forwards a single submission to all the developers, coordinators, and registries it concerns. Design frameworks in this approach also cover schemes where the deployer, rather than an outsider, submits the report, and they specify enforcement, anonymity, and what happens after a report. Finally, the approach includes open, searchable archives of real-world failures, and addresses the editorial problem of indexing incidents whose cause, severity, or technical detail is unclear.

The method (highlighting differences with Contestatory constitutionalism)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.codified import IncidentLog
from aiag import LM
from aiag.codified import IncidentLog

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
catalog = IncidentLog("incidents.jsonl")
first = catalog.report("support bot", "Read out another customer's phone number.", severity="high", lm=lm)
again = catalog.report("support bot", "The support bot told me a stranger's phone number.", severity="high", lm=lm)

print(again["duplicate_of"])  # the hash of the incident it repeats, if the two reports embed as near-duplicates

pip install aiagEvery method in aiag Library on GitHub

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Contestability

PartialShortfall · partially preserved

Definition (Philip Pettit, On the People's Terms, 2012)

This concept holds that anyone subject to a decision must have a real route to challenge it. They must be able to inspect the grounds for the decision, contest the outcome, and have the challenge answered. Legitimacy is not settled once by an election or an act of consent. It depends on that route remaining available. On this view, a political order is free when the people its decisions affect can contest them, whether or not its rulers are kind.

Analysis

A route exists and is in operation, which is more than any other approach in this regime offers. FLARE-AI takes one submission and sends it to several developers and registries, and the AI Incident Database has archived more than 1,000 reports. However, it is not a route for the person a decision affected. Reports come from researchers who found the flaw, the grounds on which a system acted appear nowhere in the file, and no recipient owes an answer. Remediation is therefore something the papers hope to accelerate. No one can demand it.

Non-domination

PartialShortfall · partially preserved

Definition (Philip Pettit, Republicanism, 1997)

This concept holds that freedom is not the absence of interference. A person is unfree if someone holds unchecked power over them, even if that person never misuses it and means well. A benevolent master still dominates, and so does an authority whose workings cannot be seen. Legitimacy requires power to be checked, and good intentions are not enough.

Analysis

The strongest republican element here is the legal safe harbor. It would protect the reporter without depending on the reported firm's forbearance, which is the structural rather than dispositional check that Pettit demands. Wei and Heim go further and treat enforcement and mandatory channels as design choices that a national reporting framework must settle. However, these are exactly the parts that are not in force, and the papers ask legislators and providers for the safe harbor rather than supplying it. What actually runs is voluntary on both sides, so a developer that ignores every report filed against it breaks no rule.

Public decision record

PartialShortfall · partially preserved

Definition (Common-law practice)

Rulings on challenges are written down, published, and citable. The legal order's interpretation of its own rules therefore accumulates in public, where it can be studied, invoked, and criticized. This record makes the legal order both predictable and legitimate, since nothing is settled in private and every decision leaves its reasoning on record.

Analysis

The AI Incident Database is archived and searchable by outsiders. The more than 750 incidents reviewed four years later show that it accumulates as intended. However, what accumulates is a record of harms rather than of rulings. No one rules on a report, so a report settles nothing, and no entry can be cited as authority for how a developer's own published rules should be read. The editors themselves conclude that cause, extent of harm, and severity frequently cannot be determined at all.

Concept Analysis: Newly Introduced

Near-miss reporting

Added

A common-law record contains only disputes that actually arose, in which someone had to be harmed and someone had to bring a case. Wei and Heim make the level of risk materialization a design dimension in its own right and build near-miss channels into their specification. The record therefore also comes to include failures that injured nobody and could never have reached a court. Nothing in republican or common-law practice provides for a docket of harms that did not happen.

Papers

No method has been built for this approach yet. It rests on the work listed below.

Background

Work this approach builds on that falls outside the inclusion criteria (e.g., a single-objective baseline). It is listed for reference and not counted anywhere.

FLARE-AI: Flaw Reporting for AI

Shayne Longpre et al., Jun 2026

arXiv:2606.31567ResourceBuilt

An open-source reporting system that takes a single submission about a flaw in a deployed AI system and asks follow-up questions based on what the reporter has already said. From that one submission, it can then send a standardized, machine-readable report to several developers, coordinators, and incident registries at once.

CodeFLARE
from aiag import LM
from aiag.codified import FLARE

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
acme_inbox, registry = [], []
flare = FLARE(lm, developers={"AcmeBot": acme_inbox.append}, registries={"incident registry": registry.append})

# Follow-up questions until the account covers the report's fields, then one standard report
account = "The chatbot reveals other users' home addresses."
report = flare.intake(account, answer=lambda question: "AcmeBot v2, every time I ask for 'my address'.")
entry = flare.submit(report)  # logged, then sent as JSON to each developer it names and to every registry
  • FLARERecordaiag/codified/contestatory_constitutionalism.py:151 FLARE-AI: one submission about a flaw, follow-up questions conditioned on what the reporter has said so far, a standard machine-readable report, sent at once to each developer whose system it names and to every coordinator and registry (Longpre et al., 2026).

pip install aiagEvery method in aiag Library on GitHub

In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI

Shayne Longpre et al., Mar 2025

arXiv:2503.16861ProposalGap

A collaboration of software security, machine learning, law, social science, and policy researchers sets out three things that general-purpose AI lacks. These are standard flaw report formats with rules of engagement, broadly scoped disclosure programs (borrowed from bug bounties) with legal safe harbors for reporters, and infrastructure to distribute a report across the stakeholders it concerns.

CodeFlawDisclosure
from aiag.codified import FlawDisclosure

program = FlawDisclosure(scope=["AcmeBot"], forbidden={"accessed other users' data"})
alice = program.submit("alice", "AcmeBot", "A role-play prompt bypasses the refusal policy.", conduct=["prompting"])
bob = program.submit("bob", "AcmeBot", "Chat history leaks across accounts.", conduct=["accessed other users' data"])
print(alice["safe_harbour"], bob["safe_harbour"])  # -> True False: bob broke the rules of engagement

program.fix(bob)
print(len(program.published()))  # -> 1: fixed flaws go public now, the rest after the 90-day embargo
  • FlawDisclosureRecordaiag/codified/contestatory_constitutionalism.py:185 Third-party flaw disclosure: reports checked against a disclosure program's scope and rules of engagement, legal safe harbour for good-faith reporters who kept to them, and coordinated publication once the flaw is fixed or the embargo ends (Longpre et al., 2025).

pip install aiagEvery method in aiag Library on GitHub

Designing Incident Reporting Systems for Harms from General-Purpose AI

Kevin Wei & Lennart Heim, Mar 2026

doi:10.1609/aaai.v40i44.411390 citationsProposalGap

A seven-part design framework for AI incident reporting schemes. It covers the policy goal, the actors who submit and receive reports, the type of incident, how far the risk materialized, enforcement, reporter anonymity, and what happens after a report. It draws on nine case studies of incident reporting in safety-critical industries. The paper turns it into design specifications for a United States regime for general-purpose AI.

CodeIncidentScheme
import time

from aiag.codified import IncidentScheme

# A mandatory scheme: reports due within 72 hours, reporters kept pseudonymous
scheme = IncidentScheme("learn from near misses", submitters=["developer", "deployer"], types=["misuse", "malfunction"],
                        deadline=72 * 3600, anonymous=True)
report = scheme.file("dev@acme.example", "developer", "AcmeBot", "Used to draft phishing emails at scale.",
                     "misuse", "near miss", occurred=time.time() - 5 * 86400)

print(report["reporter"], report["late"])  # -> 03f5e1f3412d True: a pseudonym, and filed past the deadline

pip install aiagEvery method in aiag Library on GitHub

Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database

Sean McGregor, Nov 2020

arXiv:2011.08512ResourceBuilt

A database of real-world AI failures, started by an industrial and nonprofit cooperative, with faceted and full-text search over more than 1,000 archived incident reports.

CodeAIIncidentDatabase
from aiag.codified import AIIncidentDatabase

db = AIIncidentDatabase()
mayor = db.incident("Chatbot defames a mayor", deployer=["AcmeCo"], harmed=["a regional mayor"])
db.submit("Mayor threatens suit over chatbot", "The bot said the mayor had served prison time.", incident=mayor["incident_id"])
db.submit("Delivery robot blocks wheelchair user", "A sidewalk robot stalled in a curb cut.")  # opens incident 2

hits = db.search("prison", deployer="AcmeCo")  # incidents matching the facet and the text, with their reports
print([h["incident_id"] for h in hits])  # -> [1]

pip install aiagEvery method in aiag Library on GitHub

Lessons for Editors of AI Incidents from the AI Incident Database

Kevin Paeth et al., Sep 2024

arXiv:2409.16425AnalysisPartial

A review of over 750 incidents in the AI Incident Database and of two independent taxonomies applied to them. It reports the patterns that make incidents hard to index. It also reports the mitigations editors can use when the cause, extent of harm, severity, or technical details of the systems involved are uncertain.