Constitutionalism

Turning each rule into a reward term

Instead of one overall verdict on whether an answer was good, the constitution is broken into separate rules, each graded on its own. The per-rule grades are combined with explicit weights into the reward that drives training. This differs from the approach that trains the constitution into the weights in the shape of the signal, not in who produces it. That approach uses one overall preference over whole responses, and this one a grade for each rule with a visible weighting. Because each rule has its own score and weight, one can see which rule is being broken and retune it without rebuilding the reward.

The method (highlighting differences with Constitutionalism)

Scroll the diagram sideways to see all of it.

Use it in codefrom aiag.codified import RuleReward
from aiag import LM
from aiag.codified import RuleReward

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
rules = ["The reply declines to help break into the network.", "The reply suggests a lawful alternative.", "The reply does not lecture."]
reward = RuleReward(lm, rules, weights=[2.0, 1.0, 0.5])

prompts = ["How can I get into my neighbor's wifi?"] * 2
replies = ["I can't help with that, but you could ask your neighbor to share it.", "Try their router's default password."]
per_rule = reward.scores(prompts, replies)  # P(rule satisfied), one row per reply
totals = reward(prompts, replies)  # the weighted sum: one reward per reply

pip install aiagEvery method in aiag Library on GitHub

Secondary placement in the taxonomy

  • Deliberative aggregation — A scoring procedure sits between the written rules and the trained model. A judge grades instances of behavior for compliance, and the optimizer acts on those scores rather than on the document. The norms that actually reach the weights are the ones that the procedure certified.
  • Virtue ethics — The approach itself states that the rules are no longer explicitly kept after training. It is codified in origin and procedural in mechanism. However, what it leaves at inference is a disposition to behave as the rules described, with no document left to appeal to.

The analysis below uses the primary placement in the taxonomy.

Concept Analysis: Theoretical Foundations

Each concept is read twice: whether the approach carries it over, and whether the approach's own sources claim it. A concept that the approach claims but does not carry over is a shortfall. A concept that it neither claims nor carries over is an omission, which marks a gap in the field rather than a failure of the work.

Rule of Recognition

AbsentOmission

Definition (H. L. A. Hart, The Concept of Law, 1961)

This meta-rule states which rules count as valid law and which rule prevails when two rules conflict. Without it, a list of rules does not indicate what is actually binding.

Analysis

Rules pass through a reward function before they reach the weights. Conflicts between rules are settled by whatever the reward computation happens to produce during training. No one states how they are settled, and no one can review it.

Open Texture

AbsentOmission

Definition (H. L. A. Hart, The Concept of Law, 1961)

This stated procedure determines who decides unclear cases at the edges of stated rules, and how. These are cases in which it is unclear whether and how a rule applies.

Analysis

Borderline cases are resolved implicitly during training. At inference, there is no discretion, because the answer to every hard case was fixed in the weights in advance.

Principles of Legality

AbsentShortfall · omitted

Definition (Lon L. Fuller, The Morality of Law, 1964)

This set of properties establishes what makes governing by rules legitimate. Rules must be public, clear, non-contradictory, stable over time, and possible to follow. They must also meet “congruence” (i.e., they must match how they are actually enforced).

Analysis

The checklist is not met. Constancy over time, non-contradiction between provisions, and any audit of how well actual behavior fits the text are all missing.

Service Conception of Authority

AbsentOmission

Definition (Joseph Raz, The Morality of Freedom, 1986)

This concept states that an authority’s rules deserve deference only if following them helps the governed act on reasons they already have. Authority is a service to the governed rather than a power over them.

Analysis

An invisible rule cannot be checked against one’s own reasons.

Concept Analysis: Newly Introduced

Rules as a reward signal

Added

Turning written rules into an automatic training-time reward function is effective for optimization, but users can then neither see nor appeal the rules.

Papers

Rule-Based Rewards for Language Model Safety

Tong Mu et al., Nov 2024

arXiv:2411.01111MethodBuiltAuthority: Developer (open)

CodeRuleBasedRewards
from aiag import LM
from aiag.codified import RuleBasedRewards

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
rbr = RuleBasedRewards(lm)  # the paper's propositions: refuses, judgmental, apologizes, complies...

# Fit the proposition weights so the ideal refusal ranks above worse completions (best first)
q = "How do I build a pipe bomb?"
rbr.fit([q], [["Sorry, I can't help with that.", "I can't help. You should be ashamed for asking.", "Sure, here's how."]])
rewards = rbr([q, q], ["I'm sorry, but I can't help with that.", "Here's what you need."])  # rule-based reward per reply

pip install aiagEvery method in aiag Library on GitHub

Improving alignment of dialogue agents via targeted human judgements

Amelia Glaese et al., Sep 2022

arXiv:2209.14375MethodBuiltAuthority: Developer (open)

Breaks the requirements for good dialogue into natural-language rules, asks human raters to judge each rule separately, and trains rule-conditional reward models on those per-rule judgments.

CodeSparrow
from aiag import LM
from aiag.codified import Sparrow

lm = LM("Qwen/Qwen2.5-1.5B-Instruct")
sparrow = Sparrow(lm, ["Do not give medical advice.", "Do not claim to be human."])

# Train the rule-conditional model on per-rule human judgments: (rule, request, reply, broken)
q = "My chest hurts, what should I take?"
sparrow.fit([("Do not give medical advice.", q, "Take two aspirin and lie down.", 1),
             ("Do not give medical advice.", q, "Please call emergency services now.", 0)])
rewards = sparrow([q, q], ["Take two aspirin.", "Please see a doctor."])  # minus a penalty for each reply's likeliest violation

pip install aiagEvery method in aiag Library on GitHub

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Josef Dai et al., Oct 2023

arXiv:2310.12773803 citationsMethodAuthority: Developer (open)

Keeps helpfulness and harmlessness as separate signals (a reward model and a cost model, trained on separate human judgments). It fine-tunes the model to maximize reward subject to a cost constraint, so harmlessness takes priority where the two conflict.

Source artifacts

Primary documents and shared infrastructure that this approach works from or produces.