IE-002: Hallucination Gauge

Implementation Status

Implemented deterministic v0 / deeper runtime proposed. The current operator workflow has a pure deterministic read-only evaluator that maps already-loaded quality, drift, contradiction, proposal, provenance, Roslyn, and related workflow signals into a 0–100 score, color band, ordered reasons, recommended action, and hard-stop state. Diagnosis uses the authoritative subset loaded server-side; client claims cannot force a blocker.

Gauge v0 is surfaced as review evidence in operator and handoff artifacts. It does not call an LLM, persist scores, perform semantic detection, certify truth, or autonomously stop or mutate a workflow. Continuous per-agent-action scoring, learned or adaptive weights, semantic alignment, and automatic runtime intervention described later in this specification remain proposed.

Abstract

The Hallucination Gauge is a runtime evaluator that estimates how far a current agent session has diverged from the project’s approved, governed state. It produces a normalized operational risk score used by the Session Integrity Layer to determine whether intervention is needed. The score is not a probability of hallucination, not a truth score, and not comparable across projects unless the same signal model and thresholds are used. This specification defines the gauge’s inputs, scoring model, thresholds, and runtime behavior.

Problem

Agent sessions accumulate drift over time. Each action that relies on approximate retrieval, stale context, or unverified assumptions increases the distance between the session’s working state and the project’s approved state. Without a quantitative risk estimate, the system cannot make informed decisions about when to intervene.

Definition

The Hallucination Gauge is a runtime component that evaluates the current session state and produces a normalized drift-risk score between 0.0 (low estimated risk) and 1.0 (high estimated risk). The score is computed by evaluating multiple configurable drift signals and aggregating them into a single operational metric. The score represents estimated risk, not measured truth.

Motivation

Post-hoc hallucination detection is insufficient for long-running systems. By the time a hallucination is detected in output, the session may have already built subsequent decisions on fabricated foundations. The Hallucination Gauge provides continuous, runtime-level drift awareness.

Runtime Behavior

1. Collect signals: After each agent action, the gauge collects drift signals from memory alignment, source grounding, invariant compliance, and negative memory checks.
2. Score: Each signal is weighted and combined into a composite drift-risk score.
3. Threshold evaluation: The score is compared against configurable thresholds for warn, repair, re-ground, escalate, and abandon actions.
4. Report: The score and contributing factors are reported to the Session Integrity Layer.

Inputs

• Current agent action and output
• Durable memory alignment results
• Source grounding verification results
• Invariant validation results
• Negative memory check results
• Session history (previous drift scores)

Outputs

• Drift-risk score (0.0 to 1.0)
• Contributing signal breakdown
• Recommended action based on threshold evaluation
• Score history for trend analysis

Failure Modes

• Score inflation: Too many minor signals aggregate into false alarms.
• Score suppression: Critical signals are diluted by low-risk signals.
• Threshold rigidity: Static thresholds do not adapt to project context or session phase.
• Signal blindness: The gauge does not evaluate a relevant drift dimension.

Open Questions

• What drift signals should be included in the default configuration?
• How should signal weights be determined? Static configuration, learned, or adaptive?
• Should the gauge maintain a rolling window or cumulative session score?
• How should the gauge handle sessions that recover from high drift through re-grounding?

Non-goals

• The Hallucination Gauge is not a truth oracle. It does not certify outputs as correct or safe.
• It is not a hallucination probability. The score estimates operational drift risk, not the likelihood of a specific factual error.
• It does not prove an output is true or false.
• Scores are only comparable across projects if the same signal model and thresholds are used.

Relationship to Other IE Primitives

• IE-001 Session Integrity Layer consumes the gauge’s score to make runtime decisions.
• IE-003 Negative Memory provides one of the gauge’s input signals.
• IE-004 Review Councils may produce findings that affect the drift score.