Memory-Grounded Review Councils
Explores multi-perspective evaluation of agent output through structured critic councils grounded in durable memory.
Introduction
Evaluating agent output is difficult. A single evaluator, whether human or automated, brings a single perspective. It may catch certain types of errors while missing others. Single-perspective evaluation creates a new single point of failure in systems that are specifically designed to avoid single points of failure.
The proposed Review Council pattern would distribute evaluation across multiple independent critics, each grounded in durable memory and evaluating from a different perspective. Review Councils are not implemented in the current product.
Council Architecture
A review council consists of multiple critics configured to evaluate the same agent output. Each critic has:
- A defined evaluation perspective (correctness, consistency, source grounding, invariant compliance, style, safety)
- Access to relevant durable memory
- Access to relevant negative memory
- Independence from other critics in the same council
Critics run in parallel and produce individual findings. These findings are then aggregated into a council verdict.
Findings as Evidence
A critical design decision in InvariantEngine is that council findings are treated as evidence, not truth. No single critic’s finding overrides the Session Integrity Layer’s judgment. No council verdict is automatically enforced. Instead, findings and verdicts are provided to the Session Integrity Layer as inputs to its decision-making process.
This distinction matters. If council verdicts were treated as authoritative, the council would simply replace one single point of failure with another. By treating verdicts as evidence, the system maintains its multi-layered validation architecture.
Grounding in Memory
What distinguishes InvariantEngine’s review councils from generic multi-agent evaluation is memory grounding. Each critic is not just evaluating the output in isolation—it is evaluating the output against the project’s durable memory. A critic checking for consistency is checking against the project’s recorded decisions, not against its own general knowledge. A critic checking for source grounding is verifying evidence chains against the project’s approved sources.
This grounding helps ensure that evaluation is project-specific and context-aware, not generic.
Councils Are Not Debates
A Review Council is not a debate among agents. Critics do not negotiate consensus with each other. They produce independent findings, and a fresh-context synthesis step aggregates those findings as evidence. The synthesis step checks aggregated findings against durable memory, negative memory, source context, and invariants before influencing decisions.
Aggregation
Council verdicts synthesize individual findings into an overall assessment. The aggregation process must handle conflicting findings—where one critic approves and another flags concerns—without simply taking a majority vote. The aggregation logic considers the confidence and severity of each finding, the perspective it represents, and the relevance of the evaluated dimension to the current action.
Conclusion
Review Councils are a proposed pattern for structured, multi-perspective evaluation grounded in durable memory. If implemented, their findings would remain evidence rather than truth. The proposed specification is defined in IE-004.