Agent Self-Improvement Harness (4/12) — self-audit: Self-Checking Harness Compliance
Key criteria, automated inspection process, and the reality of harness maintenance
Summary
- A prompt harness does not stay correct after creation. Over time, structure drifts and rules become dead letters. The self-audit skill runs periodic compliance checks.
- The skill scans harness criteria across multiple items and grades each as pass / partial / missing, then produces a percentage score and letter grade.
- Initial compliance on first audit was low — only a small fraction of items fully passed. Targeted remediation raised the score significantly. A harness is operational from the day it is audited, not the day it was written.
Background
Projects begin with a carefully designed harness: rules defined in CLAUDE.md, agents separated by role, gate conditions enforced, memory systems initialized.
What happens once the project moves from design into active operation?
New skills are added without updating CLAUDE.md. A SKILL.md file is created but never registered in the trigger table. A reference file is dropped under docs/ but not indexed in memory-map.md. plan.md grows past its 50-line limit and reaches 200 lines.
Each gap is minor in isolation. Accumulated, they erode the harness structure — rules exist on paper but nothing enforces them. Detecting this state requires systematic inspection.
That is the purpose of the self-audit skill.
Body
1. Harness Criteria
self-audit evaluates a project's harness against criteria organized into five domains.
CLAUDE.md sections:
A representative subset:
| # | Item | Pass Condition |
|---|---|---|
| 1 | Development Environment | Table containing OS, Language/Framework, Package Manager |
| 4 | Workflow Pipeline | Ambiguous Request Gate + Task Flow present |
| 5 | Agent Role Separation | orchestrator / executor / quality roles defined, Writes Code column present |
| 7 | Gate Conditions | Phase / Entry / Exit table, minimum 4 rows |
| 8 | Built-in Rules | brainstorming alternatives, writing-plans concreteness, execution circuit-breaker, verification evidence — all 4 rules present |
| 10 | Model Routing | Task-type → model table (haiku / sonnet / opus) |
| 11 | Memory System | All 3 layers described (docs, lessons, sessions) |
| 15 | Sub-agent Isolation | 4 rules: no history passthrough, handoff extraction, 1:1 task, results only |
| 20 | Token Efficiency | 4 rules: no re-read, no restatement, scope-limited, session ground truth |
| 21 | Principles | no-action default, simplicity, root-cause, prohibitions > guidance, no filler |
Each item is evaluated not merely for presence but for content completeness. A Memory System section that exists but omits one of the three layers receives a partial pass (warning), not a full pass.
Non-CLAUDE.md checks:
| Domain | Targets |
|---|---|
| .claude/ components | Core skills, agent files, hooks |
| docs/ structure | memory-map.md + required subdirectories |
| tasks/ files | Core tracking files + required directories |
| Cross-validation | Skill path existence, memory-map reference validity, agent file existence |
Cross-validation is critical. If CLAUDE.md's skill trigger table lists a path that does not exist on disk, the agent will fail when it attempts to invoke that skill. self-audit catches these phantom references.
2. Automated Inspection Process
Executing self-audit runs five phases.
Phase 1: CLAUDE.md section scan. Each criterion is matched against search patterns. For example, the Development Environment item searches for a ## Development Environment or ## Dev Environment heading and then confirms that OS, Language/Framework, and Package Manager rows appear beneath it.
Phase 2: .claude/ component check. The file system is scanned: .claude/skills/*/SKILL.md for skills, .claude/agents/*.md for agents, .claude/hooks/*.sh for hooks. Findings are compared against the expected file list and missing items are reported.
Phase 3: docs/ structure check. Verifies that memory-map.md exists and is not empty (not in template state), and that required subdirectories are present.
Phase 4: tasks/ file check. Confirms existence of core tracking files and directories. Flags plan.md if it exceeds 50 lines.
Phase 5: Cross-validation. Paths declared in CLAUDE.md are verified against the actual file system.
Each item is scored: full pass = 1, partial pass = 0.5, missing = 0. Total score is divided by item count and expressed as a percentage.
| Grade | Score | Meaning |
|---|---|---|
| A | 90–100% | Full compliance |
| B | 70–89% | Minor gaps — address in current session |
| C | 50–69% | Significant gaps — dedicated remediation session required |
| D | < 50% | Major overhaul required |
3. First Audit: Starting from Low Compliance
Results from running self-audit immediately after initial harness deployment:
Audit breakdown: - Full pass: a small number of items - Partial pass: some items - Missing: the majority
Compliance grade: low.
At the point the harness was considered "complete," actual compliance was low. CLAUDE.md had section headings present but content still in template state. Skill files existed but their SKILL.md was empty. memory-map.md existed but contained no keyword entries. These patterns dominated the initial results.
The audit generated specific remediation instructions for each missing item. Items were prioritized and addressed one by one. The process also informed the design of an inter-agent remediation flow, where the harness agent forwards correction messages to each project agent.
Prompt Harness Agent
├─ Execute self-audit → collect results
├─ Generate remediation instructions per missing item
└─ Send messages to target agents
└─ "Memory System section in CLAUDE.md requires all 3-layer description"
Each agent self-corrects its own harness, then re-runs self-audit to confirm resolution.
4. Audit Frequency and Triggers
Recommended cadence: once per month or after any major change.
Major changes include: adding a new skill, restructuring agents, modifying core CLAUDE.md sections, or expanding project scope. Such changes can break consistency with existing structure.
self-audit reports findings only — it does not auto-correct. This is an intentional design decision. Allowing automatic correction would cause agents to fill in templates without understanding the required context. How to address each missing item is a human judgment call.
Design Decisions
Initially, the project had only the project-doctor skill — no self-audit. project-doctor checks build integrity, security, and memory consistency. Its question is: "Is the project functioning?"
The limitation is that project-doctor does not inspect structural harness compliance. A project can function correctly even without a Gate Conditions section in CLAUDE.md. Agents will simply operate without gates, and output quality will degrade. Functioning and compliant are different dimensions.
That distinction motivated splitting self-audit into its own skill. project-doctor asks "Is it broken?"; self-audit asks "Is it following the rules?"
The scoring model was initially binary (pass / fail). In practice, the dominant pattern was "section exists but content is incomplete." Adding partial pass (0.5 points) made audit results reflect reality more accurately.
Closing
A harness is operational from the day it is audited, not the day it was written.
No matter how carefully CLAUDE.md is authored, operational drift accumulates: new skills added but missing from the trigger table, directories created but not indexed in memory-map, plan.md grown too large for agents to read effectively.
self-audit quantifies this drift. Each criterion rated pass / partial / missing. A total score and grade. Specific remediation instructions.
Across this series: gate systems control task flow, model routing balances cost and quality, and self-audit confirms the overall structure remains intact. Design, operation, maintenance — these three axes define a prompt harness.
댓글
댓글 쓰기