opsward#

Diagnose, generate, and maintain the AI agent setup of your projects — CLAUDE.md, skills, subagents, rules, and supporting docs.

Opsward works in two modes:

  • CLI mode — deterministic, pure-code analysis you run directly. No AI involved.

  • Claude Code mode — install opsward as Claude Code skills so that Claude runs the CLI tools, interprets results intelligently, and acts on suggestions. No API keys needed — Claude Code is the AI engine.

Install#

pip install opsward

CLI Mode (no AI)#

These commands are deterministic Python code — regex scoring, filesystem checks, template substitution. Same input always gives the same output.

Diagnose#

Score your project’s AI setup health:

opsward diagnose .

Diagnosis Report: myproject
Project type: python
Overall score: 72/100  (Grade: C)

Components:
  CLAUDE.md quality         [################....] 81/100
  Documentation             [##############......] 70/100
  Skills                    [############........] 60/100
  Setup (rules/agents/hooks) [##########..........] 50/100
  Cross-references          [####################] 100/100

Missing:
  [ ] docs_guide.md
  [ ] docs/known_issues.md

Suggestions:
  1. Create a docs_guide.md to index your documentation
  2. Consider adding hooks in .claude/hooks.json

Generate#

Create missing artifacts (dry run by default):

opsward generate .
opsward generate . --write   # actually create files

Generates CLAUDE.md, docs (architecture, conventions, known_issues, etc.), skill templates, and agents — only what’s missing, never overwrites existing files.

Maintain#

Find stale references and drift:

opsward maintain .

myproject: 3 issue(s)

  [stale_path → align] CLAUDE.md references `src/old_module.py` but it does not exist
  [sync_issue → align] `new_doc.md` exists in docs/ but is not listed in docs_guide.md
  [empty_doc → align] `conventions.md` appears to be an empty stub (12 bytes)

Each finding carries a recommended triage: align (the code is right, update the config), revert (the config is right, restore the code) or ignore (the reference is legitimately volatile). To ignore one, put <!-- opsward: ignore-path --> at the end of its line, or alone on the line before it; every path check (stale paths, the currency score, cross-references, ADR drift) then skips it.

Config changelog (opt-in): create config_changelog.md in the docs folder and give every change to the agent config (CLAUDE.md, AGENTS.md, .claude/) one dated line, newest first, saying what changed and why, e.g. - 2026-10-20: moved the CLI-golden procedure out of CLAUDE.md (over budget); align. Once the file exists, opsward maintain reports a config_changelog finding whenever a config commit is newer than the last changelog commit.

Fleet#

Diagnose many repos at once and roll the results up (grades, what is missing where, the weakest components, the worst repos):

opsward fleet ~/proj/a ~/proj/b            # explicit roots
opsward fleet --under ~/proj --depth 2     # every git repo under a folder
opsward fleet --under ~/proj --manifest ~/.local/share/opsward/fleet.json
opsward fleet --manifest ~/.local/share/opsward/fleet.json   # re-run the remembered fleet

--manifest remembers the fleet: it writes the roots and the latest name-keyed summary to that JSON file. The roots are this machine’s paths, so keep the manifest out of version control (opsward warns when it sits in a git work tree unignored).

Output Formats#

All CLI commands support --format json for machine-parseable output:

opsward diagnose . --format json
opsward generate . --format json
opsward maintain . --format json

Claude Code Mode (AI-enhanced)#

Install opsward’s skills into Claude Code, and Claude becomes an intelligent layer on top of the deterministic tools. It doesn’t just run opsward — it goes beyond the heuristic scores by reading actual source code, reasoning about accuracy, and making intelligent edits.

Install Skills#

opsward install-skills --write                    # into ./.claude/ (project-level)
opsward install-skills --global-install --write   # into ~/.claude/ (all projects)

What the Skills Do#

Once installed, these skills activate automatically in Claude Code when you ask the right thing:

Skill

Trigger

What it does

opsward

“check my setup”, “opsward”

Diagnose → decide next step → generate or maintain → re-diagnose

opsward-diagnose

“audit my AI config”

Run opsward diagnose, then read code to assess semantic quality, offer fixes

opsward-generate

“scaffold AI setup”

Run opsward generate, then read the codebase and replace templates with real content

opsward-maintain

“check for staleness”

Run opsward maintain, then check for semantic drift (docs that no longer match code)

How It Works#

  1. Opsward CLI runs deterministic checks (regex scoring, path validation, template substitution) — fast, reproducible, no AI

  2. Claude reads the output and adds deeper analysis: reads actual source code, checks if docs match reality, verifies commands are correct

  3. Claude proposes fixes — not just what opsward suggests, but what it discovers by understanding the code

  4. Claude applies fixes with user approval, then re-runs opsward to show improvement

The CLI provides the structural analysis. Claude provides the semantic understanding and action.

Permissions#

The skills use Claude Code’s standard permission model — no special permissions are assumed or required:

  • Read-only operations (reading files, searching code, running opsward diagnose): always safe, used freely

  • Write operations (creating docs, editing CLAUDE.md): Claude Code prompts the user for each action per their permission settings

  • Destructive operations (deleting files, removing content): always ask for explicit confirmation

If you want faster workflows (e.g., auto-approve file creation during generation), you can configure that in your Claude Code permission settings — but opsward skills never assume it.

What It Checks#

CLAUDE.md quality (6 dimensions):

  • Commands & workflows — are build/test/lint commands documented?

  • Architecture clarity — is there a module map with role descriptions?

  • Conventions — are project-specific style rules present?

  • Conciseness — is the file scannable, not bloated?

  • Currency — do referenced paths actually exist?

  • Actionability — are instructions specific enough to act on?

Documentation completeness: docs_guide.md, architecture.md, conventions.md, known_issues.md, and content quality.

Skills & agents: SKILL.md presence, descriptions, setup-auditor agent.

Cross-references: paths in CLAUDE.md validated against the filesystem.

Portability: sections where a render (CLAUDE.md, GEMINI.md, …) drifts from AGENTS.md, 10 points each; never how many hosts a repo covers. Without an AGENTS.md it is not scored and its weight goes to CLAUDE.md quality (ADR 0012).

Overall health: weighted score (A–F grade) combining all components.

Security advisories (reported, never scored; modelled on the OWASP Top 10 for agentic applications): unrestricted Bash in a skill’s allowed-tools or in permissions.allow, bypassPermissions, a download piped into a shell, eval of a built string, unpinned npx/uvx packages in hooks and MCP servers, text that tries to override the agent’s other instructions, and committed credentials. They are heuristics, so they never move the grade or fail --min-score.

Python API#

from pathlib import Path
from opsward import scan, diagnose, generate, generate_skills, maintain
from opsward import recommend_skills, validate_skill_spec

sr = scan(".")
report = diagnose(sr)
print(report)  # human-readable report card
print(report.grade)  # 'A', 'B', 'C', 'D', or 'F'

files = generate(sr)  # list[GeneratedFile]
issues = maintain(sr)  # list[MaintenanceSuggestion]

# Recommend ecosystem skills based on tech stack
recs = recommend_skills(sr)  # list[SkillRecommendation]

# Validate skills against agentskills.io spec
for skill in sr.skills:
    violations = validate_skill_spec(skill)

# Install skills programmatically
skill_files = generate_skills(Path.home() / ".claude")

Evals (opt-in, report-only)#

The rubric scores form; evals ask whether a config actually helps. They make model calls, so they never gate and never run by default:

from pathlib import Path
from opsward.evals import (
    judge_claude_md,
    anthropic_judge,
    run_golden_tasks,
    GoldenTask,
    anthropic_agent,
)

# LLM-as-judge for the subjective dimensions (actionability, conciseness), next to the rubric's score
print(
    judge_claude_md(Path("CLAUDE.md").read_text(), judge=anthropic_judge(), samples=3)
)

# Golden tasks: does the agent honour a convention with the config, and not without it?
tasks = [GoldenTask("kw-only", "Write a function f(a, b, c).", expect=[r"\*,"])]
print(
    run_golden_tasks(
        tasks, agent=anthropic_agent(), config=Path("CLAUDE.md").read_text(), runs=3
    )
)

pip install 'opsward[eval]' for the paid judge and agent. Without them, judge_claude_md defaults to rubric_judge (opsward’s own rubric, free) and echo_agent exercises the golden-task harness without a model.

From the shell, opsward eval runs both and exits 0 whenever it ran (a report, never a gate; 2 on a usage error). It finds a repo’s golden tasks at <docs>/evals/golden_tasks.json (misc/docs/ for Python, docs/ for JS/TS) unless --tasks names a file:

opsward eval .                                   # free: the rubric judge and the echo agent
opsward eval . --judge anthropic --samples 3     # paid model calls (pip install 'opsward[eval]')
opsward eval . --tasks evals.json --runs 5 --format json

CI Integration#

Use opsward in CI to enforce AI setup quality:

# Fail if overall score drops below 60
opsward diagnose . --min-score 60

# Machine-parseable output for CI tooling
opsward diagnose . --format json --min-score 60

# .github/workflows/ai-setup-check.yml
name: AI Setup Check
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install opsward
      - run: opsward diagnose . --min-score 60

Two gates, for two jobs: diagnose --min-score is the cheap scored gate, and lint is the strict one, pass or fail with no score. lint checks every project skill against the agentskills.io spec and the hooks config for shapes Claude Code silently ignores, and exits 1 on any finding:

opsward lint .                          # opsward's own spec check, plus skill.validate() when installed
opsward lint . --validators opsward     # only opsward's check
pip install 'opsward[lint]'             # adds the `skill` package's validate()
# external linters, opt-in by name (opsward does not install them; `auto` never runs them):
npm install -g agnix                    # --validators agnix
pip install skillcheck                  # --validators skillcheck

One brief for every agent host: AGENTS.md is the canonical, and each host’s own file (CLAUDE.md, GEMINI.md, .github/copilot-instructions.md, …) is a symlink, an @AGENTS.md import, or a copy that can drift. portability shows which hosts find instructions and every section where a render differs from AGENTS.md; fleet rolls the same counts up across repos. With opsward[lint] installed, portability also lists project skills rendered for a host whose format drops some of their fields (allowed-tools, license, bundled resources): the lossy kind, reported and never scored. A CLAUDE.md that is @AGENTS.md plus Claude-only deltas is scored as Claude reads it, with the import inlined.

opsward portability .                   # host coverage table + drift findings (report-only)
opsward portability . --format json

The repo ships a reusable GitHub Action and pre-commit hooks:

# a workflow step, after actions/checkout
- uses: thorwhalen/opsward@<a release tag>
  with:
    min-score: "70"   # "" skips the scored gate
    lint: "true"

# .pre-commit-config.yaml
- repo: https://github.com/thorwhalen/opsward
  rev: <a release tag>
  hooks:
    - id: opsward-lint
    - id: opsward-diagnose
      args: [--min-score, "70"]

built 2026-11-08 17:24 UTC from b4ebda3 (main) · opsward 0.0.41 · about this build