Code Intelligence Roadmap

coding-ethos should grow from policy enforcement into a local code intelligence substrate for agents. The target is not a generic RAG database. The target is ETHOS-grounded code and remediation memory that helps agents find the right code, understand prior failures, and choose the enforced repair path before they run broad shell commands or repeat failed edits.

Design Position

Use explicit storage layers with separate logical responsibilities:

Vector indexes and DuckDB query indexes are derived artifacts. Source readiness and vector readiness are reported separately so a missing embedding cannot make an exact source generation look stale, or vice versa.

Goals

Non-Goals

Architecture

repository
  files
  traces
  SARIF
  policy bundle
      |
      v
Central language registry
  Tree-sitter / Goldmark / PurRDF batch extractor
      |
      +--> immutable source generation
      |      shared content-addressed fragments + base manifest
      |      lane-local delta manifest + tombstones + status receipt
      |
      +--> DuckDB derived/telemetry store (v1-compatible)
      |      files
      |      symbols
      |      AST nodes
      |      graph edges
      |      decisions
      |      remediations
      |      attempts
      |      DuckDB term indexes
      |
      +--> duckdb-vss vector backend
             code/remediation vectors
             metadata-filtered similarity search
	     pure-Go github.com/duckdb/duckdb-go/v2

Foundation Already In Place

The first enabling layer is the normalized evidence model in go/internal/evidence. It gives CEL, SARIF, traces, lint diagnostics, hook decisions, and future code intelligence one shared contract:

SARIF result properties now carry the normalized finding, source span, search text, remediation payload, and remediation events. Hook and lint traces carry a schema version, trace ID, normalized findings, remediation summaries, and remediation events. This makes code intelligence an ingestion problem instead of a second policy interpretation layer.

CEL inputs also expose code-intelligence fields under source, finding, and edit/diff facts. proposed_symbol_changes compares current and proposed Tree-sitter symbols for Edit/Write/MultiEdit actions, while changed_symbols maps staged diff hunks to the affected Tree-sitter symbols. Both surfaces report the file, language, node kind, symbol kind/name/path, line spans, content hashes, action (added, deleted, or modified), and line-count delta. This lets principle-owned CEL block growth of oversized functions, classes/types, shell functions, and YAML config entries while still allowing refactors that shrink large files.

Source-aware policy follows AST_CEL_SARIF_ARCHITECTURE.md: Go collects Tree-sitter facts, CEL evaluates configurable predicates, and SARIF reports stable AST-backed findings. Code-intel storage and MCP retrieval build on those same facts; they must not become a second parsing or policy interpretation path.

The AST layer follows the resolver pattern proven in ~/Active/pyqa_lint: language detection, parser binding, parser reuse, tree traversal, and line-to-nearest-context lookup live behind one Go entrypoint. The active resolver supports Go, Python, JavaScript/TypeScript/TSX, Rust, shell, YAML, JSON, TOML, and Markdown. JSON and TOML are treated as first-class config-policy surfaces, so agents can retrieve precise config entries instead of reading whole config files. Every registry entry exposes an exact parser fingerprint. Turtle, TriG, N-Triples, N-Quads, and SPARQL use the optional one-shot coding-ethos-purrdf-extractor; a missing or mismatched external extractor fails sync, and is never silently replaced by regex facts.

Tree-sitter-backed policy diagnostics carry AST metadata into SARIF result properties and partial fingerprints. Code scanning can therefore track the symbol-level finding across unrelated line movement instead of treating every nearby edit as a new whole-file violation.

Python policy enforcement now uses the same AST foundation for import and functional-idiom rules. The python.conditional_imports evaluator blocks write-time attempts to introduce nested imports, TYPE_CHECKING import branches, module __getattr__ shims, __import__, and importlib.import_module. The python.functional_idioms evaluator records assigned lambdas and closure factories so agents get principle-grounded advice to use functools, operator, or itertools helpers instead of ad-hoc closures.

The storage layer lives in go/internal/codeintel. It creates the canonical .coding-ethos/code-intel.duckdb DuckDB store, ingests retained lint and hook traces, stores normalized findings/remediations/remediation events, indexes Tree-sitter chunks, records SARIF-to-AST links, and builds DuckDB search-term tables over policy IDs, skill IDs, paths, messages, code chunks, and remediation text. duckdb-vss is active for derived vector rows, but DuckDB facts remain the auditable source of truth.

Writable stores use schema v2. Search identities are enforced by unique constraints on code_intel_fts.fts_id and on (term, fts_id) in code_intel_search_terms. Upgrading or replaying a v1 store collapses exact duplicate rows, fails closed when one identity carries conflicting content, and reports row and duplicate counts through store statistics. Migration manifests use kind v2, record deduplicated_rows, and exclude the schema metadata row from migrated-data accounting. File, chunk, and graph-edge replays use conflict-aware updates so rebuilding search state does not discard retained AST or foreign-key evidence.

Graph facts expose provenance classes wherever repo maps, graph reports, and MCP graph surfaces show those facts. EXTRACTED marks parser/static-analysis facts, while GIT_DERIVED, POLICY_DERIVED, TRACE_DERIVED, and DOC_DERIVED mark deterministic derived evidence. INFERRED and AMBIGUOUS are advisory only and must not become enforcement inputs without a deterministic source fact behind them. Existing records without explicit provenance default to EXTRACTED so migrated indexes stay conservative.

Code health scoring is another derived DuckDB surface. code-intel health and MCP code_intel_health compute deterministic refactoring rankings from indexed files/chunks, large files/functions, complex functions, complex conditionals, exact structural clones, git hotspots, co-change coupling, ownership risk, repeated lint/hook failures, and LCOV line coverage. Snapshots, targets, evidence rows, and bounded trend history are persisted so every score is explainable by biomarker and evidence ID. Refreshes persist repo-wide snapshots and apply path filters only when reading the ranked targets, so targeted review queries cannot replace the repository health baseline. Consumer repos can reweight or disable biomarkers by glob under code_intel.health in repo_config.yaml for generated, vendor, legacy, or test paths.

Session snapshots are exposed as code-intel session-snapshot and MCP code_intel_session_snapshot. They derive the stable coding_ethos.session.v1 contract from existing hook traces, proxy sessions/events/transforms, memory trace activity, and code-intel freshness metadata. Provider-specific details stay nested under provider.adapters, so agents can inspect current blockers and linked trace IDs without coupling to provider-specific event files or doing broad source reads.

Automatic output pruning applies the configured code_intel_db row-retention policy after high-volume code-intel writes. The default keeps 90 days of trace and proxy-event rows, leaves current AST/code chunks to the index refresh path, checkpoints the DuckDB store, and leaves DuckDB WAL files to checkpoint-managed cleanup instead of retention deletion. Use explicit output-prune maintenance with --vacuum when database compaction is needed.

Hook traces are also normalized into analytics tables. Each hook event stores provider, tool, status, tracking ID, operation kind, target kind, risk category, command and target-set fingerprints, runtime, rewrite state, and target paths. Each decision stores policy/skill IDs, implementation, severity, principle IDs, diagnostic counts, and message/suggestion variant hashes. This lets later analysis answer which hook checks create the most friction, which advice text reduces repeat violations, which operations are rewritten versus blocked, and which targets are frequently involved without reparsing raw provider payloads. Operator review rows can label a hook event as a correct block, false positive, unclear message, over-broad policy, or missing allow-list case.

Skill use is measured through the same remediation-outcome evidence store. MCP skill_lookup and skill_recommend record unknown-outcome observations when a code-intel root is configured, while explicit remediation outcomes record observable success, repeated failure, abandonment, and retry counts. skill-health and MCP code_intel_skill_health report unused, frequently failing, improving, stale, and unknown skill IDs with 7-day and 30-day windows in JSON or TOON. Learned or evolved skills are not promoted into generated repo artifacts by this report; promotion requires a separate explicit policy change.

Agent Proxy foundation data uses the same store. Proxy events are provider-neutral records for outbound provider calls, tool calls, file reads, file listings, payload injections, payload truncations, cache hits, and edit proposals. The session ledger stores request counts, file-read/listing counts, edit counts, cache hits, injection/truncation/denial counts, payload hashes, cache keys, trace/tracking IDs, direction, payload kind, DLP facts, policy evidence, token usage, payload byte counts, and ordered transform records. This is the storage substrate for future context-economy controls; it is not a separate shadow database. Inbound and outbound policy denials are exposed through MCP code_intel_proxy_denials, which queries the same ledger instead of reinterpreting proxy policy. The trust boundary and event contract are documented in AGENT_PROXY.md.

Exact Source Generations (coding-ethos.code-intel/v2)

Run a source sync for the exact worktree that an agent will use, then inspect the typed receipt:

coding-ethos-code-intel sync --root /path/to/worktree
coding-ethos-code-intel status --root /path/to/worktree

RepositoryID hashes the normalized origin URL plus the sorted repository root-commit set, so equivalent clones share an identity while unrelated histories cannot collide. The normalization removes credentials, default ports, a trailing .git, query, and fragment; SCP-like SSH and canonical SSH URLs normalize identically. A repository without origin uses the explicit path-local fallback local-common-dir:<canonical-git-common-dir> plus the same root-commit set. Shallow repositories fail closed because their true root commit is unavailable. SourceSnapshotID hashes repository identity, HEAD, current eligible source bytes, config, and the exact extractor set, and GenerationID hashes the snapshot plus its base and delta manifests. A status is exact only when every eligible file is indexed by the declared extractor. partial, failed, missing, and stale are explicit non-exact states. Coverage is reported per language with eligible, indexed, cache_hits, unsupported, excluded, oversized, failed, and stale counts.

When Turtle, TriG, N-Triples, N-Quads, or SPARQL input is eligible, sync first performs an empty protocol exchange with the pinned PurRDF helper. A missing helper or any protocol, version, name, or PurRDF-revision mismatch is a hard sync error, including when immutable fragments could otherwise be reused. Non-RDF repositories do not require the helper. Per-file parser diagnostics from a correctly identified helper remain explicit failed coverage.

Both commands emit deterministic field shapes. A sync receipt has this form:

{
  "contract": "coding-ethos.code-intel/v2",
  "kind": "coding-ethos.code-intel.sync/v2",
  "generated_at_utc": "...",
  "repair": "coding-ethos-code-intel sync --root .",
  "source_readiness": {
    "identity": {
      "repository_id": "repository:sha256:...",
      "source_snapshot_id": "snapshot:sha256:...",
      "generation_id": "generation:sha256:...",
      "head_oid": "...",
      "worktree_id": "worktree:sha256:...",
      "worktree_fingerprint": "worktree-content:sha256:...",
      "config_hash": "config:sha256:...",
      "extractor_set_hash": "extractors:sha256:..."
    },
    "coverage": [],
    "status": "exact"
  },
  "vector_readiness": {
    "status": "not_evaluated",
    "ready_records": 0,
    "missing_vectors": 0
  },
  "storage": {
    "shared_root": "...",
    "lane_root": "...",
    "base_manifest_id": "base:sha256:...",
    "delta_manifest_id": "delta:sha256:..."
  },
  "sync": {
    "files_indexed": 0,
    "files_failed": 0,
    "fragments_reused": 0,
    "fragments_written": 0
  }
}

The status receipt uses kind coding-ethos.code-intel.status/v2 and omits the sync summary. Hook TOON and MCP response metadata expose the same receipt under the additive key coding-ethos.code-intel/v2. The Go query boundary is QuerySourceIndex; it refuses missing or stale generations and returns the generation identity with path-bound symbol, import, and PurRDF facts. Built-in facts carry EXTRACTED provenance, the exact parser fingerprint, span fidelity, and deterministic confidence. PurRDF facts preserve the helper’s parser revision and explicit subject_start or none span fidelity.

Shared objects live under <git-common-dir>/coding-ethos/code-intel/v2/{fragments,bases} and are created immutably. Lane deltas, tombstones, and current.json live under the resolved state root plus a worktree-specific lane ID. A v2 sync does not delete or rewrite a v1 DuckDB. rebuild-derived refreshes the derived DuckDB explicitly; rebuild-index is retained only as a deprecated alias during migration.

The PurRDF helper protocol is coding-ethos.external-extractor.v1. The helper receives UTF-8 file content through one stdin JSON batch and returns facts on stdout; it does not read the worktree, use a network, or run as a daemon. A valid batch may contain per-file errors, which remain visible as failed language coverage. A missing or mismatched helper aborts sync before a current generation receipt is published.

Warm-cache effectiveness is observable through cache_hits, fragments_reused, and fragments_written. Run the focused benchmark with go test ./internal/codeintel -run '^$' -bench SourceV2 when comparing cold and warm generation costs; keep repository size and extractor fingerprints in the benchmark report.

agent-hooks capabilities advertises this optional surface as:

{
  "code_intel": {
    "contract": "coding-ethos.code-intel/v2",
    "sync_command": "coding-ethos-code-intel sync --root <worktree>",
    "status_command": "coding-ethos-code-intel status --root <worktree>",
    "source_readiness_statuses": ["exact", "partial", "failed", "missing", "stale"],
    "vector_readiness_statuses": ["not_evaluated", "partial", "ready"]
  }
}

Canonical DuckDB Store

Code-intel commands use CODE_ETHOS_STATE_ROOT for their default DuckDB and vector paths while continuing to inspect source from --root. Explicit --db, --duckdb, and --uri paths take precedence. This lets an orchestrator keep mutable intelligence state outside a consumer worktree without changing the repository identity recorded in facts.

Use code-intel migrate-store --root REPOSITORY --source LEGACY --destination PRIVATE to merge a repository-local legacy store into private state. migrate-store opens the source read-only, validates schema and repository identity, rejects unequal rows with the same primary key, and writes a JSON audit manifest plus a detached SHA-256 digest. The source is retained and its before/after file hashes must match. When --destination is omitted, the command resolves it through CODE_ETHOS_STATE_ROOT; when --source is omitted, the repository-local store is used.

Token Economy Evidence

code-intel token-economy measures context reduction without treating an enabled-only trace as a causal savings result. Its evidence lives in the separate private store .coding-ethos/token-economy.duckdb, so benchmark records do not change the code-intel migration inventory.

Inspect a provider-native ledger without retaining prompt or response bodies:

bin/coding-ethos-run code-intel token-economy ledger \
  --provider codex --path /absolute/path/to/rollout.jsonl

Codex ledgers use the final cumulative provider total and treat cached input as a subset. Claude ledgers deduplicate message IDs, exclude synthetic messages, and sum input, cache-creation, cache-read, and output tokens. A ledger that changes while being read, contains conflicting identities, or has decreasing cumulative usage fails closed.

Write an observational historical report from existing proxy transforms:

bin/coding-ethos-run code-intel token-economy report \
  --historical \
  --db /private/lane-1/code-intel.duckdb \
  --db /private/lane-2/code-intel.duckdb \
  --from 2026-08-01T00:00:00Z \
  --to 2026-09-01T00:00:00Z \
  --output-prefix /private/reports/token-economy-history

Historical inputs are explicit and --db is repeatable. The UTC interval is half-open: --from is inclusive and --to is exclusive. Sources are canonicalized and ordered before aggregation. The command rejects repeated source files and event IDs found in more than one source, hashes every source before and after the read, and records the ordered paths and both hashes in the report.

Write a controlled report after benchmark runs have been recorded:

bin/coding-ethos-run code-intel token-economy report \
  --experiment-id EXPERIMENT \
  --output-prefix /private/reports/token-economy-EXPERIMENT

Both report modes create JSON, Markdown, and a detached SHA-256 receipt with mode 0600; existing paths are never overwritten. Historical reports always set causal=false. Controlled reports require randomized and isolated full, static, and off arms, complete provider ledgers and validation receipts, at least ten tasks, a savings interval excluding zero, and acceptance-rate noninferiority before returning causal_savings. Controlled intervals spend alpha across predeclared complete-block checkpoints to preserve 95% family-wise confidence during adaptive collection.

The complete frozen-manifest, three-arm runner, adaptive collection, and failure-accounting protocol is documented in TOKEN_ECONOMY.md.

The first implementation should create .coding-ethos/code-intel.duckdb with tables for:

Use the DuckDB term index for text, symbol, file path, policy, skill, command, and advice search. Text search is not a fallback; it is part of hybrid retrieval.

Initial command surface:

bin/coding-ethos-run code-intel ingest-traces
bin/coding-ethos-run code-intel stats
bin/coding-ethos-run code-intel hook-usage --risk-category bypass
bin/coding-ethos-run code-intel record-hook-review --trace-id hook-1 --disposition false_positive
bin/coding-ethos-run code-intel hook-reviews --disposition false_positive
bin/coding-ethos-run code-intel repeated-failures --policy-id python.unused_imports
bin/coding-ethos-run code-intel index-code pkg scripts config.yml
bin/coding-ethos-run code-intel git-signals --path pkg/app.go --paths pkg/app.go,pkg/store.go
bin/coding-ethos-run code-intel anatomy-map --path pkg --format toon
ls pkg | bin/coding-ethos-run code-intel enrich-listing --command 'ls pkg'
bin/coding-ethos-run code-intel code-chunks --path pkg/app.go --symbol-name BuildMessage
bin/coding-ethos-run code-intel code-context --path pkg/app.go --symbol-path BuildMessage
bin/coding-ethos-run code-intel repo-map --path pkg/app.go
bin/coding-ethos-run code-intel centrality --path pkg --format toon
bin/coding-ethos-run code-intel surprises --path pkg --format toon
bin/coding-ethos-run code-intel downstream-analysis --root .
bin/coding-ethos-run code-intel decisions add --title 'Use explicit startup' --rationale 'Startup should be inspectable.' --path pkg/app.go
bin/coding-ethos-run code-intel decisions import docs/decisions
bin/coding-ethos-run code-intel decisions list --path pkg/app.go --query startup
bin/coding-ethos-run code-intel decisions health --path pkg/app.go
bin/coding-ethos-run code-intel compact-context --path pkg/app.go
bin/coding-ethos-run code-intel ingest-sarif --file policy.sarif
bin/coding-ethos-run code-intel sarif-results --policy-id python.unused_imports
bin/coding-ethos-run code-intel proxy-file-read --session-id sess-1 --path pkg/app.go
bin/coding-ethos-run code-intel record-proxy-event --event-id evt-1 --session-id sess-1 --kind file_read --provider codex --target-path pkg/app.go
bin/coding-ethos-run code-intel proxy-sessions --provider codex
bin/coding-ethos-run code-intel proxy-events --session-id sess-1
bin/coding-ethos-run code-intel session-snapshot --session-id sess-1 --format toon
bin/coding-ethos-run code-intel context-advice --format toon
bin/coding-ethos-run code-intel token-economy report --historical --db /private/code-intel.duckdb --from 2026-08-01T00:00:00Z --to 2026-09-01T00:00:00Z --output-prefix /private/reports/history
bin/coding-ethos-run code-intel remediation-outcomes --outcome repeated
bin/coding-ethos-run code-intel remediation-effectiveness --policy-id python.unused_imports
bin/coding-ethos-run code-intel skill-health --format toon
bin/coding-ethos-run code-intel repeated-edits --path pkg/app.go
bin/coding-ethos-run code-intel embedding-candidates --record-kind remediation_outcome
bin/coding-ethos-run code-intel record-embedding --backend duckdb-vss --collection remediations --model-id voyage-code-3 --record-kind remediation --record-id rem-1 --dimension 3
bin/coding-ethos-run code-intel record-outcome --remediation-id rem-1 --outcome fixed
bin/coding-ethos-run code-intel upsert-vector --id rem-1 --model-id voyage-code-3 --vector '0.1,0.2,0.3'
bin/coding-ethos-run code-intel embedding-records --backend duckdb-vss
bin/coding-ethos-run code-intel hybrid-search --text 'unused import' --model-id voyage-code-3 --vector '0.1,0.2,0.3'
bin/coding-ethos-run code-intel index-status --model-id voyage-code-3
bin/coding-ethos-run code-intel vector-stats
bin/coding-ethos-run code-intel search --text 'unused import'
bin/coding-ethos-run code-intel sync --root .
bin/coding-ethos-run code-intel status --root .
bin/coding-ethos-run code-intel rebuild-derived --root .
# Deprecated compatibility alias:
bin/coding-ethos-run code-intel rebuild-index --root .
bin/coding-ethos-run code-intel workspace status --root ..

Legacy analytical commands read retained .coding-ethos traces and write the repo-local .coding-ethos/code-intel.duckdb store. The v2 sync and status commands use the immutable shared-base and lane-delta layout described above.

Workspace commands are the exception to repo-local state because they model a parent directory or worktree family. They store only registry and derived workspace facts under .coding-ethos-workspace/ at the workspace root. Each registered repository keeps its own .coding-ethos/code-intel.duckdb store; workspace queries federate across those stores and annotate results with repo aliases instead of merging databases. scan, add, remove, list, status, and refresh manage the registry, stale HEAD/store metadata, conservative git-history co-change candidates, and contract links derived from existing AST graph edges.

git-signals records the HEAD commit and indexed timestamp before exposing stored signals. A refresh is skipped when the current HEAD is already indexed; --force rebuilds the bounded default window, and --commits raises or lowers that window for large repositories. Reviewer suggestions are deterministic: direct authorship percentage, co-change ownership, and recency relative to the indexed history are reported as score inputs. Co-change rows also mark hidden coupling when no direct static edge explains the pair.

anatomy-map is inspired by Aider’s repo map, which gives agents a compact symbol-level view before they spend context on full file reads. coding-ethos keeps the same core idea but uses the repo-local Go AST/code-intel store rather than Aider’s prompt-time Python parser/cache and global PageRank ranking. The first implementation is intentionally directory-local so proxy listing enrichment can append concise file anatomy without replacing the original tool output. The directory-anatomy-map proxy transform preserves the raw directory listing and appends a compact TOON block with in-memory transform hashes and token counts returned by the shared agent proxy pipeline. The live Bash PostToolUse path uses the shared shell parser plus the conservative ls/tree classifier, refreshes the listed source files, and emits the enriched listing as context with proxy.directory_anatomy evidence. ls stays direct-child only, while tree includes nested files and honors tree -L N as an anatomy depth cap. enrich-listing accepts raw listing output from stdin or --listing-file and can infer the target directory and recursive depth from a static, single-target ls or tree command for replay and debugging.

repo-map and the startup coding_ethos_repo_map hook context provide the global variant. The hook refreshes the repo-local AST index at SessionStart, ranks indexed files by compact symbol/chunk signals, and emits the most useful file/symbol signatures as TOON. The same renderer backs MCP code_intel_repo_map and the read-only coding-ethos://code-intel/repo-map resource so agents can request the current map explicitly before broad exploration.

MCP workspace support adds code_intel_workspace_status plus an optional repo argument on code_intel_search, code_intel_answer, and code_intel_repo_map. Omitting repo preserves repo-local behavior. repo="<alias>" opens that registered repo’s independent code-intel store and reports provenance in the result metadata. repo="all" is bounded to read-only aggregate queries and returns per-repo provenance rather than a synthetic merged repository.

graph-report composes the repo map, code-intel store counts, and the latest stored health snapshot into a human, JSON, or TOON orientation report. It is read-only with respect to indexing and tells agents when the AST index or health snapshot needs an explicit refresh.

graph-report also includes deterministic topology communities derived from existing file-level code_edges and git_cochanges. The v1 implementation uses weighted connected components with stable ordering rather than Leiden or another external graph dependency. Community IDs are advisory orientation hints exposed in graph-report, repo-map, and context-card output; enforcement decisions must not depend on community detection.

centrality and surprises expose deterministic graph-orientation views over the same DuckDB facts. Central nodes rank files by explainable structural degree, git co-change, health-priority, finding, and remediation-outcome signals. Surprise edges highlight deterministic cross-boundary relationships such as cross-directory, cross-language, policy/config-to-code, documentation-to-code, test-to-production, and hidden git co-change links. These views are included in graph-report and available as standalone code-intel centrality and code-intel surprises commands. They carry provenance and explanation fields and remain advisory: they help agents choose what to inspect before editing, but they do not block or permit policy decisions.

Markdown documentation is also indexed into the same graph when it contains explicit repo path references. Headings and code blocks remain ordinary Markdown chunks; path references such as go/internal/codeintel/store.go create documents edges, and path#symbol references create mentions edges. Repo-root paths stay repo-relative; Markdown ./ and ../ references resolve relative to the referencing Markdown file’s directory before being stored. Rationale-like headings such as “Rationale”, “Decision”, “Reasoning”, or “Why” classify explicit references as advisory rationale_for links. Documentation links use DOC_DERIVED provenance for extracted references and INFERRED provenance for rationale classification, and graph-report exposes them in a document_links section. This is deterministic Markdown-only graph context: it does not parse PDFs or images, does not call an LLM, and does not turn examples or prose into policy decisions.

Decision intelligence stores explicit architectural rationale in the same DuckDB index. The decisions CLI group records manual decisions, links them to paths or symbols, indexes inline WHY:, DECISION:, and TRADEOFF: markers found during code indexing, and reports stale, conflicting, overlapping, or ungoverned decision areas through decisions health.

decisions import reads explicit Markdown decision records from a supplied file or directory. Full-repo code indexing also imports the default decision locations: adr/, docs/adr/, docs/decisions/, and .coding-ethos/decisions/. Imported Markdown must opt in with YAML front matter, either coding_ethos_decision: true or a decision / architecture-decision tag. Generic headings such as “Decision”, “Rationale”, or README examples are not imported. Front matter may include title, status, rationale, alternatives, author, recorded_at_utc, updated_at_utc, affected_paths, affected_files, affected_modules, and affected_symbols entries with path plus symbol_path.

proxy-file-read is the current bridge for read deduplication. It reads a repo-relative file, computes the current content hash, records the first read as a file_read proxy event, and records later unchanged reads in the same session as cache_hit events with a file-read-cache transform. Changed content is a cache miss and returns the full file body again. The command gives transparent proxy work a tested cache primitive without requiring provider/API interception to exist first.

Implemented Storage Foundation

The local store is the durable evidence ledger for CEL, SARIF, AST facts, hook analytics, remediation advice, vector metadata, and remediation outcomes. The target is a complete storage foundation, not a minimal placeholder.

DuckDB remains canonical and should gain:

The first query/CLI surface should answer:

Vector work uses always-built duckdb-vss tables. Metadata stays in normal DuckDB tables for auditability and filtering, while vectors live in dimension-specific vec0 virtual tables. This preserves the ETHOS one-path build contract and keeps vector indexes rebuildable from DuckDB facts.

Vector Backends

Define a narrow vector backend interface before binding to any implementation:

The initial backend priority:

  1. duckdb-vss search for AST-aware code and remediation vectors.
  2. Future service backends only if team-scale deployment needs them and they can be kept outside local enforcement.

Backend records must include enough metadata to filter before or during vector search:

AST Chunking

Use Tree-sitter to produce semantic chunks instead of line windows:

Every chunk should have:

Incremental indexing is required. A changed file should only re-embed changed chunks and dependent summary chunks when their content or structural hash changes.

The active implementation indexes Go, Python, JavaScript/TypeScript, shell, and YAML through Tree-sitter. Each indexed file is written to the canonical DuckDB store with a content hash, line count, parser metadata, and index timestamp. Each extracted symbol or configuration entry is written as a stable code_chunk, mirrored into the term index, and exposed as an embedding candidate with record_kind=code_chunk. Parent chunk IDs, containment edges, import edges, and same-file reference edges are stored in DuckDB. SARIF/CEL results that carry AST identity are linked back to matching chunks through ast_finding_links. Markdown remains planned until the selected parser exposes a maintained Go binding or the project adds a first-class adapter for its split parser layout.

Code indexing always ignores repository metadata, runtime cache directories, and nested coding-ethos/ tool checkouts. Consumer-specific generated output belongs in repo-local configuration rather than global code. For example, a static-site repo can exclude generated output in repo_config.yaml:

code_intel:
  exclude_paths:
    - "**/dist/**"

Embedding Strategy

Embedding providers must be pluggable and recorded per vector row.

Recommended initial model classes:

Queries should use the matching query/document input mode when the provider supports it. The system must refuse to compare vectors across incompatible model IDs or dimensions.

Hybrid Retrieval

Search should combine exact filters, the term index, vectors, and reranking:

  1. Apply hard filters: repo, path prefix, language, policy, skill, symbol kind, provider, or time range.
  2. Retrieve candidates from the DuckDB term index.
  3. Retrieve candidates from the vector backend.
  4. Fuse ranks with reciprocal rank fusion.
  5. Boost exact symbol/path/policy matches.
  6. Return traceable results with file spans and policy/remediation context.

Vector-only retrieval should be available for diagnostics, but not the default agent path.

MCP Tool Surface

Add tools only after the store has a stable schema:

All tools are advisory. They must not bypass hooks or edit files.

Phases

Phase 1 - Schema and Trace Ingestion

Acceptance criteria:

Phase 2 - AST Indexing

Acceptance criteria:

Phase 3 - duckdb-vss Vector Backend

Acceptance criteria:

Phase 4 - Hybrid Retrieval Hardening

Acceptance criteria:

Phase 5 - MCP Search Tools

Acceptance criteria:

Phase 6 - Outcome Measurement

Acceptance criteria:

Code Similarity Detection

The code-intelligence store supports real-time structural clone detection through MinHash LSH (Locality-Sensitive Hashing). This enables CEL policies to warn agents about duplicate or near-duplicate code at write time.

Storage Schema

Three fields in code_chunks support similarity:

A separate lsh_bands table stores band hashes for sub-linear candidate retrieval:

Column Purpose
chunk_id FK to code_chunks.chunk_id
band_index Band number (0–15)
band_hash Hash of the 8-row band slice
path File path (for filtering self-matches)
symbol_name Symbol name (for diagnostics)

Detection Pipeline

  1. AST indexing normalizes each chunk’s tokens and computes normalized_hash and minhash_sig.
  2. LSH bands are computed (16 bands × 8 rows per band) and stored in lsh_bands.
  3. At policy evaluation time, similarity_facts queries:
    • Exact matches via normalized_hash equality (100% similarity).
    • LSH candidates via band hash collisions, refined with full Jaccard estimation. The default config uses ≥0.7 for candidates and ≥0.8 for policy activation.
  4. Results are exposed to CEL as similarity_facts and reported through SARIF relatedLocations.
  5. Agents can call MCP code_similarity_check with proposed code to inspect matches before writing a duplicate implementation.

Reconciliation

LSH bands are transactionally reconciled with code chunks. When a file is reindexed, bands for the old path are deleted before new chunks and bands are inserted within the same transaction. This prevents stale band entries from producing false-positive candidates.

Configuration

The default similarity config uses 128 hash functions, 16 bands, and 8 rows per band. This produces a candidate threshold of approximately 0.54 Jaccard similarity (the S-curve inflection point), which means most pairs above ~54% similarity will share at least one band hash. The runtime candidate threshold defaults to 0.7, and the structural threshold defaults to 0.8.

similarity:
  enabled: true
  minhash_size: 128
  shingle_size: 5
  lsh_bands: 16
  lsh_rows: 8
  min_symbol_lines: 5
  exact_normalized: true
  candidate_threshold: 0.7
  structural_threshold: 0.8
  max_matches: 10

Risks and Mitigations

Open Questions