Run tests with coverage and gate on a minimum percentage

codecharter coverage [root] [options]

codecharter coverage finds the test projects under root (the current directory by default), runs them with coverage collection, merges the results, and fails the build when line coverage falls below the required minimum. When root contains exactly one .sln/.slnx file, it is restored and built once up front, and the test projects run against that build instead of each building the same shared projects again. The same run also reports every uncovered region with its source snippet, so you know exactly which lines are missing a test.

It is the second half of the quality gate: analyze answers "is the code well written", coverage answers "is it actually tested".

Prerequisite

Coverage is collected through the XPlat Code Coverage collector, which needs the coverlet.collector package in every test project:

<PackageReference Include="coverlet.collector" Version="6.*" PrivateAssets="all" />

A test project without it still runs its tests but produces no coverage data. The run names each affected project and prints the exact line to add.

Quick start

# Gate the repository at the configured minimum
codecharter coverage

# Gate one folder at 90%
codecharter coverage src --min-coverage 90

# Reuse the results of a test run you already did
codecharter coverage --skip-tests

# Write the report to a file instead of stdout
codecharter coverage --output-file coverage-report.json

While the run works it writes progress to stderr, so a long run is never silent: the number of discovered test projects, the shared build with its duration, and every test project as it starts and as it finishes with its duration and outcome. The JSON report goes to stdout, diagnostics and a one-line summary go to stderr. That split keeps codecharter coverage > report.json usable in a pipeline without stripping anything out.

Options

Option	Default	Description
[root]	current directory	Directory tree searched for test projects.
--min-coverage <percent>	from the configuration	Minimum required line coverage (0-100). Overrides the configured value for this run. The number is read locale-independently, so 99.5 is correct everywhere and 99,5 is not.
--git-ref <range>	off	Gate the lines this git range changed instead of the whole repository. See Diff coverage.
--affected-by <range>	off	Run only the test projects affected by this git range instead of every discovered test project. Requires --git-ref naming the exact same range. See Affected-by: narrowing the test suite.
--min-diff-coverage <percent>	the effective --min-coverage value	Minimum required coverage of the changed lines (0-100). Requires --git-ref. Same locale-independent parsing as --min-coverage.
--results-root <dir>	<root>/TestResults/coverage	Where test and coverage artifacts are written. A relative path is resolved against the current working directory, not against root.
--skip-tests	off	Do not run the tests; analyse the coverage files already present under the results root.
--max-parallel <n>	processor count, capped at 4	How many test projects run at once after the shared build (1 or more). Overrides the configured value for this run. Use 1 to serialise the runs, which tells a genuine failure apart from two projects contending on the same port, pipe or file.
--output-file <path>	stdout	Write the JSON report to this file.
--test-timeout <seconds>	1800 (30 minutes)	How long a single test project may run. On expiry the test process and everything it started are killed, the project is reported as timed out with that reason, and the run continues with the remaining projects. Bounds one project, not the whole run.
--findings	off	Also report every uncovered region as a finding, rendered through the same formatters analyze uses.
--findings-format <name>	console	Format for --findings: console, json, sarif, or github-annotations.
--findings-file <path>	stderr	Write the --findings output to this file.
--verbose	off	Also print informational progress lines to stderr.
--no-color	off	Disable ANSI color codes.

--test-timeout is a command-line option only, not a configuration key: the right value follows the machine and the CI job's own limit, not the repository. A timed-out project is never retried — a hang repeats, and a second attempt would only spend the same wall-clock time again.

While a run works it owns its results root. A second run against the same results root waits for the first one, and says so on stderr, because both would otherwise purge and rewrite the same per-project directories and each end up analysing a mixture of the two. The wait is bounded at ten minutes, after which the run stops with exit code 64 instead of producing a number nobody can trust. A lock left behind by a run that was killed is detected and removed automatically, so a cancelled CI job never blocks the next one.

--skip-tests checks that a coverage file is present for every discovered test project before it trusts the result, so a stale or half-finished run cannot pass the gate unnoticed. If the results directory was produced by a different tool and does not carry the per-project layout, the check falls back to comparing counts and says so.

Configuration

The policy lives in .codecharter/config.yml, so it is committed once and applies to everybody:

coverage:
  minimum-percent: 99.5           # gate threshold (0-100), default 100
  snippet-context-lines: 3        # context lines around each uncovered region
  max-parallel-test-projects: 4   # test projects running at once, default: cores capped at 4
  exclude:                        # files excluded from the metric (globs)
    - "**/*.Designer.cs"
    - "src/Generated/**"
  excluded-directories:           # directory trees skipped during discovery (globs)
    - "vendor/**"

All five keys are repository-wide; unlike the analysis sections, coverage cannot be varied per path scope. Globs are matched case-sensitively against paths relative to the run root, written with forward slashes (**/ spans directories, * stays inside one path segment).

exclude versus excluded-directories

The two lists look alike and do very different things:

• exclude acts at measurement time. The tests still run; the matching source files simply do not count towards the percentage and produce no uncovered regions. Use it for generated code you cannot meaningfully test.
• excluded-directories acts at discovery time. A matching directory tree is never walked, so any test project inside it is never found and never run. Use it for a vendored or third-party tree whose tests are none of your repository's business.

Because discovery prunes the whole tree once the directory itself matches, vendor, vendor/ and vendor/** all mean the same thing. Write **/third-party to match a directory of that name at any depth. Directory names bin, obj, node_modules and .git are always skipped and need no entry.

Note that coverage.exclude is separate from the top-level exclude used for analysis. Excluding a folder from rule analysis does not silently drop it from your coverage number, and the other way round.

The effective threshold is resolved per key in this order:

1. --min-coverage on the command line
2. .codecharter/config.local.yml (your personal, machine-local overlay)
3. .codecharter/config.yml (the committed team setting)
4. the built-in default of 100

The report states which of the four won, so a surprising gate result can always be traced back to its source. max-parallel-test-projects follows the same order with --max-parallel in the first position, but carries no provenance: only the threshold reports where it came from. Note that a coverage.exclude or coverage.excluded-directories list in the local overlay replaces the committed list rather than extending it.

You can edit all of this from the command line instead of by hand — see codecharter config.

What counts towards the number

Positional records that only carry data have no behaviour worth testing, so they are left out of the metric. A record is treated as pure data as long as none of its members has a hand-written body: auto-properties (including init) and the generated primary constructor keep it pure, while a computed property, a validating constructor, a method body, an operator, an indexer, or an event puts the whole record back into the gate.

Classes and structs are never excluded this way. The exception is deliberately limited to records, because only a record states the "data, not logic" contract outright.

Everything else you exclude is your decision, through coverage.exclude.

Diff coverage

--git-ref <range> switches the gate from the whole repository to the lines the range changed:

codecharter coverage --git-ref origin/main..HEAD

The range uses the same syntax as analyze --git-ref and is handed to git diff <range> --unified=0. Every added or modified line on the new side of the diff is intersected with the merged per-line coverage of the run. Diff coverage is then covered measurable changed lines over measurable changed lines. A changed line is measurable when the coverage data has something to say about it: a comment, a blank line, a file without coverage data, and a file coverage.exclude drops all change nothing and count nowhere. Renamed files are tracked under their new path.

This is the gate a pull request wants. A repository at 62% overall does not have to climb to 100% before it can demand that new code arrives tested — every change gates itself, and the overall number rises on its own.

The overall coverage is still measured and reported exactly as before; it simply no longer decides the outcome. The threshold for the diff gate is --min-diff-coverage when you pass it, and otherwise the effective --min-coverage value — whatever the configuration precedence resolved. The report names which of the two applied.

A diff that changed no measurable line at all — a docs-only commit, a pure rename — passes the gate, and says so explicitly: the reported percentage is null, never a fabricated 100%.

There is deliberately no configuration key for the diff threshold yet: a git range only makes sense per invocation, and --git-ref belongs in the pipeline definition, not in a committed file.

In a pull-request pipeline

- uses: actions/checkout@v4
  with:
    fetch-depth: 0   # the base commit must be in the clone
- name: Changed-lines coverage gate
  run: codecharter coverage --git-ref origin/main..HEAD --min-diff-coverage 90

fetch-depth: 0 is the one thing to get right. A CI checkout is shallow by default, so the base commit is missing and the range cannot be resolved. That is a usage error (exit code 64), never a silent pass, and the message says so.

Affected-by: narrowing the test suite

--git-ref narrows which lines the gate cares about; --affected-by narrows which test projects run at all:

codecharter coverage --git-ref origin/main..HEAD --affected-by origin/main..HEAD

--affected-by must name the exact same range as --git-ref — the two options describe one change, seen from two angles, and a mismatch between them is rejected before anything runs. Given the range, the command walks the reverse project-reference graph from every project a changed file belongs to and runs only the test projects that graph reaches, instead of every test project discovery finds. On a large solution where a pull request typically touches a handful of projects, this turns a coverage run that would otherwise build and test everything into one that runs only what the change could possibly affect.

A changed file that does not resolve to a project at all, or that resolves to a file the whole solution depends on — Directory.Build.props, a .sln, .codecharter/config.yml, anything under build/, .github/workflows/, or tools/ — widens the run back to every discovered test project automatically. A change that could affect anything is treated as affecting everything, rather than silently under-testing it.

Three usage errors, all exit code 64:

• --affected-by without --git-ref.
• --affected-by naming a different range than --git-ref.
• --affected-by together with --skip-tests — the two options answer the same question ("which projects count") from incompatible sources: a coverage file already on disk, or a freshly computed project closure.

The gate stays fail-closed even with a narrower suite. After the run, every file the range changed is checked against what was actually measured. If a changed file has no coverage data at all — because its owning project's tests were not selected, or the file is genuinely untested — the run fails with exit code 3 and names the file, exactly as if nothing had been measured. --affected-by can only make a run faster, never make a gap in coverage look like a pass.

When the cross-check does fail, the message on stderr names the file directly: "--affected-by completeness check failed: changed file '...' has no coverage data at all. Its owning project's tests were not selected, or the file is genuinely untested; either way this suite selection cannot gate it. Run without --affected-by, or widen the selected suites." — followed by the same exit code 3 a run with no coverage data at all would produce.

If the results root was produced by a plain dotnet test --results-directory rather than by this command (a "flat layout", the same case --skip-tests falls back to a file-count comparison for), the completeness check for --affected-by needs a per-project layout: in a flat layout a selected project's coverage looks the same as a stale file a skipped project left behind from an earlier run. Rather than guess, the check is skipped with a warning on stderr, and the gate falls back to trusting the suite selection.

Exit codes

Code	Meaning
0	Coverage met the required minimum.
1	Coverage below the required minimum.
2	Tests failed, or the coverage data was incomplete.
3	No coverage data at all.
64	Usage, configuration, or environment error (for example a missing .NET SDK or an unwritable --output-file).

With --affected-by, a mismatched or incompatible combination of options (see Affected-by) is a usage error (64), and the completeness cross-check failing after the run is reported as 3, same as any other run with nothing measured for a file that matters.

With --git-ref the codes keep their meaning and apply to the diff gate: 1 means the changed lines missed their minimum, and 64 covers an unresolvable git range alongside the other usage errors.

The gate is fail-closed: coverage data that is missing, empty, or unreadable never produces a pass. Exit code 3 says "nothing was measured", it never appears as a green 100%.

The displayed percentage is rounded down, so 100.00% means every measurable line is covered and nothing else. A single uncovered line in a large solution shows as 99.99% and fails a 100% gate, instead of rounding up into a false pass.

The report

The JSON report has four parts:

• summary — measurable lines, covered lines, the percentage, the required minimum, where that minimum came from, whether the gate passed, and the age of the coverage files that were analysed.

• testResults — one entry per test project with its exit code and how many tests it ran: total, passed, failed, and skipped. The four counts are null when the project produced no test results to read, for example after a test host that died mid-run, so "not measured" never looks like a run of zero tests. The whole list is empty when you pass --skip-tests.

• uncoveredRegions — every contiguous block of never-executed lines, with the file (as an absolute path and relative to the run root), the containing method, the line numbers, and a source snippet. A region also ends at a method boundary, so a block that runs from the end of one method into the next is reported as two regions, each attributed to the method it really sits in.

• diffCoverage — null unless the run was scoped with --git-ref. Otherwise the git range, measurableChangedLines, coveredChangedLines, the percent (null when nothing measurable changed), the requiredPercent with its thresholdProvenance (--min-diff-coverage or inherited), whether the gate was met, and uncoveredChangedRegions — the uncovered regions restricted to the changed lines, in the same shape as uncoveredRegions.

The relative path is the stable key for CI annotations: it does not depend on where the build agent checked the repository out.

Uncovered regions as findings

With --findings the same uncovered regions are additionally reported as findings — the shape analyze produces, run through the very same formatters. Each finding carries the rule id coverage/uncovered-region, a message naming the method and the line span, and the repo-relative file path plus the region's first line as its anchor. Severity follows the gate: informational while the run still meets its threshold, an error once it misses it. With --git-ref the severity follows the diff gate, and only a region that actually contains a changed line can carry the error — every other region stays informational, so one small pull request never lights up an entire legacy codebase in red.

# Annotate the uncovered lines on a GitHub pull request
codecharter coverage --output-file coverage-report.json \
  --findings --findings-format github-annotations

The switch is purely additive. The JSON report stays the default output on stdout and does not change, and the exit code is exactly the one the run would have produced without it. The findings go to stderr unless you name a target with --findings-file, so stdout keeps carrying the report. Coverage findings never appear in codecharter analyze output; the two gates stay separate.

In CI

- name: Coverage gate
  run: codecharter coverage --output-file coverage-report.json
  # Exit code 1 turns the step red when coverage drops below the minimum.

Because the threshold lives in the configuration file, the pipeline definition does not need to change when the team raises the bar.

With an AI assistant

The same gate is available to AI coding assistants as the run_coverage tool of the MCP server. The assistant gets the uncovered regions including file, method, line numbers, and snippet, which is everything it needs to write the missing tests without searching for them first. The tool result additionally carries a findings array with the same regions in the shape the analyze_* tools return, so an assistant can treat a coverage gap exactly like a rule violation. The tool takes an optional git_ref parameter with the same meaning as --git-ref, and then returns the diffCoverage section alongside the report, so an assistant can work through exactly the lines its own change left untested. It also takes an affected_by parameter with the same meaning and rules as --affected-by, and, unlike the CLI's stdout report, its result carries a coverageScope object naming exactly which suites were selected and skipped and, on a failed completeness check, which changed file caused it.

Related

• codecharter analyze — the code-quality half of the gate.
• Configuration file — the coverage section in context.
• Exit codes — the exit codes of the other commands.