TS009: TestCoverageTrend

Overview

Property Value
ID TS009
Name TestCoverageTrend
Group tests
Severity NOTE

Description

Compares current test coverage against a stored baseline and flags if coverage has dropped.

Coverage trend tracking is valuable because:

  • It prevents gradual erosion of test coverage over time
  • It catches coverage regressions before they’re merged
  • It encourages maintaining or improving coverage
  • It provides historical context for coverage changes

What it checks

The check compares current coverage to a baseline stored in .pycmdcheck_coverage_baseline:

  • PASSED: Coverage is at or above the baseline (updates baseline if improved)
  • FAILED: Coverage has dropped below the baseline (reports percentage drop)
  • NOT_APPLICABLE: No coverage data found (run pytest with coverage first)

Baseline file format

The baseline file contains a single floating-point number:

85.50

Coverage data sources

The check looks for coverage data in order:

  1. coverage.xml - XML format (preferred, easier to parse)
  2. .coverage - SQLite database format

How to fix

Generate coverage data first

# Generate coverage.xml for the check
pytest --cov=mypackage --cov-report=xml

# Or generate both terminal and XML
pytest --cov=mypackage --cov-report=term --cov-report=xml

Restore coverage to baseline

If coverage dropped, add tests to restore it:

# See what's not covered
pytest --cov=mypackage --cov-report=term-missing

# Or generate HTML report
pytest --cov=mypackage --cov-report=html
# Open htmlcov/index.html in browser

Accept a new baseline

If the coverage drop is intentional (e.g., removing dead code):

# Update the baseline to current coverage
pycmdcheck --only TS009 --update-baseline

Or manually edit .pycmdcheck_coverage_baseline:

# Set new baseline
echo "78.50" > .pycmdcheck_coverage_baseline

Initialize baseline for new projects

The check automatically creates a baseline on first run if none exists.

To explicitly initialize:

# Run tests with coverage
pytest --cov=mypackage --cov-report=xml

# Run check to create baseline
pycmdcheck --only TS009

Why NOTE severity?

This check is a NOTE because:

  • Coverage drops may be intentional during refactoring
  • The baseline can be easily updated when needed
  • It’s a trend indicator, not an absolute quality gate
  • Some coverage changes are acceptable with justification

Configuration

Update baseline

To accept current coverage as the new baseline:

[tool.pycmdcheck.checks.TS009]
update_baseline = true

Or via CLI:

pycmdcheck --only TS009 --check-config 'TS009.update_baseline=true'

Skip this check

[tool.pycmdcheck]
skip = ["TS009"]

CLI

pycmdcheck --skip TS009

Best practices

  1. Commit the baseline: Add .pycmdcheck_coverage_baseline to version control
  2. Run in CI: Catch coverage regressions before merge
  3. Review drops: Require justification for coverage decreases
  4. Ratchet up: Automatically update baseline when coverage improves
  5. Combine with threshold: Use both TS003 (absolute) and TS009 (trend)

CI workflow example

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest pytest-cov pycmdcheck
      - run: pytest --cov=mypackage --cov-report=xml
      - run: pycmdcheck --only TS009

  # Optional: update baseline on main branch
  update-baseline:
    if: github.ref == 'refs/heads/main'
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pytest pytest-cov pycmdcheck
      - run: pytest --cov=mypackage --cov-report=xml
      - run: pycmdcheck --only TS009 --check-config 'TS009.update_baseline=true'
      - uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: "chore: update coverage baseline"
          file_pattern: .pycmdcheck_coverage_baseline

Gitignore considerations

Keep the baseline in version control:

# Coverage artifacts (ignored)
.coverage
coverage.xml
htmlcov/

# Coverage baseline (tracked)
# .pycmdcheck_coverage_baseline  # DO NOT ignore this