Skip to main content
The OSS benchmark harness freezes cohorts and re-runs the same repositories across scanner iterations. It is used to reduce false positives without moving the target after seeing results. The harness lives in the CLI repository under tools/oss-benchmark.mjs.

​
Default cohort

The default cohort uses GitHub Trending repositories from these language tracks:
  • typescript
  • python
  • go
  • rust
  • ruby
  • php
  • java
The default limit is 10 repositories per language.

​
Deterministic scan command

Each benchmark scan runs:
That disables telemetry, disables ANSI output, and keeps output stable for overnight runs.

​
Workflow

Build the CLI first:
Capture a new frozen cohort:
Run the latest frozen cohort:
Capture and run in one command:
Re-run the same cohort after rule fixes:
Run a smaller smoke test:

​
Output layout

Generated benchmark data is ignored by git and written under tools/benchmark-data/.

​
Review loop

  1. Check failures first so the cohort is complete.
  2. Check the lowest-score repositories.
  3. Check the highest-volume rules across many repositories.
  4. Open per-repo scan.json files for likely false positives.
  5. Fix the rules and add regression tests.
  6. Re-run the same manifest as the next iteration.
Only refresh the cohort when you want a new market snapshot. For rule iteration, keep the manifest stable.