codec-rd-eval

Files (7)

• SKILL.md
• aws_batch_submit.py · scripts/
• bd_rate.py · scripts/
• build_encoder.sh · scripts/
• run_eval.py · scripts/
• rd-eval-report.md · templates/
• test-config.hjson · templates/

SKILL.md (first lines)

<Purpose>
Build and evaluate ref_model C encoder binaries across multiple encoding configurations,
computing BD-PSNR and BD-rate metrics (VCEG-M33 methodology) to quantitatively compare
codec algorithm candidates.

This skill automates the full Rate-Distortion evaluation pipeline:
1. Build encoder binaries from refc/*.c (C11, gcc)
2. Run parallel encoding simulations across (sequence, QP, config) combinations
3. Compute BD-PSNR/BD-rate using VCEG-M33 polynomial interpolation (3+ points)
4. Generate comparison reports with per-sequence and aggregate metrics

**Scope: Encoder RD evaluation only.**
This skill evaluates encoder quality metrics (BD-PSNR, BD-rate, optional SSIM/VMAF).
For decoder conformance testing against JVET/JCTVC bitstreams, use `/rtl-agent-team:codec-conformance-eval`.

**Phase-agnostic**: While commonly used during rat-dse Step 3b, this skill can be invoked
at any Phase where quantitative RD comparison of encoder configurations is needed —
Phase 1 (algorithm exploration), Phase 2 (architecture validation), Phase 4 (fixed-point
precision impact), or standalone evaluation outside the pipeline.

**Execution modes**:
- **local**: ProcessPoolExecutor-based parallel encoding on local CPU cores
- **aws-batch**: Optional AWS Batch spot instance submission for large-scale evaluation

**Key features**:
- **N-candidate comparison**: Compare 2+ configurations via candidates[] array (anchor + N tests)
- **Configurable encoder CLI**: encoder_cmd_template supports any encoder (HM, VTM, custom). Available variables: `{encoder}`, `{cfg}`, `{input}`, `{width}`, `{height}`, `{fps}`, `{frames}`, `{qp}`, `{bitstream}`, `{recon}`, `{bit_depth}`, `{chroma_format}`
- **Configurable output parsing**: Custom regex patterns for bitrate/PSNR extraction
- **SSIM/VMAF opt-in**: Additional quality metrics on explicit user request only
- **bit_depth/chroma_format aware**: YUV weighting adjusts per chroma format (420/422/444)
- **3+ QP point support**: Standard 4-point (exact fit), 5+ (least-squares), 3 (quadratic fallback)
</Purpose>

<Use_When>
- Comparing algorithm candidates with objective quality metrics (any Phase)
- Measuring BD-PSNR/BD-rate between anchor and modified encoder configurations
- Evaluating fixed-point precision impact on codec quality (e.g., 12-bit vs 16-bit paths)
- Validating that HW-friendly algorithm modifications preserve acceptable quality
- N-way comparison of multiple encoder configurations (candidates[] mode)
- The user explicitly says "RD eval", "BD-PSNR", "BD-rate", "codec quality", "algorithm quality evaluation"
</Use_When>

<Do_Not_Use_When>
- No ref C model encoder exists yet (build ref model first via ref-model skill)
- Comparing RTL vs C model output (use `/rtl-agent-team:rtl-model-consistency` instead)
- Running decoder conformance tests against standard bitstreams (use `/rtl-agent-team:codec-conformance-eval`)
- Running RTL-level conformance against reference decoder (use `/rtl-agent-team:rtl-conformance-test`)
- Non-codec designs where RD metrics don't apply
</Do_Not_Use_When>

<Why_This_Exists>
In codec design, algorithm selection has the highest impact on final quality and area.
Theoretical complexity analysis (operations/pixel, gate estimates) provides useful guidance
but cannot capture the full picture — actual RD performance on representative sequences
is the definitive metric.

BD-PSNR/BD-rate (VCEG-M33) is the universally accepted method in the video coding community
for comparing codec configurations. It normalizes across different operating points (QP values)
to produce a single, meaningful comparison metric.

Without this skill, teams either skip quantitative RD evaluation (risking suboptimal algorithm
selection) or manually set up evaluation infrastructure (time-consuming and error-prone).
</Why_This_Exists>

<Execution_Policy>
- Requires refc/*.c to exist (or user-specified encoder source path)
- HJSON test configuration defines all evaluation parameters
- Local execution is the default; AWS Batch is opt-in via configuration
- Simulation results are cached at .rat/scratch/rd-eval/ for re-analysis
- Report is generated at the path specified in test configuration (default: docs/phase-1-research/rd-eval-report.md)
- On build failure: report error details and stop (do not proceed with stale binaries)
- On simulation failure: report failed jobs, compute BD metrics from successful jobs with warnings
- On metric parsing failure (bitrate=0 or PSNR=0): mark job as failed with guidance to check output_parsing
- timeout_per_job is in seconds (default: 3600s = 1 hour per encoding job)
- SSIM/VMAF are computed ONLY when explicitly requested via quality_metrics config
- Dependencies: gcc (C11), Python 3.9+, numpy, hjson. Optional: ffmpeg (required for SSIM/VMAF computation when quality_metrics includes "ssim" or "vmaf"), boto3 (required only for aws-batch execution mode)
- Self-test: `python3 {plugin_root}/skills/codec-rd-eval/scripts/bd_rate.py --test` runs built-in unit tests (`{plugin_root}` = plugin root resolved from `.rat/state/spawn-context.json`)
</Execution_Policy>

<Steps>
1. **Prerequisite validation**
   - Verify refc/*.c (or configured encoder_src) exists
   - Verify test configuration file exists (HJSON format)
     - If not provided, generate from template at templates/test-config.hjson
     - Ask user to fill in sequence paths and encoder configurations
   - Check anchor encoder binary existence (skip build if already built)
   - Verify Python dependencies: `python3 -c "import numpy; import hjson"`
     - If missing, report: `python3 -m venv .venv && ".venv/bin/python" -m pip install numpy hjson`

2. **Encoder build** (build_encoder.sh)
   - For each unique encoder_src in configurations (anchor/test or candidates[]):
     `bash {plugin_root}/skills/codec-rd-eval/scripts/build_encoder.sh <src> <binary> [extra_cflags]`
   - Build flags: `gcc -std=c11 -O2 -Wall -Wextra -lm` (C11 standard per CLAUDE.md)
   - On build failure: capture stderr, report to user, STOP

3. **Simulation execution** (run_eval.py)
   - Parse HJSON test configuration
   - Generate job matrix: (sequence x QP x config) combinations
   - Execute in configured mode:
     - **local**: `python3 {plugin_root}/skills/codec-rd-eval/scripts/run_eval.py <config.hjson> --mode local`
     - **aws-batch**: `python3 {plugin_root}/skills/codec-rd-eval/scripts/run_eval.py <config.hjson> --mode aws-batch`
   - Each job produces: bitrate_kbps, psnr_y/u/v/yuv (dB), encode_time_s
   - Optional: SSIM, VMAF (when quality_metrics includes them)
   - Results saved to: `.rat/scratch/rd-eval/results.json`

4. **BD-PSNR/BD-rate calculation** (bd_rate.py)
   - `python3 {plugin_root}/skills/codec-rd-eval/scripts/bd_rate.py .rat/scratch/rd-eval/results.json --output .rat/scratch/rd-eval/bd-metrics.json`
   - VCEG-M33 algorithm with N-point support:
     1. Transform rates to log10 domain
     2. Fit polynomial (degree = min(3, N-1)) — exact interpolation for 4 points, least-squares fitting for 5+
     3. Integrate over common PSNR range
     4. BD-rate (%) and BD-PSNR (dB)
   - N-candidate mode: compute metrics for each test vs anchor
   - Per-sequence results + aggregate average + encoding time summary
   - Output: `.rat/scratch/rd-eval/bd-metrics.json`

5. **Report generation**
   - Generate report from template at templates/rd-eval-report.md
   - Output path: the config's `output.report_path` (default: docs/phase-1-research/rd-eval-report.md).

From babyworm/rtl-agent-team, MIT.

More from babyworm/rtl-agent-team