codec-conformance-eval

Files (7)

• SKILL.md
• aws_batch_conformance.py · scripts/
• build_decoder.sh · scripts/
• compare_output.py · scripts/
• run_conformance.py · scripts/
• conformance-config.hjson · templates/
• conformance-report.md · templates/

SKILL.md (first lines)

<Purpose>
Build and evaluate ref_model C decoder binaries against official conformance bitstreams,
verifying bitexact output match with golden references to ensure decoder algorithmic correctness.

This skill automates the full decoder conformance evaluation pipeline:
1. Build decoder binary from refc/*.c (C11, gcc)
2. Run parallel decoding of conformance bitstreams (JVET, JCTVC, 3rd party)
3. Compare decoded output against golden references (MD5, bitexact, optional PSNR/SSIM/VMAF)
4. Generate conformance report with profile/level coverage matrix

**Scope: Decoder conformance evaluation at the algorithm/C model level.**
- This skill tests the **C reference model decoder** against official conformance streams
- For **RTL-level** conformance testing, use `/rtl-agent-team:rtl-conformance-test` (Phase 5)
- For **encoder RD evaluation**, use `/rtl-agent-team:codec-rd-eval`

**Phase-agnostic**: Commonly used during Phase 1-2 (Research/Architecture) to validate
decoder algorithm correctness before committing to hardware implementation. Can also be
used at any Phase where decoder conformance verification is needed.

| | rtl-conformance-test | codec-conformance-eval |
|---|---|---|
| Target | RTL implementation (.sv) | C ref model decoder |
| Phase | Phase 5 (Verify) | Phase 1-2 (Research/Architecture) |
| Comparison | JM/HM reference output | Official conformance golden output |
| Purpose | RTL-level bitexact verification | Algorithm-level conformance verification |

**Execution modes**:
- **local**: ProcessPoolExecutor-based parallel decoding on local CPU cores
- **aws-batch**: Optional AWS Batch spot instance submission for large stream sets

**Key features**:
- Profile/level filtering for targeted conformance testing
- Multiple conformance sources: JVET/JCTVC (mandatory per standard), 3rd party (optional)
- Configurable comparison: MD5 checksum, bitexact byte comparison, PSNR threshold
- SSIM/VMAF opt-in (only on explicit user request)
- Profile coverage matrix generation
</Purpose>

<Use_When>
- Validating decoder algorithm correctness against official conformance streams
- Verifying that a C ref model decoder passes JVET/JCTVC conformance requirements
- Testing decoder support for specific profiles/levels before hardware implementation
- Evaluating 3rd-party conformance streams for additional coverage
- The user explicitly says "decoder conformance", "conformance stream", "conformance test", "decoder verify", "bitexact decoder"
</Use_When>

<Do_Not_Use_When>
- No ref C model decoder exists yet (build ref model first via `/rtl-agent-team:ref-model`)
- Testing RTL decoder implementation (use `/rtl-agent-team:rtl-conformance-test`, Phase 5)
- Comparing encoder quality (use `/rtl-agent-team:codec-rd-eval`)
- Comparing RTL vs C model output (use `/rtl-agent-team:rtl-model-consistency`)
</Do_Not_Use_When>

<Why_This_Exists>
Decoder conformance is a hard requirement for any video codec implementation. A decoder
that fails official conformance streams has algorithmic bugs that will propagate to hardware.

Catching conformance failures at the C model level (Phase 1-2) is orders of magnitude
cheaper than discovering them after RTL implementation (Phase 5). Official conformance
streams from JVET and JCTVC exercise specific codec features and corner cases that
typical test sequences may not cover.

This skill automates the tedious process of running hundreds of conformance streams,
comparing outputs, and tracking which profile features are covered.
</Why_This_Exists>

<Execution_Policy>
- Requires refc/*.c with decoder functionality (or configured decoder_src)
- HJSON conformance configuration defines all test parameters
- Local execution is the default; AWS Batch is opt-in via configuration
- **AWS Batch decoder contract**: the job definition runs `/app/decode.sh` from a
  container image you supply, so `decoder.decoder_binary` does not apply — the image
  provides the decoder. `decoder_cmd_template` and `extra_args` are forwarded as the
  `DECODER_CMD_TEMPLATE` and `DECODER_EXTRA_ARGS` environment variables; an AWS run
  only matches a local run if the image reads them. The submitter prints a note
  whenever these settings are set, so the divergence is never silent.
- Profile and level filters (`target.profile`, `target.level`) apply identically in
  local and AWS Batch mode. Level matching accepts `4.1`, `4_1` and the packed `L41`
  filename conventions
- Conformance results are cached at .rat/scratch/conformance-eval/
- Report is generated at configured path (default: docs/phase-1-research/conformance-eval-report.md)
- On build failure: report error details and stop
- On decoding failure: mark stream as FAIL, continue with remaining streams
- Mandatory streams (JVET, JCTVC) must all PASS for overall conformance PASS
- Optional streams (3rd party) failures are reported but do not affect overall verdict
- SSIM/VMAF are computed ONLY when explicitly requested via quality_metrics config
- timeout_per_job is in seconds (default: 300s = 5 min per decoding job)
- Dependencies: gcc (C11), Python 3.9+, hjson, numpy (optional, for PSNR computation). Optional: ffmpeg (required for SSIM/VMAF computation when quality_metrics includes "ssim" or "vmaf"), boto3 (required only for aws-batch execution mode)
- Self-test: `python3 {plugin_root}/skills/codec-conformance-eval/scripts/compare_output.py --test` runs built-in unit tests (`{plugin_root}` = plugin root resolved from `.rat/state/spawn-context.json`)
</Execution_Policy>

<Steps>
1. **Prerequisite validation**
   - Verify refc/*.c (or configured decoder_src) exists with decoder code
   - Verify conformance configuration file exists (HJSON format)
     - If not provided, generate from template at templates/conformance-config.hjson
   - Verify conformance bitstream directories exist
   - Verify golden output references exist (MD5 checksums or reference YUV files)
   - Check Python dependencies: `python3 -c "import hjson; print('OK')"`
     - Optional: `python3 -c "import numpy; print('numpy OK')"` (needed for PSNR comparison mode)

2. **Decoder build** (build_decoder.sh)
   - `bash {plugin_root}/skills/codec-conformance-eval/scripts/build_decoder.sh <decoder_src> <decoder_binary> [extra_cflags...]`
   - Build flags: `gcc -std=c11 -O2 -Wall -Wextra -lm` (C11 standard per CLAUDE.md)
   - Optional extra_cflags: e.g., `-DDECODER_ONLY`, `-DMAX_DPB_SIZE=16`
   - On build failure: capture stderr, report to user, STOP

3. **Conformance test execution** (run_conformance.py)
   - Parse HJSON conformance configuration
   - Auto-discover conformance bitstreams from configured source paths
   - Filter by target profile/level (if specified)
   - Execute in configured mode:
     - **local**: `python3 {plugin_root}/skills/codec-conformance-eval/scripts/run_conformance.py <config.hjson> --mode local`
     - **aws-batch**: `python3 {plugin_root}/skills/codec-conformance-eval/scripts/run_conformance.py <config.hjson> --mode aws-batch`
   - Each job produces: decoded YUV (or MD5) + decode_time + status
   - Results saved to: `.rat/scratch/conformance-eval/results.json`

4. **Output comparison** (compare_output.py)
   - `python3 {plugin_root}/skills/codec-conformance-eval/scripts/compare_output.py <results.json> <config.hjson>`

From babyworm/rtl-agent-team, MIT.

More from babyworm/rtl-agent-team