nvflare-autofl-report

Files (8)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• report-contract.md · references/
• generate_report.py · scripts/
• skill-card.md
• skill.oms.sig
• helper_scripts.md · tests/

SKILL.md (first lines)

# NVFLARE Auto-FL Report

## Purpose

Turn the recorded evidence from a stopped NVFLARE Auto-FL campaign into a reproducible
Markdown report, machine-readable JSON summary, and refreshed progress plot without
changing the campaign or its results.

## Use When

Use this skill after an NVFLARE Auto-FL campaign has stopped, reached an
explicit cap, hit a hard blocker, or was manually interrupted. Use it when the
user asks for the final report, achieved improvement, literature findings,
failed ideas, reproduction details, or a refreshed progress plot.

## Do Not Use When

Do not use this skill to start or continue optimization, invent missing
results, or finalize a campaign that is still running. Use `nvflare-autofl` for
the active candidate loop. Do not stop an active campaign merely because the
user asks for a status snapshot.

## Available Scripts

| Script | Purpose | Arguments |
| --- | --- | --- |
| `scripts/generate_report.py` | Validate stopped campaign evidence and generate the report, JSON summary, and optional progress plot. | Campaign job directory; optional evidence/output paths, interruption confirmation, plot settings, and agent context. Run `python scripts/generate_report.py --help` for the complete CLI. |

Run this bundled CLI directly with Python. This skill has no NVFLARE or agent
`run_script()` helper; do not invent or call one. Resolve the script from the
directory containing this `SKILL.md`, as shown in the workflow.

## Workflow

1. Locate the job directory containing `results.tsv`, `autofl.yaml`,
   `.nvflare/autofl/campaign_state.json`, and candidate manifests.
2. Confirm execution has stopped. Prefer campaign state with
   `final_response_allowed=true`. When a process was abruptly interrupted and
   state is stale, independently confirm no campaign or job process remains,
   then use `--confirm-interrupted`. Before finalizing, confirm that campaign
   state, `results.tsv`, and available candidate manifests contain no pending
   candidate. Finalize or abandon pending work through `nvflare-autofl` first.
3. Generate the deterministic report artifacts:

   ```bash
   python "$REPORTER" <job-dir>
   ```

   Resolve `REPORTER` once to the absolute path of
   `scripts/generate_report.py` relative to this `SKILL.md`. Do not assume a
   provider-specific skill installation directory or `$CODEX_HOME`.

4. Read both **autofl_final_report.md** and `autofl_report_summary.json`. Check
   warnings about metric use, executed budget changes, campaign-state/ledger
   disagreement, missing provenance, and incomplete interruption state.
5. Give the user the baseline, best score, delta, strongest candidate lineage,
   concise "what helped" and "what did not help" findings, literature ideas
   that helped or failed, selection rationale, reliability caveats, and
   absolute artifact paths.

The helper attempts to refresh `progress.png` by reusing the product Auto-FL
plotter. Plotting is optional evidence: if plotting dependencies are missing or
the artifact is not a valid PNG, the helper preserves the artifact, records a
warning and `artifacts.progress_plot_available=false`, and still writes the
Markdown and JSON reports without embedding the broken image. It does not
modify source, candidate manifests, `results.tsv`, or campaign state.
All relative helper path options, including `--plotter`, resolve from the
campaign job directory rather than the shell's current working directory.
The helper holds the campaign lifecycle lock from evidence loading through
artifact writes. It refuses a busy campaign and rejects writable output paths
that alias campaign evidence, `job.py`, trust-contract source paths, or another
output, including filesystem aliases. Outputs must not match the trust
contract's allowed source-creation patterns.
A read-only campaign archive is reportable only when its persisted
`campaign.lock` already exists and writable output paths are supplied; the
persisted lock target alone does not imply live contention.

## Interrupted Campaigns

The report helper must refuse state with `final_response_allowed=false` unless
the human has said the campaign was stopped/interrupted and the agent confirms
that execution is no longer active. Only then run:

```bash
python "$REPORTER" <job-dir> --confirm-interrupted
```

This records a reporting-time interruption assertion; it does not rewrite the
campaign state or pretend the runner finalized cleanly. It bypasses only stale
stop state. A `candidate` ledger row or pending-candidate state always blocks
finalization. Any available candidate manifest whose status is not a recognized
terminal value (`keep`, `discard`, `crash`, or `abandoned`) also blocks. Missing,
unknown, or unreadable status is unfinished evidence because completion cannot
be established. The agent must finalize or abandon that candidate first.

## Troubleshooting

| Error or symptom | Cause | Solution |
| --- | --- | --- |
| Reporting is refused because `final_response_allowed=false`. | The campaign may still be active, or persisted state may be stale after an interruption. | Confirm no campaign or job process remains. If a human confirmed the interruption, rerun with `--confirm-interrupted`; otherwise continue or stop the campaign through `nvflare-autofl`. |
| Reporting is refused for a pending candidate or unknown manifest status. | The recorded evidence cannot establish that candidate execution finished. | Finalize or abandon the candidate through `nvflare-autofl`, then regenerate the report. |
| The campaign lock is busy. | Another campaign or reporting process holds the lifecycle lock. | Identify and wait for the active process. Never bypass or delete a live lock. |
| The JSON says `progress_plot_available=false`. | Plotting dependencies are unavailable or the generated artifact is not a valid PNG. | Use the completed Markdown and JSON reports, review their warning, and install the campaign's plotting dependencies before retrying if a plot is required. |
| An output path is rejected. | The path aliases protected campaign evidence, source, or another output. | Choose distinct writable output paths outside protected campaign inputs and rerun. |

## Report Contract

The final report must include:

- campaign termination reason, objective, metric source, direction,
  environment, cap, abandoned-candidate count, and declared fixed budget;
- baseline, best retained result, score delta, runtime, failures, and status
  counts;
- selected-candidate rationale, strict retained improvements, representative
  non-improvements, grouped failures, and outcomes by recorded algorithm
  family;
- running-best trajectory selected by first, final, and largest objective improvements,
  plus a refreshed `progress.png` when plotting is available, with explicit
  plot availability in the JSON summary otherwise;

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills