ci

Files (1)

• SKILL.md

SKILL.md (first lines)

# Wire coder-eval into GitHub Actions

The user's request is: `$ARGUMENTS`

## Step 1 — Check the repository

Find the repository's task tree by following
`${CLAUDE_PLUGIN_ROOT}/reference/repo-layout.md`, and check whether `.github/workflows/`
exists. The paths you resolve here become `args:` entries in the workflow in step 3 — that
input is written from discovery, never from a fixed guess.

If there is no `.github/` directory at all, say that this skill targets GitHub Actions
and stop — do not invent an equivalent for another CI system unless the user asks.

If a workflow already runs coder-eval (grep the workflows for `coder_eval`), do not add a
second one. Show what is there and offer to update it.

## Step 2 — Choose the trigger

Ask, or infer from the request:

- **On pull request** — gate changes to the tasks or to whatever they exercise.
- **On a schedule** — the skill-drift case: re-run the suite weekly against the current
  model so a skill that quietly stops triggering surfaces before users hit it. This is
  the trigger most repositories actually want, and the one they forget. If the suite is
  an activation suite, the environment note in step 3 is **not optional** for this
  trigger — without it the scheduled run reports total drift every week regardless of
  whether anything drifted.
- **Both**, which is fine — one workflow, two `on:` keys.

## Step 3 — Emit the workflow

The composite action installs the `coder-eval` CLI and nothing else: it is
agent-agnostic and installs **no coding-agent runtime**. A task using the default
`claude-code` agent therefore needs Node plus the Claude CLI provided by the job first,
or the run dies on a missing `claude` binary. There is no Marketplace install step for
the action itself, but those two prerequisite steps are not optional.

```yaml
name: Coder Eval

on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"   # Mondays 06:00 UTC — catches model/skill drift

# Least privilege: this job runs agent-generated code, so it gets no write scope.
permissions:
  contents: read

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v6
        with:
          # Do not leave a credentialed .git/config in a workspace where
          # agent-generated code runs.
          persist-credentials: false

      # The action installs no coding-agent runtime — provide it here.
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
      - run: npm install -g @anthropic-ai/claude-code

      - id: eval
        uses: UiPath/coder_eval@v0
        with:
          run-dir: runs/ci
          args: |
            tasks/**/*.yaml
            --model
            claude-haiku-4-5-20251001
          env: |
            ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}

      - if: always()
        run: cat "${{ steps.eval.outputs.run-md-path }}" >> "$GITHUB_STEP_SUMMARY"
```

Adjust the model and the cron to the repository. Pin the action at `@v0`, the moving major
tag. Then work through the four things the snippet cannot guess.

Note the shape of `args:`: the action promotes **none** of `coder-eval run`'s flags to a
named input, so task paths and flags all go there, one argument per line, with a flag and
its value on separate lines. There is no `tasks:`, `tags:` or `model:` input.

### The task paths — from discovery

The value above is a placeholder for whatever step 1 discovered. `args:` entries are
handed to the CLI **verbatim**: no word splitting, no pathname expansion. The CLI expands
the globs itself, which makes this simpler than it looks:

- **`**` works.** `tasks/**/*.yaml` is genuinely recursive, so one pattern covers a tree
  of any depth. No per-depth ladder, no `globstar` caveat.
- **A glob matching nothing fails loudly** with `No task files found!` and exit 1, rather
  than reaching the CLI as a literal path.
- **One path per line.** Two globs are two lines, not one space-separated string, which
  would arrive as a single malformed argument.

So emit what step 1 found, one entry per line:

```yaml
args: |
  tests/tasks/**/*.yaml
```

### `version:` — conditional on the repository's pin

If the repository pins a coder-eval version, **pass it** and say why: the gate should run
the CLI the repo is authored against, not whichever release the action defaults to. If
there is no pin, omit the input and let the action's default track the matching release.
`${CLAUDE_PLUGIN_ROOT}/reference/cli-setup.md` covers how to find a pin — and passing one
explicitly is right even when it happens to match today's default, because it is
self-documenting.

### The experiment, if the suite runs through one

From UiPath/coder_eval, Apache-2.0.

More from UiPath/coder_eval