scaffold-task

Files (1)

• SKILL.md

SKILL.md (first lines)

# Scaffold Task

Interactively scaffold a new Harbor-compatible benchmark task for CodeScaleBench. Supports both **SDLC tasks** (code changes in a single repo) and **org-scale tasks** (cross-repo discovery, compliance, incident triage — artifact-based evaluation). Generates all required files and registers the task in the selection registry.

This is an **interactive skill**. Walk the user through configuration using AskUserQuestion in multiple phases. Do NOT generate files without first collecting all required inputs.

---

## Phase 1: Mode Selection

Ask one question:

**Question 1** — Header: "Mode"
- Question: "What would you like to create?"
- Options:
  - **Add SDLC task to existing suite** — "Create a new task in an existing csb_sdlc_* benchmark (agent makes code changes)"
  - **Add org-scale task to existing suite** — "Create a new task in an existing csb_org_* benchmark (agent produces an answer artifact)"
  - **Create new SDLC suite** — "Create a new csb_sdlc_* benchmark with its first task and run config"
  - **Create new org-scale suite** — "Create a new csb_org_* benchmark with its first task and run config"

Set `{TASK_FAMILY}` to either `sdlc` or `org` based on the selection.

---

## Phase 2: Core Details

Ask 3-4 questions depending on mode.

### If adding to existing SDLC suite:

**Question 1** — Header: "Benchmark"
- Question: "Which SDLC benchmark suite?"
- Options:
  - **swebenchpro** — "Real-world SWE across repos (Go, TS, Python)"
  - **pytorch** — "PyTorch PR-level tasks (Python/C++)"
  - **locobench** — "Long-context understanding (mixed languages)"
  - **k8sdocs** — "Kubernetes documentation (Go)"
- Note: User can type "Other" for tac, largerepo, sweperf, crossrepo, dibench

### If adding to existing org-scale suite:

**Question 1** — Header: "Benchmark"
- Question: "Which org-scale benchmark suite?"
- Options:
  - **crossrepo** — "Cross-repo dependency tracing"
  - **crossrepo_tracing** — "Cross-repo config/dep tracing with provenance"
  - **crossorg** — "Cross-organization discovery"
  - **compliance** — "Compliance audit tasks"
  - **migration** — "Migration inventory and planning"
  - **incident** — "Incident debug and triage"
  - **onboarding** — "Onboarding comprehension"
  - **domain** — "Domain lineage tasks"
  - **platform** — "Platform knowledge tasks"
  - **security** — "Vulnerability remediation"
  - **org** — "Agentic correctness"

### If creating new suite:

Prompt the user (not AskUserQuestion) to provide:
- **Suite name**: Will become `csb_{sdlc|org}_{name}` (lowercase, alphanumeric + hyphens)

Then ask:

**Question 1** — Header: "Language"
- Question: "Primary language for this benchmark?"
- Options:
  - **python** — "Python 3.11 base image"
  - **go** — "Go 1.23 base image"
  - **typescript** — "Node 20 base image"
  - **cpp** — "GCC 13 base image"

**Question 2** — Header: "Difficulty"
- Question: "Task difficulty level?"
- Options:
  - **medium** — "1-3 files changed, straightforward"
  - **hard** — "4-10 files or complex logic"
  - **very_hard** — "10+ files, deep codebase knowledge"
  - **expert** — "Architectural-level, cross-module"

**Question 3** — Header: "Task type"
- Question: "How is the task environment set up?"
- Options:
  - **repo-clone** — "Clone a git repo at a specific commit (most common)"
  - **multi-repo-clone** — "Clone multiple repos (org-scale tasks)" *(org only)*
  - **pre-built-image** — "FROM an existing Docker image (e.g., TAC tasks)"
  - **standalone** — "Empty workspace, no repo (agent creates everything)"

For existing suites, also ask Language, Difficulty, and Task type (same questions).

---

## Phase 3: Task-Specific Inputs

Prompt the user for these values (use text prompts, not AskUserQuestion since these are free-form):

### SDLC tasks:

| Input | Required | Default | Example |
|-------|----------|---------|---------|
| Task ID | Yes | — | `my-feature-001` |
| Description | Yes | — | "Fix race condition in connection pool" |
| Repo (owner/name) | If repo-clone | — | `pytorch/pytorch` |
| Commit hash | If repo-clone | — | `ca2466126a00ba8fd877f5a185e40e36ddaceb87` |
| Base image | If pre-built | — | `ghcr.io/theagentcompany/tac-base:latest` |
| SDLC phase | Yes | "Implementation (feature)" | See list below |
| Category | Yes | "feature" | `bug_fix`, `refactoring`, `documentation`, etc. |
| Time limit (sec) | No | 900 | `600` |

Valid SDLC phases: "Requirements & Discovery", "Architecture & Design", "Implementation (feature)", "Implementation (bug fix)", "Implementation (refactoring)", "Testing & QA", "Documentation", "Maintenance"

### Org-scale tasks:

| Input | Required | Default | Example |
|-------|----------|---------|---------|
| Task ID | Yes | — | `CCX-crossorg-301` |
| Description | Yes | — | "Trace gRPC service dependency chain across K8s repos" |
| Repos (owner/name) | Yes (1+) | — | `kubernetes/kubernetes`, `kubernetes/client-go` |
| Commit/tag per repo | If repo-clone | — | `v1.32.0` |
| Category | Yes | — | `cross-repo-dep-trace`, `compliance-audit`, etc. |

From sourcegraph/CodeScaleBench, Apache-2.0.

More from sourcegraph/CodeScaleBench