brainharness-autoresearch

Files (13)

• plugin.json · .claude-plugin/
• .clawhubignore
• .gitignore
• CHANGELOG.md
• LICENSE
• README.md
• SKILL.md
• autoresearch.py
• eval-guide.md
• search-skill.json · examples/
• weekly-report.json · examples/
• plugin.json
• self-eval.json

SKILL.md (first lines)

# brainharness-autoresearch

> Previously published as `brainforge-autoresearch`, with earlier names `autoresearch` / `openclaw-autoresearch`. Renamed for the BrainHarness marketplace; functionality unchanged.

Autonomous prompt optimization for AI agent skills. Runs controlled experiments to find better prompt variants using the [Karpathy autoresearch pattern](https://github.com/karpathy/autoresearch): generate hypothesis, mutate prompt, evaluate, repeat.

## When to use

- 用户说"优化一下这个 skill" / User says "optimize this skill's prompt"
- 用户要对比不同 prompt 版本的效果 / User wants to benchmark prompt variants
- 用户说"run autoresearch on X" / "eval skill X" / "improve skill X"
- 用户对 skill 输出质量不满，想系统性改进 / User is unhappy with skill output quality and wants systematic improvement

**Do not use:**
- 一次性的小改动（直接改 prompt 即可） / One-off prompt tweaks — just edit the prompt directly
- 调试某个特定失败 case / Debugging a specific failure — investigate the root cause instead
- Skill 脚本本身有 bug（代码逻辑问题不是 prompt 问题） / Skill script has a bug — fix the code, not the prompt

## Requirements

- Python 3.10+
- `autoresearch.py` script in the skill directory
- LLM API access (MiniMax, OpenAI, or Anthropic)
- Target skill must have a prompt file (SKILL.md, SYSTEM.md, or similar)

## Procedure

Always follow these steps in order: (1) Create eval.json, (2) Run autoresearch command, (3) Review results and apply best prompt.

### Step 1: Gather context

Before running, you need:

| Parameter | Description | Example |
|-----------|-------------|---------|
| `--target` | Path to the skill directory or prompt file to optimize | `../workspace/skills/brain-search/SKILL.md` |
| `--evals` | Path to eval definition JSON file | `eval.json` |
| `--provider` | LLM provider for running experiments | `minimax` (default), `openai`, `anthropic` |
| `--runs` | Number of runs per experiment (statistical significance) | `5` (default) |
| `--max-experiments` | Maximum experiments before stopping | `30` (default) |
| `--dashboard` | Open live results dashboard in browser | flag, no value |

### Step 2: Create eval.json

Define test inputs and evaluation criteria. Each eval is a binary pass/fail check.

```json
{
  "test_inputs": [
    "search for latest AI agent frameworks",
    "find news about LLM inference optimization",
    "搜一下 transformer 架构的最新进展"
  ],
  "evals": [
    {
      "name": "has_sources",
      "type": "rule",
      "rule": "regex",
      "pattern": "(https?://|Source:|来源:)"
    },
    {
      "name": "no_hallucinated_urls",
      "type": "rule",
      "rule": "banned_phrases",
      "phrases": ["example.com", "placeholder.url"]
    },
    {
      "name": "sufficient_detail",
      "type": "rule",
      "rule": "word_count",
      "min": 50,
      "max": 500
    },
    {
      "name": "contains_summary",
      "type": "rule",
      "rule": "contains",
      "values": ["summary", "key findings", "结论"]
    },
    {
      "name": "no_apology_prefix",
      "type": "rule",
      "rule": "not_contains",
      "values": ["I apologize", "I'm sorry, but"]
    },
    {
      "name": "actionable_output",
      "type": "llm",
      "question": "Does the response provide actionable information the user can immediately use (links, specific facts, concrete next steps)?",
      "pass_description": "The response contains specific actionable items like URLs, concrete facts, or clear next steps",
      "fail_description": "The response is vague, generic, or lacks specific actionable information"
    }
  ]
}
```

**Rule types:**

| Rule | Parameters | Description |
|------|-----------|-------------|
| `regex` | `pattern` | Pass if regex matches output |
| `banned_phrases` | `phrases` (list) | Pass if NONE of the phrases appear |
| `word_count` | `min`, `max` (optional) | Pass if word count is within range |
| `contains` | `values` (list), optional `match`: `"any"` (default) or `"all"` | Pass if any/all values appear in output (case-insensitive) |
| `not_contains` | `values` (list) | Pass if NONE of the values appear in output (case-insensitive) |

**LLM eval type:**

| Field | Description |
|-------|-------------|
| `type` | Must be `"llm"` |
| `name` | Unique name for this eval |
| `question` | What to ask the judge LLM about the output |
| `pass_description` | Description of what a passing output looks like |
| `fail_description` | Description of what a failing output looks like |

See `eval-guide.md` for detailed guidance on writing effective evals.

### Step 3: Run autoresearch

From zning1994/brainharness-autoresearch, MIT.