artificial-analysis

Files (8)

• SKILL.md
• open_weights_overrides.json · assets/
• aa.py · scripts/
• README.md · tests/fixtures/
• legacy_models.json · tests/fixtures/
• media_text_to_image_free.json · tests/fixtures/
• v2_free_page1.json · tests/fixtures/
• v2_free_page4.json · tests/fixtures/

SKILL.md (first lines)

# artificial-analysis

Stdlib-only client (`scripts/aa.py`) for the Artificial Analysis Data API. It answers three
kinds of question:

1. **Compare** — head-to-head benchmark, capability, speed and price table for named models.
2. **Rank / search** — sort the catalog by intelligence, coding, agentic index, speed or price.
3. **Local fit** — which *open-weights* models actually fit a given RAM budget at a given
   quantisation, so the user knows what to download for a llama.cpp / vLLM box.

## Prerequisites

```bash
python3 --version                                    # 3.8+, standard library only
echo "$AA_API_KEY" | head -c 4                       # key present? (never print the whole key)
```

An API key is needed for live calls only — get a free one at
<https://artificialanalysis.ai/data-api>. `--help` and `--fixture-dir` work with no key and
no network.

## Key resolution

Checked in this order, first match wins. The key is never printed or logged.

1. Environment variable `AA_API_KEY`
2. `~/.config/artificialanalysis/config.env`, any line of the form `AA_API_KEY="..."`
   (also accepts `ARTIFICIAL_ANALYSIS_API_KEY`, `api_key`, `API_KEY`, and a leading `export`)

If neither exists the command exits 1 and names both locations.

## Tier and rate-limit situation (important)

- Base URL `https://artificialanalysis.ai/api/v2`, auth header `x-api-key`.
- **Free tier: 100 requests per fixed 24-hour window**, shared across the whole org scope.
  Responses carry `X-RateLimit-Limit/Remaining/Reset` and `X-AA-Tier`; `aa.py` prints
  `AA rate limit remaining: N` to stderr after every live call.
- A `429` response includes `Retry-After`; `aa.py` prints it and exits non-zero. Back off —
  do not retry in a loop, the window is fixed and cannot be reset.
- Pro endpoints (e.g. `/api/v2/language/models`) return **403** on a free key. Only use the
  `.../free` endpoints and `/data/llms/models`.
- The free tier has **no** `open_weights` flag, no context window, no parameter counts and no
  `huggingface_url` (Pro-only). Open-weights status and parameter counts are therefore
  *heuristics* in this skill — see [Open-weights heuristic](#open-weights-heuristic).

**Budget discipline:** develop and test offline with `--fixture-dir`. One `models` query costs
1 request plus one per page of the paginated free endpoint (4 pages at page_size 200). The
cache (below) makes repeats free.

## Caching

Every live response is cached as JSON under the cache directory with a timestamp.

- Default cache dir: `~/.cache/artificial-analysis` (override with `--cache-dir DIR`)
- TTL: 21600 s (6 h), override with env `AA_CACHE_TTL_SECS`
- `--refresh` ignores the cache and re-fetches (still writes it back)
- `--fixture-dir` makes **zero** HTTP requests and reads nothing from the cache

## Global flags

```text
aa.py [--cache-dir DIR] [--refresh] [--fixture-dir DIR] [--json] <subcommand> ...
```

The four global flags are accepted before *or* after the subcommand. `--json` emits machine
readable output instead of a table (except `raw`, which is always JSON).

Every human-readable output ends with the attribution line required by Artificial Analysis:

```text
Data: artificialanalysis.ai (Intelligence Index v4.3)
```

**Keep that line** whenever you paste results into a report, README, ticket or chat message.

## Subcommands

### 1. `models` — browse and rank the catalog

```bash
aa.py models --limit 15                              # top 15 by intelligence index
aa.py models --min-intel 20 --sort speed
aa.py models --creator deepseek --since 2026-01-01
aa.py models --sort price --json
```

- Filters: `--creator SUBSTR` (matches creator *or* model name), `--min-intel N`,
  `--since YYYY-MM-DD`
- `--sort intel|coding|agentic|speed|price|release` (default `intel`; `price` is cheapest
  first, everything else descending), `--limit N` (default 25)
- Columns: `MODEL CREATOR INTEL CODING AGENTIC TOK/S $/1M OUT RELEASED`
- Data comes from **both** endpoints joined on the stable model `id`

### 2. `compare SLUG [SLUG...]` — side-by-side table

```bash
aa.py compare claude-opus-5-5 gpt-6-astra-xhigh
aa.py compare qwen3-5-27b deepseek-v3-1-terminus-reasoning --json
```

Rows: creator, release date, Intelligence/Coding/Agentic indices, the six Capability Indexes
(finance, strategy/ops, legal, healthcare, engineering, economics), per-benchmark scores
(MMLU-Pro, GPQA, HLE, LiveCodeBench, SciCode, MATH-500, AIME as 0-1 fractions),
input/output/blended price per 1M tokens, median tokens/s, median TTFT, open-weights flag.

Unknown slug → exit non-zero with `difflib` suggestions; slugs may change over time, prefer
model `id`s in scripts (`--json` includes `id`).

### 3. `search TEXT` — substring search

```bash
aa.py search qwen
aa.py search "deepseek" --json
```

Case-insensitive match on model name and creator, prints `MODEL CREATOR INTEL RELEASED`
sorted by intelligence index.

### 4. `open-weights` — what fits on my machine

From Kubementat/ai_agent_skills, MIT.

More from Kubementat/ai_agent_skills