custom-metrics

Files (1)

• SKILL.md

SKILL.md (first lines)

# Custom Metrics for Configs

Full lifecycle management of custom business metrics: create metric definitions with the MCP server, track events via SDK, retrieve metric data, and manage metrics programmatically.

## Prerequisites

- LaunchDarkly AI SDK initialized (see `sdk-install`). The SDK reads `LD_SDK_KEY` from the environment.
- The remotely hosted LaunchDarkly MCP server configured in your environment
- A LaunchDarkly API token with the `writer` role, only to update or delete metrics, or for scripts that can't use MCP
- Understanding of built-in agent metrics (see `built-in-metrics`)

## API Access

Use the LaunchDarkly MCP server tools wherever one exists: they authenticate via OAuth and need no token. Create metrics with `create-metric`, check that events are arriving with `list-metric-events`, and read definitions with `get-metric` and `list-metrics`. The MCP server can't update or delete metrics, so those steps use the REST API. For them, or for scripts that can't use MCP, ask the user to paste an API access token for this session and substitute it for `{api_token}` in the examples below. Do not search environment variables, `.env` files, or agent/MCP config files for credentials.

## Metrics Lifecycle Overview

| Step | Method | Purpose |
|------|--------|---------|
| 1. Create | MCP `create-metric` | Define metric in LaunchDarkly |
| 2. Track | SDK | Send events to the metric |
| 3. Get | MCP `get-metric`, `list-metrics` | Retrieve metric definition/data |
| 4. Update | API | Modify metric properties |
| 5. Delete | API | Remove metric |

## 1. Create Metric

Call `create-metric`. `eventKey` must match the key your code passes to `track()`:

```json
{
  "projectKey": "support-ai",
  "key": "ai.task.completion",
  "name": "Agent Task Completion Rate",
  "kind": "custom",
  "eventKey": "ai.task.completion",
  "measureType": "value",
  "valueAggregation": "average",
  "successCriteria": "HigherThanBaseline",
  "unit": "count",
  "description": "Tracks successful agent task completions"
}
```

`measureType` is `count` (how many times the event fired), `occurrence` (whether each user fired it at all), or `value` (a number sent with the event). Then confirm with `get-metric`.

### Scripts without MCP

The REST API equivalent:

**Required fields for numeric custom metrics:**
- `successCriteria` - Must be one of: `"HigherThanBaseline"`, `"LowerThanBaseline"`
- `unit` - e.g., `"count"`, `"percent"`, `"milliseconds"`

The API will return `400 Bad Request` if these are missing for numeric metrics.

```python
import requests
import os

def create_metric(
    project_key: str,
    metric_key: str,
    name: str,
    kind: str = "custom",
    is_numeric: bool = True,
    unit: str = "count",
    success_criteria: str = "HigherThanBaseline",
    event_key: str = None,
    description: str = None
):
    """Create a new metric definition in LaunchDarkly."""
    API_TOKEN = "{api_token}"  # token the user provided for this session

    url = f"https://app.launchdarkly.com/api/v2/metrics/{project_key}"

    payload = {
        "key": metric_key,
        "name": name,
        "kind": kind,
        "isNumeric": is_numeric,
        "eventKey": event_key or metric_key
    }

    # Unit and successCriteria are required for numeric custom metrics
    if is_numeric and kind == "custom":
        payload["unit"] = unit
        payload["successCriteria"] = success_criteria

    if description:
        payload["description"] = description

    headers = {
        "Authorization": API_TOKEN,
        "Content-Type": "application/json"
    }

    response = requests.post(url, json=payload, headers=headers)

    if response.status_code == 201:
        print(f"[OK] Created metric: {metric_key}")
        return response.json()
    elif response.status_code == 409:
        print(f"[INFO] Metric already exists: {metric_key}")
        return None
    else:
        print(f"[ERROR] Failed to create metric: {response.status_code}")
        print(f"        {response.text}")
        return None
```

**Metric Kinds:**
- `custom` - Track any event (most common for agent metrics)
- `pageview` - Track page views
- `click` - Track click events

**Success Criteria** (for numeric metrics):
- `HigherThanBaseline` - Higher values are better (e.g., revenue, satisfaction)
- `LowerThanBaseline` - Lower values are better (e.g., errors, latency)

From launchdarkly/ai-tooling, Apache-2.0.

More from launchdarkly/ai-tooling