agent-run-data-collection

Files (2)

• SKILL.md
• trajectory-schema.md · references/

SKILL.md (first lines)

# Agent Run Data Collection

Collect complete, replayable agent trajectories without making telemetry a new
source of fabricated success or lost operator work.

## Required runtime configuration

Resolve these from owner-only runtime configuration; never commit their values:

- `NPA_AGENT_DATASET_TENANT_ID`: exact tenant that owns the destination.
- `NPA_AGENT_DATASET_URI`: exact `s3://<bucket>/<prefix>` dataset root.
- Existing NPA S3 endpoint and credentials for that bucket. Do not copy secrets
  into a trajectory, command line, workflow spec, log, or skill file.

Before enabling collection, verify the active agent deployment reports the same
tenant and that the destination bucket is a writable resource in that tenant.
Pass the verified active tenant and authorized dataset bucket to the emitter;
empty identity arguments fail closed. S3 access alone does not establish bucket
ownership: retain the provider ownership and deployment identity proof privately.
Tenant-wide discovery alone does not authorize cross-project writes. Do not
create a bucket, grant IAM, or change tenant/project configuration unless the
operator separately requests it.

## Collection hierarchy and episode boundary

Use three levels:

- **Session:** the whole conversation or operator work period.
- **Episode:** one accepted goal pursued until objective success, failure,
  refusal, cancellation, or an explicit handoff. Follow-up messages that refine
  the same unfinished goal stay in the episode; a materially new goal starts a
  new episode.
- **Event:** each prompt, plan, model decision, tool call, observation,
  confirmation, retry, correction, or final response inside an episode.

The episode is the dataset row. Assign its id when the goal is accepted and
finalize exactly one `npa.agent.trajectory.v1` record at the terminal boundary.
Link it to its parent session and include all nested events; do not create a
separate dataset row for every chat turn or tool call.

## Preamble integration

Link the skill explicitly from the agent preamble. Use this canonical line:

> For every goal-level episode, load and follow `$agent-run-data-collection` at `skills/atomic/agent-run-data-collection/SKILL.md`; record the episode from goal acceptance through success, failure, refusal, cancellation, or handoff as one sanitized trajectory containing all nested events, linked to its parent session and stored using `NPA_AGENT_DATASET_TENANT_ID` and `NPA_AGENT_DATASET_URI`.

The preamble selects the skill; the runtime emitter still must be implemented
and configured. Do not claim collection is active merely because this sentence
is present.

## Write contract

Write a new immutable object for every episode:

```text
<dataset-root>/episodes/<yyyy>/<mm>/<dd>/<episode-id>-<content-sha256>.json
```

Never append by rewriting one shared JSONL object: concurrent S3 writers lose
rows. Retrying the same finalized payload must resolve to the same object key;
different payloads for one episode id remain visible and must be treated as a data
quality conflict rather than overwritten. A sanitized correction does not
supersede an existing immutable claim: retain it pending and exclude the whole
conflicted episode from training until explicit data-quality reconciliation. A
delivery receipt proves byte delivery, not privacy or training eligibility.

Freeze the canonical raw record once with `collection.status=pending`. Its bytes
and content hash never change during delivery or retry. After exact S3
read-after-write verification, the runtime publishes a separate immutable
`npa.agent.trajectory-receipt.v1` delivery receipt and verifies that receipt too.
Only then may the emitter return `collected`. The pending status inside an
immutable raw record describes its state at finalization; consumers use its
verified receipt to establish delivery. A conditional episode claim identifies
divergent payloads for the same episode as a conflict while preserving both raw
objects. Do not retry without conditional-write support.

If the S3 write fails, persist the same finalized record to an owner-only local
outbox and mark collection `pending`. Never report the record as collected until
a read-after-write check confirms the object hash. Do not fail or repeat an
otherwise completed GPU or destructive operation merely because telemetry
failed. Retry pending records at the start of the next episode or through an
explicit flush.

The private outbox envelope binds each frozen record to the original tenant,
bucket, and prefix using a destination digest and includes a digest of the exact
payload bytes. Flush verifies that binding and both content digests before any
write, never rewrites the tenant, and retains mismatches or conflicts for
explicit reconciliation. Legacy unbound outbox records require reconciliation;
never silently redirect them to the current destination.

## Record contents

Read [references/trajectory-schema.md](references/trajectory-schema.md) before
implementing or changing the emitter. Preserve:

- sanitized request and relevant initial state;
- ordered plans, tool calls, arguments, observations, retries, and confirmations;
- objective outcome evidence, produced artifact URIs, and operator corrections;
- grounded/model routing, model name, token usage, latency, and agent/tool versions;
- tenant scope, dataset URI role, timestamps, redaction status, and collection status.

Use live workflow status, exit codes, validation reports, tests, and artifact
existence as outcome evidence. The agent's own final sentence is not proof of
success. Store operator corrections as explicit preference pairs when both the
rejected and corrected actions are available.

## Privacy and integrity

Redact before serialization and again before upload. Remove credentials,
authorization headers, environment dumps, private keys, signed URLs, raw image
data, and secret-shaped values. Replace concrete infrastructure identifiers in
free text with typed references; retain the configured tenant id only in the
access-controlled scope field needed to prove routing. Keep raw and curated
datasets separate, and never train on evaluation or held-out runs.

The runtime removes whole private-key blocks, credential fields/environment
payloads, data URIs, private URI paths, concrete resource IDs and addresses,
including identifiers used as mapping keys. Redact the configured dataset URI
and its nonempty prefix even when the prefix appears alone as a directory name

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai