databox-datasets

Files (1)

• SKILL.md

SKILL.md (first lines)

# Databox Dataset Management & Data Ingestion

Full dataset lifecycle — create, ingest data, monitor, configure, and delete — via the `databox` CLI.

## Prerequisites

Must be authenticated. If not, use the `databox-auth` skill first.

## Required Role

The API applies the Databox app's role rules, checked in the organization or account that owns the dataset:

- Reads (every command that changes nothing) and `ingest`: anyone with access to the dataset.
- `update`, `set-sync-frequency`: Admin, User or Editor. A Viewer can only read.
- `create`: an admin, or the user who created the data source it goes into.
- `delete`, `purge`, `set-timezone`, `set-permissions`, `set-metadata`, `set-column-metadata`, `update-modification`, `clear-modifications`, `duplicate`, `set-verification`: an admin, or the user who created the dataset.

A refusal is `forbidden` (exit 1) and nothing is changed. Do not retry it: tell the user which role the command needs, so an admin or the creator can run it.

## Quick Reference

| Task | Command |
|------|---------|
| List datasets | `databox dataset list` |
| List a data source's datasets | `databox dataset list --data-source-id ID` |
| List merged datasets | `databox dataset list --type mergedDataset` |
| Create dataset | `databox dataset create --name "Name" --data-source-id ID` |
| Get dataset details | `databox dataset get ID` |
| View schema and primary key | `databox dataset schema ID` |
| View data | `databox dataset data ID` |
| View data, sorted | `databox dataset data ID --sort-by COLUMN_ID --sort-order desc` |
| Export all rows as CSV | `databox dataset data ID --all --output csv > rows.csv` |
| Ingest inline | `databox dataset ingest ID --records '[...]'` |
| Ingest from file | `databox dataset ingest ID --file data.json` |
| Ingest from stdin | `cat data.json \| databox dataset ingest ID` |
| List ingestions | `databox dataset ingestions ID` |
| Get ingestion detail | `databox dataset ingestion ID INGESTION_ID` |
| Ingestion stats | `databox dataset ingestion-statistics ID` |
| Duplicate dataset | `databox dataset duplicate ID` |
| Update name | `databox dataset update ID --name "New Name"` |
| Set timezone | `databox dataset set-timezone ID --timezone "US/Eastern"` |
| Sync frequency options | `databox dataset sync-frequency-options ID` |
| Set sync frequency | `databox dataset set-sync-frequency ID --interval 60` |
| Sync history / statistics | `databox dataset sync-history ID` / `databox dataset sync-statistics ID` |
| View permissions | `databox dataset permissions ID` |
| Set permissions | `databox dataset set-permissions ID --access-level everyone` |
| View metadata | `databox dataset metadata ID` |
| Set metadata | `databox dataset set-metadata ID --description "..." --default-time-dimension COLUMN_ID` |
| View column metadata | `databox dataset column-metadata ID` |
| Set column metadata | `databox dataset set-column-metadata ID --columns '[{"id":"amount","conceptType":"measure"}]'` |
| View verification | `databox dataset verification ID` |
| Set verification | `databox dataset set-verification ID --status verified` |
| View lineage | `databox dataset lineage ID` |
| View modifications | `databox dataset modifications ID` |
| Preview a modification | `databox dataset preview-modification ID --data '{...}'` |
| Save a modification | `databox dataset update-modification ID --data '{...}'` |
| Clear modifications | `databox dataset clear-modifications ID --force` |
| Filter operators and type conversions | `databox dataset modification-rules` |
| Formula functions | `databox dataset modification-functions` |
| Purge data | `databox dataset purge ID --force` |
| Delete dataset | `databox dataset delete ID --force` |

## Create Flags

| Flag | Required | Description |
|------|----------|-------------|
| `--name` | Yes | Dataset name |
| `--data-source-id` | Yes | Parent data source ID |
| `--primary-key` | No | Primary key column id (repeat for several) |
| `--schema` | No | JSON array of columns, each `{"id", "dataType"}` |
| `--idempotency-key` | No | UUID; a retry with the same key within 24 hours does not create a second dataset |

## Schema Definition

Pass the schema as a JSON array with `--schema`. Each column is `{"id", "dataType"}`:

```bash
databox dataset create \
  --name "Web Analytics" \
  --data-source-id 42 \
  --primary-key date \
  --primary-key page \
  --schema '[{"id":"date","dataType":"datetime"},{"id":"page","dataType":"string"},{"id":"views","dataType":"number"}]'
```

Valid `dataType` values: `string`, `number`, `datetime`. The column `id` is what every other command uses to refer to the column: `--primary-key`, `--sort-by`, column metadata, modifications, and metric column references.

## Data Ingestion

Three input methods — use exactly one:

**Inline JSON:**
```bash
databox dataset ingest 67890 --records '[{"date":"2024-01-01","views":100}]'
```

**From file:**
```bash
databox dataset ingest 67890 --file ./metrics.json
```

**From stdin (pipe):**
```bash
cat metrics.json | databox dataset ingest 67890
```

Pass `--idempotency-key "$(uuidgen)"` when a retry must not ingest the same rows twice; re-run with the same key and the same records. The same key with different records fails with `idempotency_key_reused` (exit 1) and ingests nothing: use a new key for a new batch.

## Common Workflow: Full Data Pipeline

```bash
# 1. Create a data source
databox data-source create --name "My App" --json
# Returns: {"id": 42, ...}

# 2. Create a dataset with schema
databox dataset create \
  --name "Daily Metrics" \
  --data-source-id 42 \

From databox/databox-cli, MIT.

More from databox/databox-cli