data-analysis

Files (2)

• SKILL.md
• profile_data.py · scripts/

SKILL.md (first lines)

## Which engine am I in?

Read the first line of the RunPython tool description:

- **PYODIDE** — Python in WebAssembly. Pure-Python packages install on import;
  no subprocess, no system binaries.
- **LOCAL SUBPROCESS** — the user's real Python. Everything works, including
  system tools — check with `shutil.which(...)` before relying on one.
- **PODMAN CONTAINER** — isolated container with python3, nodejs and tectonic.

Files persist between RunPython calls in the same chat, so build in steps and
inspect as you go rather than writing one long script.

## Never state a number you did not compute

This is the whole discipline of the skill. Load the data, compute, and quote the
output. An estimate presented as a result is the worst failure mode here.

## Look before you aggregate

```python
import pandas as pd
df = pd.read_csv('data.csv')
print(df.shape)
print(df.dtypes)
print(df.head())
print(df.isna().sum())
print(df.describe(include='all'))
```

Check dtypes early: a numeric column read as `object` means something
non-numeric is hiding in it, and every later sum will be wrong or crash.

## Cleaning

```python
df['amount'] = pd.to_numeric(df['amount'], errors='coerce')
df['date'] = pd.to_datetime(df['date'], errors='coerce')
bad = df[df['amount'].isna() | df['date'].isna()]
print('unparsable rows:', len(bad))
```

Report what you dropped and why. Silently discarding rows changes the answer.

## Finding what is wrong with the data

Outliers are usually data-entry errors, not discoveries. Check magnitude
against the column's own distribution:

```python
q1, q3 = df['price'].quantile([.25, .75])
iqr = q3 - q1
odd = df[(df['price'] < q1 - 3*iqr) | (df['price'] > q3 + 3*iqr)]
print(odd)
```

A price 100× the others is a misplaced decimal point. Say that, show the row,
and give both the raw and the corrected figure — do not silently "fix" it.

## Aggregating

```python
g = df.groupby('region').agg(
    revenue=('amount', 'sum'),
    orders=('amount', 'size'),
    avg=('amount', 'mean'),
).sort_values('revenue', ascending=False)
print(g.to_string())
```

## Presenting

Give exact numbers with units and the period they cover. State the row count
you actually used after cleaning. If one row dominates a total, say so — a
leader created by a single anomaly is not a leader.

For a chart, see the `charts` skill. For delivering results as a document, see
`xlsx` or `docx`.

## Large files

Read in chunks (`pd.read_csv(..., chunksize=100_000)`) rather than loading
everything, and say which subset you analysed if you sampled.

From iaa2005/monet-directory, MIT.

More from iaa2005/monet-directory