general-plot-digitizer

Files (55)

• SKILL.md
• README.md · examples/01-single-curve/
• metadata.json · examples/01-single-curve/
• source.png · examples/01-single-curve/
• source_digitized.csv · examples/01-single-curve/
• source_digitized.md · examples/01-single-curve/
• source_digitized.overlay.png · examples/01-single-curve/
• source_digitized.xy · examples/01-single-curve/
• README.md · examples/02-multi-curve-color/
• metadata.json · examples/02-multi-curve-color/
• source.png · examples/02-multi-curve-color/
• source_citric_acid_aqueous_solution_digitized.csv · examples/02-multi-curve-color/
• source_citric_acid_aqueous_solution_digitized.md · examples/02-multi-curve-color/
• source_citric_acid_aqueous_solution_digitized.overlay.png · examples/02-multi-curve-color/
• source_citric_acid_aqueous_solution_digitized.xy · examples/02-multi-curve-color/
• source_citric_acid_solid_digitized.csv · examples/02-multi-curve-color/
• source_citric_acid_solid_digitized.md · examples/02-multi-curve-color/
• source_citric_acid_solid_digitized.overlay.png · examples/02-multi-curve-color/
• source_citric_acid_solid_digitized.xy · examples/02-multi-curve-color/
• README.md · examples/03-black-curve-text-mask/
• metadata.json · examples/03-black-curve-text-mask/
• source.png · examples/03-black-curve-text-mask/
• source_citric_acid_aqueous_solution_digitized.csv · examples/03-black-curve-text-mask/
• source_citric_acid_aqueous_solution_digitized.md · examples/03-black-curve-text-mask/
• source_citric_acid_aqueous_solution_digitized.overlay.png · examples/03-black-curve-text-mask/
• source_citric_acid_aqueous_solution_digitized.xy · examples/03-black-curve-text-mask/
• source_citric_acid_solid_digitized.csv · examples/03-black-curve-text-mask/
• source_citric_acid_solid_digitized.md · examples/03-black-curve-text-mask/
• source_citric_acid_solid_digitized.overlay.png · examples/03-black-curve-text-mask/
• source_citric_acid_solid_digitized.xy · examples/03-black-curve-text-mask/
• README.md · examples/04-stacked-spectra/
• metadata.json · examples/04-stacked-spectra/
• source.png · examples/04-stacked-spectra/
• source_nylon_6_6_digitized.csv · examples/04-stacked-spectra/
• source_nylon_6_6_digitized.md · examples/04-stacked-spectra/
• source_nylon_6_6_digitized.overlay.png · examples/04-stacked-spectra/
• source_nylon_6_6_digitized.xy · examples/04-stacked-spectra/
• source_polyethylene_digitized.csv · examples/04-stacked-spectra/
• source_polyethylene_digitized.md · examples/04-stacked-spectra/
• source_polyethylene_digitized.overlay.png · examples/04-stacked-spectra/
• source_polyethylene_digitized.xy · examples/04-stacked-spectra/
• source_polystyrene_digitized.csv · examples/04-stacked-spectra/
• source_polystyrene_digitized.md · examples/04-stacked-spectra/
• source_polystyrene_digitized.overlay.png · examples/04-stacked-spectra/
• source_polystyrene_digitized.xy · examples/04-stacked-spectra/
• metadata_schema.json · resources/
• digitize_pipeline.py · scripts/
• extract_metadata.py · scripts/
• isolate_curves.py · scripts/
• pixel_to_data.py · scripts/
• plot_utils.py · scripts/
• suggest_colors.py · scripts/
• upscale_image.py · scripts/
• vlm_metadata_json_prompt.txt · scripts/
• vlm_prompt_template.txt · scripts/

SKILL.md (first lines)

# General Plot Digitizer

## Goal

Extract calibrated numeric X-Y data from images of experimental spectra (Raman, XRD, UV-Vis, IR, NMR, etc.) using a deterministic "Agent-in-the-Loop" workflow.

The labor is divided between two models:
1. **Vision-Language Model (Visual Sensor)**: Reads the image and returns a rich, unstructured narrative description of axes, colors, and visual obstacles. It does **not** produce JSON.
2. **Coding Agent (Translator & Executor)**: Translates the VLM narrative into a precise `metadata.json`, runs the CV pipeline, inspects the overlay, and iterates until the curve is correctly isolated.

## Instructions

### Phase 1: Visual Inspection (VLM)

Do **not** attempt to generate JSON with the VLM. It acts only as a visual sensor.

1. **Generate grid overlay**:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/plot_utils.py plot.png --draw-grid
```
This produces `plot_grid.png` with a labeled pixel grid for precise coordinate reading.

2. **Prompt the VLM** to analyze `plot_grid.png` (not the raw image). Use the built-in vision capabilities or the `notify_user` VLM inspection tool. Provide the prompt guidelines from [scripts/vlm_prompt_template.txt](scripts/vlm_prompt_template.txt).

3. **Expected VLM output** — a natural-language report covering:
   - Axis labels, numeric ranges, and directions (is X reversed?).
   - Bounding box of the data region in pixels (read from the grid).
   - Color and style of each target curve (hex guess from the pixels on the line itself).
   - Pixel bounding boxes of **all visual obstacles** (legends, text annotations, gridlines, tick marks) that overlap the data curves.
   - Trace quality hints: thin/needle-like, thick/noisy, anti-aliased, JPEG artifacts.

### Phase 2: Metadata Construction (Coding Agent)

Read the VLM narrative and construct `metadata.json`. Schema: [resources/metadata_schema.json](resources/metadata_schema.json).

**Required fields:**
```json
{
  "plot_title": "",
  "x_axis_label": "Wavelength (nm)",
  "y_axis_label": "Absorbance",
  "x_tick_min": 400, "x_tick_max": 800,
  "y_tick_min": 0, "y_tick_max": 1,
  "x_calibration_points": [
    { "pixel": 70, "value": 400 },
    { "pixel": 450, "value": 800 }
  ],
  "x_scale": "linear", "y_scale": "linear",
  "bounding_box": {"x_min": 72, "y_min": 28, "x_max": 452, "y_max": 318},
  "x_reversed": false, "y_reversed": false,
  "spectrum_type": "UV-Vis",
  "curves": [{"label": "sample", "color_hint": "#1f77b4"}],
  "text_regions": [{"x_min": 300, "y_min": 50, "x_max": 400, "y_max": 80, "label": "legend"}]
}
```

**`x_calibration_points`** (strongly recommended): anchor the X-axis transform to exact pixel→value pairs read from the grid, rather than assuming axis ticks align perfectly with bbox edges. Pick two well-separated ticks visible on the grid. If provided, these override `x_tick_min/max` for pixel-to-data mapping.

**Translation rules** (VLM narrative → metadata fields):
- Obstacles → `text_regions[]` and/or `mask_regions[]` with pixel bounding boxes.
- Curve colors → `curves[].color_hint` (from **pixels on the plotted line**, not from legend swatches).
- "Curve is black / same as axes" → `"cli_hints": {"curve_is_black": true}`.
- "Noisy / jagged trace" → `"cli_hints": {"smooth": true}`.
- "Thin, needle-like peaks" → **do not** set smooth; set upscale in Phase 3 instead.

**If the VLM color guess is uncertain**, run:
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/suggest_colors.py plot.png \
  --bounding-box x_min,y_min,x_max,y_max
```
This reports dominant non-background colors in the cropped region. Use the top result as `color_hint`.

### Phase 3: Pipeline Execution

**Select CLI flags based on VLM visual cues:**

| VLM describes... | Required CLI flags | Avoid |
|---|---|---|
| Thin, needle-like peaks (XRD, FTIR) | `--crop-upscale 4.0` | `--smooth`, `--cluster-centroid` |
| Fuzzy / anti-aliased / JPEG artifacts | `--curve-tolerance 75` (up to 85) | — |
| Thick, noisy trace / scatter points | `--smooth --smooth-window 5 --smooth-deviation 15.0` | — |
| Black curve on black axes | `--allow-black` (auto-enables `--spatial-filter --cluster-centroid`) | — |
| Thin anti-aliased colored line | `--extraction-method edge+color` | `--morph-open` |

**Run the pipeline:**
```bash
${CLAUDE_SKILL_DIR}/../../venv/run cpu python ${CLAUDE_SKILL_DIR}/scripts/digitize_pipeline.py \
  plot.png \
  --full \
  --metadata metadata.json \
  --output-dir ./output \
  --overlay \
  --format both
```
Append the VLM-dictated flags from the table above. The pipeline also reads `cli_hints` from metadata and auto-applies safe flags (`--allow-black`, `--smooth`, `--all-curves`).

### Phase 4: Overlay Inspection & Iteration

1. **Inspect** `*_digitized.overlay.png` visually.
2. **If extraction is wrong**, diagnose using this table and re-run Phase 3:

| Symptom | Fix (metadata or CLI) |
|---------|----------------------|
| Wrong curve extracted (e.g. legend ink) | Set `curves[].color_hint` from actual line pixels; or run `suggest_colors.py` on a tight crop |
| Text / labels contaminating trace | Add bounding boxes to `text_regions[]` in metadata; add `--smooth` |
| Black curve picks up axis lines | `--allow-black` + add axis regions to `mask_regions[]` |
| Trace too sparse / broken gaps | `--curve-tolerance 55` or `--preset lowres` |
| Thin line lost entirely | `--extraction-method edge+color`; or `--crop-upscale 4.0` for needle peaks |
| Same-color text blobs on thick curve | `--morph-open` (caution: destroys thin <3px curves) |
| Low-res image, everything pixelated | `--upscale-strategy force` (pre-upscales image, scales metadata bbox) |
| X/Y values shifted or inverted | Fix `x_tick_min/max`, `x_reversed`, `y_reversed`, or `x_calibration_points` in metadata |

3. **Prefer metadata edits** (adjusting `color_hint`, `text_regions`, `mask_regions`, `bounding_box`) over adding CLI flags. Re-run the same pipeline command after editing `metadata.json`.

**Outputs:**
- `*_digitized.csv` — comma-separated with `x,y` header
- `*_digitized.xy` — space-separated, no header (if `--format xy` or `both`)
- `*_digitized.overlay.png` — visual QC
- `*_digitized.md` — summary

From learningmatter-mit/AtomisticSkills, MIT.

More from learningmatter-mit/AtomisticSkills