conversation-review

Files (1)

• SKILL.md

SKILL.md (first lines)

## Overview

This skill is for **semantic and context layer curators** who want to use accumulated user feedback to drive continuous improvement. It structures the work into two phases:

1. **Categorize** — review past conversations with feedback and classify each piece of feedback into an actionable improvement category
2. **Act** — turn the categorized feedback into a concrete list of changes, then apply them to the semantic model or context layer on a branch

---

## Prerequisites

Before starting, ensure a workspace and branch are set. Use `get_session_workspace_and_branch` to check the current context. If nothing is set, use `list_workspaces` and `set_session_workspace_and_branch`. See the `workspace-branch` skill for the full reference.

You will be reading the model as-is (use `prod` branch for reading), then applying changes on a development branch later.

---

## Phase 1: Collect Conversations

### 1.1 Fetch conversations with feedback

Use `list_analysis_chats` to retrieve past conversations. Focus on those that have received feedback — positive or negative.

```
list_analysis_chats(limit=50, offset=0)
```

Paginate with `offset` until you have covered the desired time window or conversation count. For each conversation, note:

- `conversation_id`
- `title`
- `user_feedback` (the stored feedback text, if any)
- `agent` and `domain`
- creation time and the display name of the user who created it
- the client it was started from and the model it ran with
- how its last question ended: `DONE`, `ASK`, `FAIL`, or `ABORTED`

**Filter to conversations with feedback.** Skip conversations with no feedback unless the user explicitly wants to review all conversations. One exception: include conversations whose last question ended in `FAIL` even without feedback — a failed analysis is a signal in itself, and users rarely rate a conversation that never answered.

If the stored feedback is inaccurate, or a conversation you are reviewing has none, use `provide_analysis_feedback` to record the correct label before categorizing. A `FAIL` conversation with no feedback is the exception: read it first with `get_stored_conversation` (see 1.2) and label it only once you know what went wrong. Leave its feedback empty if the failure has no clear cause.

```
provide_analysis_feedback(conversation_id="abc123", feedback="Data Issue: net revenue metric missing")
provide_analysis_feedback(conversation_id="def456", feedback="Good")
```

This keeps the stored feedback in sync with the categorization, so future reviews start from an accurate baseline.

### 1.2 Read the full conversation when needed

If the feedback text alone is not enough to understand what went wrong, retrieve the full conversation content with `get_stored_conversation`:

```
get_stored_conversation(conversation_id="abc123")
```

This returns all final responses and plan/interpretation messages. Read these to understand what the AI actually did — what question it interpreted, what approach it took, and what it concluded.

To also see individual analysis steps (useful when you want to identify which step produced wrong data), pass `with_step_ids: true`:

```
get_stored_conversation(conversation_id="abc123", with_step_ids=true)
```

This includes `step_start` and `step_insight` entries with `step_id` values you can then pass to `get_analysis_step_details` to retrieve the exact semantic query and SQL used in that step.

---

## Phase 2: Categorize Feedback

For each conversation with feedback, assign it to one or more of the following **action categories**. A single conversation can yield multiple categories.

### Feedback → Action Category Mapping

| Feedback Signal | Root Cause | Action Category |
|---|---|---|
| `Data Issue: X metric doesn't exist` | Missing aggregation | **Add metric** |
| `Data Issue: wrong value / incorrect result` | Metric or attribute SQL is wrong | **Fix calculation** |
| `Data Issue: missing dimension / can't slice by X` | Missing attribute | **Add attribute** |
| `Wrong Judgement: used wrong field` | Ambiguous names or descriptions | **Improve metadata** |
| `Wrong Judgement: missed a standing rule` | Missing instruction | **Add instruction** |
| `Wrong Judgement: didn't follow a procedure` | Missing analytical skill | **Add skill** |
| `Wrong Judgement: lacked business context` | Missing knowledge or memory | **Add knowledge / memory** |
| `Wrong Judgement: wrong domain or agent` | Agent misconfigured | **Update agent** |
| `Chart Issue` | Visualization only | **No semantic action** |
| Last question ended in `FAIL` (no feedback) | Analysis could not complete | **Manual review** — read the conversation with `get_stored_conversation` to find what it tried |
| `Good` / positive | Working as intended | **No action** |
| `Other: X` | Ambiguous | **Manual review** |

### Categorization Output

Produce a table summarizing all categorized feedback before proceeding:

| Conversation | Feedback | Category | Proposed Change |
|---|---|---|---|
| `abc123` — "Revenue by region" | `Data Issue: net revenue metric missing` | Add metric | Add `net_revenue` to `orders` entity |
| `def456` — "Churn analysis" | `Wrong Judgement: used gross revenue` | Add instruction | "Always use `orders.net_revenue` for revenue questions" |
| `ghi789` — "Top products" | `Good` | No action | — |

Group the proposed changes by category to make patterns visible. If three conversations all flag the same missing metric, that is one action item, not three.

---

## Phase 3: Build Action Items

Turn the categorized feedback into a concrete, prioritized action list. Each action item should be self-contained enough to execute in a single MCP call.

### Action Item Format

For each item, record:

- **Type**: what kind of change (`add_metric`, `fix_attribute`, `add_instruction`, `add_skill`, `add_knowledge`, `add_memory`, `update_agent`, `improve_description`)
- **Target**: the entity, field, agent, or context item to create or update
- **Evidence**: the conversation IDs and feedback text that motivated this item
- **What to do**: a one-sentence description of the specific change

### Action Item Types and Tools

| Type | Tool(s) |

From honeydew-ai/honeydew-ai-coding-agents-plugins, Apache-2.0.

More from honeydew-ai/honeydew-ai-coding-agents-plugins