rag-anti-hallucination-workflow

Files (1)

• SKILL.md

SKILL.md (first lines)

# rag-anti-hallucination-workflow

## Goal

Reduce hallucinations in RAG through explicit query routing, proper response mode selection, and measurable response quality validation.

## When to use

- When the response is decision-oriented and requires high grounding.
- When you analyze status/debug and need fast, controlled retrieval.
- When telemetry shows grounding/freshness decline or waste increase.

## Workflow 01..05

### 01 deterministic vs retrieval

- Choose `deterministic` when the result can be derived from local data/rules.
- Choose `retrieval` when context from graph or trend history is required.

### 02 query_type + workflow_phase

- Set `query_type` (for example `status`, `debug`, `decision`, `feature`).
- Set `workflow_phase` (for example `triage`, `analysis`, `implementation`, `verification`).
- Base routing on `query_type/workflow_phase` pair, not on a single signal.

### 03 preferred node types (decision/workflow_stats/ownership)

- For decisions prioritize `decision` nodes.
- For trends and status prioritize `workflow_stats` nodes.
- For ownership and domain boundaries prioritize `ownership` nodes.

### 04 top-k + scoring

- For `status/debug`, use lower `top-k` and lower scoring threshold (fast, narrow context).
- For `decision/feature`, use higher `top-k` and higher scoring threshold (fuller context).
- When results are inconsistent, increase scoring first, then `top-k`.

### 05 logging usefulness/grounding/freshness/waste

- Log every response with metrics: `usefulness`, `grounding`, `freshness`, `waste`.
- Evaluate trend, not only a single point.
- On degradation run historical comparison and save a snapshot.

## Operational commands

```powershell
# Do not run `pwsh ...` directly from VS Code terminal profile; use runner or fallback executable resolution.
# Runner-first pattern: execute the target command inside ScriptBlockText.
$instructionText = @'
try {
    .github/agents/report-rag-trend-history.ps1 -ProjectRoot . -LastN 30 -CompareRowsBack 7
    [PSCustomObject]@{ status='pass'; succeeded=$true; message='Execution succeeded.'; error_type=''; error_message=''; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
catch {
    [PSCustomObject]@{ status='fail'; succeeded=$false; message='Execution failed.'; error_type=$_.Exception.GetType().FullName; error_message=$_.Exception.Message; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
finally {}
'@
.github/agents/invoke-isolated-powershell.ps1 -Mode detached -ScriptBlockText $instructionText -TimeoutSec 120 -EmitJson

# Example trend script with `-EmitJson`:
$instructionText = @'
try {
    .github/agents/report-rag-trend-history.ps1 -ProjectRoot . -LastN 30 -CompareRowsBack 7 -EmitJson
    [PSCustomObject]@{ status='pass'; succeeded=$true; message='Execution succeeded.'; error_type=''; error_message=''; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
catch {
    [PSCustomObject]@{ status='fail'; succeeded=$false; message='Execution failed.'; error_type=$_.Exception.GetType().FullName; error_message=$_.Exception.Message; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
finally {}
'@
.github/agents/invoke-isolated-powershell.ps1 -Mode detached -ScriptBlockText $instructionText -TimeoutSec 120 -EmitJson
```

## Output and debug

- query/finalize/build: quiet-first (`OutputMode quiet`), diagnostic-on-error (`OutputMode diagnostic`).
- On error: first read `.github/agents/diagnostics/*.json`, then retry in diagnostic mode.
- Run trend scripts (`report-rag-trend-history.ps1`, `record-rag-trend-snapshot.ps1`) without `OutputMode`; for diagnostics use `-EmitJson` where supported.

```powershell
# query: default quiet
$instructionText = @'
try {
    .github/agents/query-rag-graph.ps1 -ProjectRoot . -OutputMode quiet
    [PSCustomObject]@{ status='pass'; succeeded=$true; message='Execution succeeded.'; error_type=''; error_message=''; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
catch {
    [PSCustomObject]@{ status='fail'; succeeded=$false; message='Execution failed.'; error_type=$_.Exception.GetType().FullName; error_message=$_.Exception.Message; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
finally {}
'@
.github/agents/invoke-isolated-powershell.ps1 -Mode detached -ScriptBlockText $instructionText -TimeoutSec 120 -EmitJson

# query: retry only after failure
$instructionText = @'
try {
    .github/agents/query-rag-graph.ps1 -ProjectRoot . -OutputMode diagnostic
    [PSCustomObject]@{ status='pass'; succeeded=$true; message='Execution succeeded.'; error_type=''; error_message=''; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
catch {
    [PSCustomObject]@{ status='fail'; succeeded=$false; message='Execution failed.'; error_type=$_.Exception.GetType().FullName; error_message=$_.Exception.Message; duration_ms=0 } | ConvertTo-Json -Compress | Write-Output
}
finally {}
'@
.github/agents/invoke-isolated-powershell.ps1 -Mode detached -ScriptBlockText $instructionText -TimeoutSec 120 -EmitJson
```

## PASS/WARN/BLOCK

- PASS: grounding>=40, freshness>=35, waste<=70
- WARN: grounding 25-39 lub freshness 20-34 lub waste 71-85
- BLOCK: grounding<25 lub freshness<20 lub waste>85

## DoD

- For each query, mode `deterministic` or `retrieval` was set.
- `query_type/workflow_phase` routing was applied and `top-k/scoring` was explicitly selected.
- `usefulness/grounding/freshness/waste` was logged and PASS/WARN/BLOCK was evaluated.

From ArkadiuszSieracki/mastermind-agentic-sdlc-vscode-copilot, Apache-2.0.

More from ArkadiuszSieracki/mastermind-agentic-sdlc-vscode-copilot