ai-governance

Files (1)

• SKILL.md

SKILL.md (first lines)

# AI Governance Audit

Systematic trust and control assessment for AI/ML systems. Maps the AI system boundary, evaluates risk controls, and identifies gaps against governance frameworks (EU AI Act, NIST AI RMF, ISO 42001).

## Usage

```
/ai-governance                   # full audit of the current project
/ai-governance <path>            # audit a specific ML service or module
/ai-governance --framework eu    # focus on EU AI Act risk classification
/ai-governance --framework nist  # focus on NIST AI RMF (Govern/Map/Measure/Manage)
```

## Behavior

### Step 1 — Map the AI system boundary

```bash
# Identify model artifacts, training scripts, inference code
find . -name "*.pkl" -o -name "*.pt" -o -name "*.onnx" -o -name "model.*" 2>/dev/null | head -10
grep -rn "model.predict\|model.generate\|pipeline\|inference" --include="*.py" -l . | head -10

# Find dataset references
grep -rn "pd.read_csv\|load_dataset\|dataloader\|train_data" --include="*.py" -l . | head -10
```

Document:
- What decisions does the model make? (classification, ranking, generation, recommendation)
- Who is affected? (internal tooling vs customer-facing vs high-stakes)
- What data does it consume?

### Step 2 — Risk classification

Classify the system under the EU AI Act tiers:

| Risk tier | Criteria | Examples |
|---|---|---|
| **Unacceptable** | Social scoring, real-time biometric surveillance | Flag for legal review immediately |
| **High** | Employment, credit, health, safety, law enforcement | Full compliance controls required |
| **Limited** | Chatbots, emotion recognition disclosure | Transparency obligations only |
| **Minimal** | Spam filters, game AI, recommenders | Voluntary codes of practice |

Flag the applicable tier and list required controls.

### Step 3 — Evaluate bias and fairness controls

```bash
# Look for fairness tooling
grep -rn "fairlearn\|aif360\|what-if-tool\|demographic\|protected\|sensitive_attr\|equalized_odds" \
  --include="*.py" -l . 2>/dev/null

# Check if sensitive attributes are in training data
grep -rn "gender\|race\|ethnicity\|age\|religion\|nationality" \
  --include="*.py" --include="*.csv" --include="*.json" -l . 2>/dev/null | head -10
```

Check for:
- Fairness metric definition (demographic parity, equalized odds, calibration)
- Bias testing in CI pipeline
- Disparate impact analysis on test set
- Sensitive attribute handling (removed, proxied, or legitimately included)

### Step 4 — Evaluate explainability

```bash
grep -rn "shap\|lime\|captum\|eli5\|anchors\|explanation\|interpret" --include="*.py" -l . 2>/dev/null
```

Check:
- Is there a method to explain individual predictions?
- Are explanations surfaced to affected users or decision-makers?
- Is there a global feature importance report?
- For LLMs: are citations / sources provided for generated claims?

### Step 5 — Evaluate model risk management

Check for model risk controls:

| Control | What to look for | Gap indicator |
|---|---|---|
| Model card | `model_card.md`, `MODEL_CARD.md`, `README` with model description | No documentation of intended use, limitations, training data |
| Version tracking | Model version in filename, MLflow, DVC, W&B | Model artifact with no version |
| Performance monitoring | Drift detection, accuracy tracking in production | No monitoring post-deployment |
| Human-in-the-loop | Approval step for high-stakes decisions | Fully automated decisions in high-risk domain |
| Rollback plan | Previous model artifact retained | Only latest model available |
| Incident process | Runbook for model failure | No defined response to model misbehavior |

### Step 6 — Evaluate data governance

```bash
# Check for data lineage documentation
find . -name "data_catalog*" -o -name "lineage*" -o -name "data_dictionary*" 2>/dev/null
grep -rn "consent\|gdpr\|pii\|personally_identifiable\|anonymize\|pseudonymize" --include="*.py" -l . 2>/dev/null
```

Check:
- Is training data provenance documented?
- Is PII in training data identified and handled?
- Is there a data retention / deletion policy?
- Can a subject request their data be removed from training?

### Step 7 — Report findings

```
AI GOVERNANCE AUDIT — <system/path>

Risk Classification
  EU AI Act tier: LIMITED (chatbot with emotion recognition disclosure required)
  NIST AI RMF: Map phase incomplete — no impact assessment documented

Bias & Fairness (2 gaps)
  No fairness metrics defined — demographic parity not measured on test set
  gender column present in training features — document why or remove

Explainability (1 gap)
  No explanation method found — users cannot understand why they received a recommendation

Model Risk Management (3 gaps)
  No model card — intended use, limitations, and training data undocumented

From manastalukdar/ai-devstudio, MIT.

More from manastalukdar/ai-devstudio