afrexai-ml-engineering

Files (1)

• SKILL.md

SKILL.md (first lines)

# ML & AI Engineering System

Complete methodology for building, deploying, and operating production ML/AI systems — from experiment to scale.

---

## Phase 1: Problem Framing

Before writing any code, define the ML problem precisely.

### ML Problem Brief

```yaml
problem_brief:
  business_objective: ""          # What business metric improves?
  success_metric: ""              # Quantified target (e.g., "reduce churn 15%")
  baseline: ""                    # Current performance without ML
  ml_task_type: ""                # classification | regression | ranking | generation | clustering | anomaly_detection | recommendation
  prediction_target: ""           # What exactly are we predicting?
  prediction_consumer: ""         # Who/what uses the prediction? (API | dashboard | email | automated action)
  latency_requirement: ""         # real-time (<100ms) | near-real-time (<1s) | batch (minutes-hours)
  data_available: ""              # What data exists today?
  data_gaps: ""                   # What's missing?
  ethical_considerations: ""      # Bias risks, fairness requirements, privacy
  kill_criteria:                  # When to abandon the ML approach
    - "Baseline heuristic achieves >90% of ML performance"
    - "Data quality too poor after 2 weeks of cleaning"
    - "Model can't beat random by >10% on holdout set"
```

### ML vs Rules Decision

| Signal | Use Rules | Use ML |
|--------|-----------|--------|
| Logic is explainable in <10 rules | ✅ | ❌ |
| Pattern is too complex for humans | ❌ | ✅ |
| Training data >1,000 labeled examples | — | ✅ |
| Needs to adapt to new patterns | ❌ | ✅ |
| Must be 100% auditable/deterministic | ✅ | ❌ |
| Pattern changes faster than you can update rules | ❌ | ✅ |

**Rule of thumb:** Start with rules/heuristics. Only add ML when rules fail to capture the pattern.

---

## Phase 2: Data Engineering for ML

### Data Quality Assessment

Score each data source (0-5 per dimension):

| Dimension | 0 (Terrible) | 5 (Excellent) |
|-----------|--------------|----------------|
| **Completeness** | >50% missing | <1% missing |
| **Accuracy** | Known errors, no validation | Validated against source of truth |
| **Consistency** | Different formats, duplicates | Standardized, deduplicated |
| **Timeliness** | Months stale | Real-time or daily refresh |
| **Relevance** | Weak proxy for target | Direct signal for prediction |
| **Volume** | <100 samples | >10,000 samples per class |

**Minimum score to proceed:** 18/30. Below 18 → fix data first, don't build models.

### Feature Engineering Patterns

```yaml
feature_types:
  numerical:
    - raw_value           # Use as-is if normally distributed
    - log_transform       # Right-skewed distributions (revenue, counts)
    - standardize         # z-score for algorithms sensitive to scale (SVM, KNN, neural nets)
    - bin_to_categorical  # When relationship is non-linear and data is limited
  categorical:
    - one_hot             # <20 categories, tree-based models handle natively
    - target_encoding     # High-cardinality (>20 categories), use with K-fold to prevent leakage
    - embedding           # Very high-cardinality (user IDs, product IDs) with deep learning
  temporal:
    - lag_features        # Value at t-1, t-7, t-30
    - rolling_statistics  # Mean, std, min, max over windows
    - time_since_event    # Days since last purchase, hours since login
    - cyclical_encoding   # sin/cos for hour-of-day, day-of-week, month
  text:
    - tfidf               # Simple, interpretable, good baseline
    - sentence_embeddings # semantic similarity, modern NLP
    - llm_extraction      # Use LLM to extract structured fields from unstructured text
  interaction:
    - ratios              # Feature A / Feature B (e.g., clicks/impressions = CTR)
    - differences         # Feature A - Feature B (e.g., price - competitor_price)
    - polynomial          # A * B, A^2 (use sparingly, high-cardinality features)
```

### Feature Store Design

```yaml
feature_store:
  offline_store:         # For training — batch computed, stored in data warehouse
    storage: "BigQuery | Snowflake | S3+Parquet"
    compute: "Spark | dbt | SQL"
    refresh: "daily | hourly"
  online_store:          # For serving — low-latency lookups
    storage: "Redis | DynamoDB | Feast online"
    latency_target: "<10ms p99"
    refresh: "streaming | near-real-time"
  registry:              # Feature metadata
    naming: "{entity}_{feature_name}_{window}_{aggregation}"  # e.g., user_purchase_count_30d_sum
    documentation:
      - description
      - data_type
      - source_table
      - owner
      - created_date
      - known_issues
```

### Data Leakage Prevention Checklist

- [ ] No future information in features (time-travel check)
- [ ] Train/val/test split done BEFORE feature engineering
- [ ] Target encoding uses only training fold statistics
- [ ] No features derived from the target variable
- [ ] Temporal splits for time-series (no random shuffle)

From xdelin/OpenClaw-Education-Skills, MIT.

More from xdelin/OpenClaw-Education-Skills