azure-aigateway

Files (1)

• SKILL.md

SKILL.md (first lines)

# Azure AI Gateway

Centralize LLM traffic through a controlled gateway for security, observability, policy enforcement, and cost control.

## Use This Skill When

- The user needs one entry point for multiple models (Azure OpenAI, third-party LLMs, internal models)
- The user needs to enforce token budgets, rate limits, or SLA policies
- The user needs centralized auditing and compliance logging
- The user needs intelligent fallback (model outage handling)
- The user wants to avoid client-side model management complexity

## Context: Gateway Maturity

**Immature**: Clients call models directly, no centralization  
**Developing**: Basic gateway, simple routing, minimal logging  
**Managed**: Multi-model routing, policy enforcement, token budgets, audit trails → **Target**  
**Optimized**: Real-time cost optimization, feedback-driven routing, self-healing, ML-based model selection

## Required Inputs

- **Model endpoints**: Azure OpenAI? Hugging Face? Internal models? Multiple versions?
- **Routing criteria**: Latency? Cost? Specific models for specific tasks?
- **Policy requirements**: Rate limits? Token budgets? Approval workflows?
- **Scale**: Throughput (req/sec)? Concurrent users? Peak vs. average?
- **Compliance**: PII redaction? Audit trail retention? Data residency?
- **Cost model**: Budget cap? Cost-per-department? Showback?

## Decision Tree

```
What's the primary goal of this gateway?
├─ Cost control → Route to cheaper model if accuracy acceptable
├─ Reliability → Route to alternative model on failure/timeout
├─ Governance → Enforce policies, audit all requests
├─ Performance → Route based on latency SLA
└─ Multi-tenancy → Tenant isolation, quota per tenant

Which models will the gateway route to?
├─ Single model (single Azure OpenAI deployment) → No routing needed
├─ Multiple Azure OpenAI models → Route by task/cost/latency
├─ Multiple providers (Azure OpenAI + third-party) → Handle auth differences
└─ Canary/blue-green (new model test) → Weighted routing

What policy enforcement is needed?
├─ None (simple routing only)
├─ Token budgets (cap usage per user/org)
├─ Rate limits (req/sec, concurrent, bursts)
├─ Approval workflows (route restricted requests for review)
└─ Compliance (PII redaction, output filtering, audit)

How much data volume?
├─ < 100 req/sec → Simple load balancer
├─ 100-1000 req/sec → Async processing, queue buffering
└─ > 1000 req/sec → Distributed gateway, database-backed quotas
```

## Workflow

### Phase 1: Design Gateway Architecture

```
┌─ Client Application ─┐
│  (web, mobile, API)  │
└──────────┬───────────┘
           │ (HTTPS)
           │
    ┌──────▼─────────────────────────────────────┐
    │ AI Gateway (entry point)                   │
    │                                            │
    │  1. Authenticate client (API key, JWT)    │
    │  2. Check token budget (quota system)     │
    │  3. Log request (audit trail)             │
    │  4. Validate payload (safety checks)      │
    └──────┬─────────────────────────────────────┘
           │
    ┌──────▼──────────────────┐
    │ Routing & Policy Engine │
    │                         │
    │  ├─ Route to model 1    │
    │  ├─ Route to model 2    │
    │  └─ Fallback to model 3 │
    └──────┬──────────────────┘
           │
    ┌──────▼──────────────────────────────────────────┐
    │ Model Pools                                      │
    │                                                  │
    │  [Azure OpenAI GPT-4]  → $0.03 per 1K in      │
    │  [Azure OpenAI GPT-3.5]→ $0.0015 per 1K in   │
    │  [Third-party LLaMA]   → $0.01 per 1K in     │
    │  [Custom Model (AKS)]  → On-premise            │
    └──────────────────────────────────────────────────┘
           │
    ┌──────▼──────────────────────────────────┐
    │ Observability                           │
    │                                         │
    │  ├─ Application Insights (traces)      │
    │  ├─ Log Analytics (audit logs)         │
    │  ├─ Cost Management (spend tracking)   │
    │  └─ Alerts (budget overrun, failures)  │
    └─────────────────────────────────────────┘
```

### Phase 2: Implement Gateway (Choose Stack)

**Option 1: Azure API Management + Logic Apps (low-code)**
```
Client → [API Management] → [Logic App] → [Model endpoints]

Benefits: 
- Built-in rate limiting, throttling
- Policy enforcement (transformations, auth)
- Analytics and metrics

Limitations:
- Limited model-specific logic
- Async processing harder
```

From manu14357/zskills, MIT.

More from manu14357/zskills