async-orchestration

Files (1)

• SKILL.md

SKILL.md (first lines)

<!-- Vendored from: platform-catalyst/skills/async-orchestration/SKILL.md (BittahCriminal/platform-catalyst, BSD-3-Clause). Adapted for Catalyst: PLAN.md/CLAUDE.md/DECISIONS.md scrubbed; ADR-008->ADR-001, ADR-009->ADR-002. -->


# Async orchestration

## Role

You guide the design and implementation of Catalyst's event-driven architecture — the message flows, queues, schedules, and state machines that connect the 8 services. You enforce async-first patterns from `AGENTS.md` and orchestration principles from *Platform Engineering for Architects* (Ch 5, pp 188-197: sustainable CI/CD, event-driven lifecycle orchestration, subscribing to events).

## Instructions

### 1. Message flow map (from `AGENTS.md` §3-4)

```
GitHub webhook → API GW → webhook-handler (Lambda)
                              │
                              ├──► SQS FIFO (pr-review queue) → pr-reviewer-consumer (Fargate)
                              └──► catalyst-api (HTTP) → GitHub Issues state machine

EventBridge rate(1 hour) → ops-intel-collector (Lambda)
                              │
                              └──► SQS fan-out → 5 probe Lambdas → S3 raw partitions
                                                                       │
                                                                       └──► ops-intel-reporter (Lambda)
                                                                            → DDB + SNS digest

Step Functions → deploy-orchestrator (Lambda)
                    │
                    ├──► ECS UpdateService (blue/green)
                    ├──► ALB ModifyRule (traffic shift)
                    ├──► DDB write (deployment record)
                    └──► SNS publish (notification)
```

### 2. SQS FIFO patterns

For the PR review queue and webhook processing:

```python
# Sending to SQS FIFO
async def enqueue_pr_review(
    sqs: SQSClient,
    queue_url: str,
    pr_event: PREvent,
) -> str:
    response = await sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=pr_event.model_dump_json(),
        MessageGroupId=f"{pr_event.repo}#{pr_event.pr_number}",
        MessageDeduplicationId=pr_event.delivery_id,
    )
    return response["MessageId"]
```

**Key design decisions:**
- `MessageGroupId` = `repo#pr_number` — serializes processing per PR (no concurrent reviews of same PR).
- `MessageDeduplicationId` = GitHub's `X-GitHub-Delivery` header — prevents duplicate processing of the same webhook.
- Visibility timeout = 6x the expected processing time (Fargate PR review ≈ 5 min → 30 min visibility).

### 3. EventBridge rules

```python
# ops-intel collector schedule (defined in Terraform, documented here for context)
# EventBridge rule: rate(1 hour)
# Target: ops-intel-collector Lambda
# Input: {"trigger": "scheduled", "domains": ["s3", "iam", "cloudwatch", "github", "inventory"]}
```

Per *Platform Engineering for Architects* Ch 5 (p191-193, analyzing events): EventBridge rules should be **specific** — one rule per trigger pattern, not a catch-all. Each rule targets exactly one Lambda with a typed input.

### 4. DLQ strategy

Every queue has a Dead Letter Queue:

| Queue | DLQ | Max receives | Action on DLQ |
|-------|-----|-------------|---------------|
| PR review FIFO | `pr-review-dlq` | 3 | Alert via SNS, create `type/incident` Issue |
| Webhook FIFO | `webhook-dlq` | 3 | Alert, log for manual replay |
| Ops-intel fan-out | `ops-intel-dlq` | 2 | Log, include in next digest as "probe failure" |

**DLQ monitoring**: CloudWatch alarm on `ApproximateNumberOfMessagesVisible > 0` triggers the Andon alert. DLQ messages are never silently discarded.

### 5. Step Functions for deploy orchestration

The deploy-orchestrator uses Step Functions for multi-step deployments:

```json
{
  "Comment": "Blue/green deploy orchestration",
  "StartAt": "ValidateDeployment",
  "States": {
    "ValidateDeployment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:deploy-orchestrator",
      "Parameters": { "action": "validate", "deployment_id.$": "$.deployment_id" },
      "Next": "UpdateECSService",
      "Catch": [{ "ErrorEquals": ["ValidationError"], "Next": "FailDeploy" }]
    },
    "UpdateECSService": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:deploy-orchestrator",
      "Parameters": { "action": "update_ecs", "deployment_id.$": "$.deployment_id" },
      "Next": "WaitForHealthy",
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RollbackDeploy" }]
    },
    "WaitForHealthy": {
      "Type": "Wait",
      "Seconds": 60,
      "Next": "ShiftTraffic"
    },
    "ShiftTraffic": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:deploy-orchestrator",
      "Parameters": { "action": "shift_traffic", "deployment_id.$": "$.deployment_id" },
      "Next": "BakeMonitor"
    },
    "BakeMonitor": {
      "Type": "Wait",
      "Seconds": 1800,
      "Next": "CompleteDeploy"

From Cloud-Byte-Consulting/Catalyst, BSD-3-Clause.

More from Cloud-Byte-Consulting/Catalyst