processing-s3-uploads-with-step-functions

Files (12)

• SKILL.md
• ecs-task-definition.md · references/
• iam-roles.md · references/
• Dockerfile · scripts/
• ecs-trust-policy.json · scripts/
• eventbridge-trust-policy.json · scripts/
• fargate_processor.py · scripts/
• lambda-trust-policy.json · scripts/
• lambda_function.py · scripts/
• requirements.txt · scripts/
• statemachine.asl.json · scripts/
• stepfunctions-trust-policy.json · scripts/

SKILL.md (first lines)

# Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

## Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to
an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine
checks the file size and routes processing to either a Lambda function (files ≤ 6 MB)
or a Fargate task (files > 6 MB).

The architecture includes:

- An S3 bucket with EventBridge notifications enabled
- An EventBridge rule that triggers Step Functions on S3 object creation
- A Step Functions state machine with a Choice state for routing
- A Lambda function for processing small files
- An ECS Fargate task for processing large files
- A VPC with two subnets, internet gateway, and security group
- An ECR repository for the Fargate container image
- Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

- You need to process S3 uploads with different compute based on file size
- You want a serverless workflow that can handle both small and large files
- You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

- All files are small enough for Lambda (use S3 → Lambda directly)
- You need real-time streaming (use Kinesis)
- You don't need file-size-based routing

## Prerequisites

1. **AWS CLI v2** — Installed and configured. Verify with `aws sts get-caller-identity`.
2. **Python 3.12** — For the Lambda function runtime.
3. **Docker** — For building and pushing the Fargate container image.

## Parameters

- bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
- region (required): AWS region for all resources
- ecr_repo_name (required): Name for the ECR repository
- state_machine_name (required): Name for the Step Functions state machine
- kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with `aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}`

Constraints for parameter acquisition:

- You MUST ask for all required parameters upfront in a single prompt
- You MUST support multiple input methods (direct input, file path, URL)
- You MUST confirm successful acquisition of all parameters before proceeding
- You MUST validate that bucket_name follows S3 naming rules

## Procedures

### Step 0: Verify Dependencies

Constraints:

- You MUST verify the following tools are available: aws-cli, python3 (3.12+), docker
- You MUST inform the user about any missing tools with a clear message
- You MUST ask if the user wants to proceed despite missing tools
- You MUST respect the customer's decision to abort at any point
- You MUST explain to the customer what step is being executed, why, and which tool is being called

### Step 1: Retrieve AWS Account ID

Constraints:

- You MUST retrieve the account ID with: `aws sts get-caller-identity --query 'Account' --output text`
- You MUST store the result as {account_id} for use in all subsequent steps
- You MUST abort if credentials are not configured

### Step 2: Get the Default VPC and Networking

Constraints:

- You MUST retrieve the default VPC ID with:
  `aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text --region {region}`
- If no default VPC exists, inform the user they must create one with `aws ec2 create-default-vpc --region {region}` or provide a VPC ID manually
- You MUST retrieve two subnet IDs from the default VPC:
  `aws ec2 describe-subnets --filters Name=vpc-id,Values={vpc_id} --query 'Subnets[0:2].SubnetId' --output text --region {region}`
- You MUST create a security group in the default VPC:
  `aws ec2 create-security-group --group-name fargate-sg --description "Security group for Fargate tasks" --vpc-id {vpc_id} --region {region}`
- You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow-all egress rule:
  `aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' --region {region}`
  Then add scoped rules:
  `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {region}` and
  `aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0 --region {region}`
- You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet-routed traffic and eliminate the need for broad egress rules
- You MUST capture {vpc_id}, {subnet1_id}, {subnet2_id}, and {sg_id} for use in later steps

### Step 3: Create the ECR Repository

Constraints:

- You MUST create the repository with:
  `aws ecr create-repository --repository-name {ecr_repo_name} --region {region}`
- You MUST capture the repositoryUri from the response

### Step 4: Build and Push the Container Image

Constraints:

- You MUST verify Docker is installed by running `docker --version`. If Docker is not installed, instruct the user to install it from https://docs.docker.com/get-docker/ and abort until it is available
- You MUST authenticate Docker with ECR:
  `aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com`
- The Dockerfile and processor code are in `scripts/Dockerfile` and `scripts/fargate_processor.py`
- You MUST build and push the image from the scripts directory:

  ```
  cd scripts
  docker build --platform linux/amd64 -t {ecr_repo_name} .
  docker tag {ecr_repo_name}:latest {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  docker push {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  cd ..
  ```

### Step 5: Create IAM Roles

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws