troubleshooting-s3-files

Files (1)

• SKILL.md

SKILL.md (first lines)

# Troubleshooting S3 Files

## Overview

Diagnoses and resolves Amazon S3 Files issues: mount failures, IAM
permissions, synchronization, conflict resolution, and performance.

For authoritative guidance, see [S3 Files Troubleshooting](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files-troubleshooting.html).

## Common Tasks

### 0. Verify Dependencies

- You MUST verify `aws` CLI is available with `s3files` subcommand support
- You MUST confirm valid AWS credentials
- You MUST ONLY check for tool existence and version — MUST NOT execute destructive or mutating commands during verification
- You MUST inform the user if any required tools are missing
- You MUST respect the user's decision to abort if tools are unavailable
- You SHOULD explain steps before executing and wait for user confirmation on write commands

### 1. Classify the Issue

| Symptom | Category |
|---|---|
| mount.s3files: command not found | A: Client Installation |
| Connection timed out during mount | B: Network/Security Group |
| Mount hangs indefinitely (no timeout) | B: Network/Security Group |
| Access denied during mount | C: IAM Permissions |
| File system stuck in "creating" | C: IAM Permissions |
| Permission denied on file operations | C: IAM Permissions |
| Files not appearing in S3 after write | D: Synchronization |
| Files in .s3files-lost+found directory | E: Conflict Resolution |
| Slow reads or high latency | F: Performance |
| NFS server error | G: Encryption/KMS |
| DNS name resolution fails | H: VPC DNS |

### 2. Category A — Client Installation

`mount.s3files: command not found` means `amazon-efs-utils` is missing or < v3.0.0.

```bash
sudo yum -y install amazon-efs-utils  # Amazon Linux
```

### 3. Category B — Network/Security Group

Connection timeout is the #1 mount failure — almost always security groups.

Verify mount target exists in the instance's AZ:

```bash
aws s3files list-mount-targets --file-system-id fs-ID --region REGION
```

Cross-AZ mounting works but adds latency.

Verify security groups — most common fix:

- Mount target SG MUST have inbound TCP 2049 from compute SG
- Compute SG MUST have outbound TCP 2049 to mount target SG
- Fix: `aws ec2 authorize-security-group-ingress --group-id sg-MT --protocol tcp --port 2049 --source-group sg-COMPUTE`

Test connectivity:

```bash
nc -zv az-ID.fs-ID.s3files.REGION.on.aws 2049
```

> **Note:** These SG troubleshooting steps also apply to EFS — use `aws efs describe-mount-targets` instead.

**Mount hangs in isolated VPC**: If the VPC has no internet access, S3 Files requires a CloudWatch Logs VPC endpoint (`com.amazonaws.REGION.logs`) for mount to complete.

### 4. Category C — IAM Permissions

**File system stuck in "creating" status:**
S3 Files does NOT validate IAM role permissions at creation time. Wrong trust policy or missing permissions → stuck in `creating` with access denied in `statusMessage`.

Check status:

```bash
aws s3files get-file-system --file-system-id fs-ID --region REGION
```

Check `statusMessage`. If access denied, fix the IAM role and delete/recreate.

**Mount access denied:** Compute role needs `s3files:ClientMount`. For dev/test only, `AmazonS3FilesClientFullAccess` is acceptable — avoid in production.

**Write permission denied:** Compute role needs `s3files:ClientWrite`

**Root access denied:** Compute role needs `s3files:ClientRootAccess`. ⚠️ Bypasses POSIX permissions — prefer access points with scoped POSIX users.

**Check file system policy:**

```bash
aws s3files get-file-system-policy --file-system-id fs-ID --region REGION
```

### 5. Category D — Synchronization

**Files not appearing in S3:** Writes sync within ~60 seconds. Check status:

```bash
getfattr -n "user.s3files.status;$(date -u +%s)" filename --only-values
```

Common ExportError values:

| Error | Fix |
|---|---|
| S3AccessDenied | File system IAM role lacks S3 write permissions |
| S3BucketNotFound | Bucket deleted or renamed |
| RoleAssumptionFailed | Trust policy misconfigured |
| EncryptionKeyInaccessible | KMS key disabled or permissions revoked |
| PathTooLong | File path exceeds 1,024 byte S3 key limit |

Monitor: `PendingExports` CloudWatch metric. Growing = exceeds 800 files/sec rate.

### 6. Category E — Conflict Resolution

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws