data-distributed-storage

Files (9)

• LICENSE
• SKILL.md
• local.patch
• distributed-storage-advanced.md · references/
• distributed-storage-fundamentals.md · references/
• hdfs-architecture.md · references/
• s3-compatible-configs.md · references/
• s3-compatible.md · references/
• storage-deployment-guide.md · references/

SKILL.md (first lines)

# Distributed Storage

## Purpose
Design distributed storage systems for data workloads. Select the right storage backend (HDFS, S3, ADLS, GCS, MinIO), configure durability and replication, optimize file formats and compression, and implement lifecycle policies.

## Agent Protocol

### Trigger
Exact user phrases: "HDFS", "S3", "ADLS", "GCS", "MinIO", "distributed storage", "object storage", "file system", "data lake storage", "storage tier", "lifecycle policy", "replication factor", "erasure coding", "storage cost".

### Input Context
- Storage backend preference (S3, ADLS, GCS, HDFS, MinIO)
- Data volume (current TB, annual growth)
- Access patterns (frequent, infrequent, archive)
- Durability and availability requirements
- Budget and cost constraints
- Compliance requirements (data residency, encryption)
- Network bandwidth and latency constraints

### Output Artifact
Distributed storage architecture with backend selection, tiering strategy, lifecycle policy, and cost model.

### Response Format
```yaml
# Storage backend
# Tier configuration
# Lifecycle rules
# Cost projection
```

### Completion Criteria
- [ ] Storage backend selected with rationale
- [ ] Storage tiers defined (hot/warm/cold/archive)
- [ ] Lifecycle policies configured for data movement
- [ ] Data durability and replication strategy documented
- [ ] Encryption (at rest, in transit) configured
- [ ] Cost projection for current and 3-year growth
- [ ] Backup and disaster recovery plan defined

## Workflow

### Step 1: Backend Selection

#### Object Store Comparison

| Feature | AWS S3 | ADLS Gen2 | GCS | MinIO |
|---|---|---|---|---|
| Consistency | Read-after-write | Strong | Strong | Strong |
| Auth | IAM roles, bucket policies | RBAC, SAS tokens | IAM, service accounts | JWT, OIDC, LDAP |
| Encryption | SSE-S3/KMS/CSE | SSE-AES/KMS/CMK | Google-managed/CMEK/CSE | KMS, auto-encryption |
| Lifecycle | Transition, expiry, versioning | Tiering, soft-delete | Nearline/Coldline/Archive | Bucket lifecycle |
| Max object size | 5 TB | 4.75 TB | 5 TB | 100 TB (configurable) |
| S3 Compatible | Native | Yes (via gateway) | Yes (XML API) | Native |
| Cost (per TB/mo) | ~$23 (standard) | ~$20 (hot) | ~$20 (standard) | Hardware + ops |
| Durability | 99.999999999% (11 9s) | 99.999999999% (11 9s) | 99.999999999% (11 9s) | Configurable |
| Availability SLA | 99.99% | 99.9% | 99.95% | Depends on setup |

#### Backend Selection Decision Tree
```
Cloud provider?
├── AWS → S3 (best integration with AWS ecosystem)
├── Azure → ADLS Gen2 (best with Azure AD, Active Directory)
├── GCP → GCS (strong consistency, best for Google ecosystem)
├── Multi-cloud → MinIO (S3-compatible layer across clouds)
└── On-premise / air-gapped → MinIO or HDFS

Primary workload?
├── Data lake with compute engines → Object store (S3/ADLS/GCS)
├── Legacy Hadoop/Spark without cloud → HDFS
├── Private cloud, edge, or air-gapped → MinIO
└── High-performance computing → Parallel file system (Lustre, GPFS)
```

### Step 2: Storage Tiering

#### Tier Characteristics

| Tier | Storage Class | Latency | Cost/TB/mo | Use Case |
|---|---|---|---|---|
| Hot | S3 Standard, ADLS Hot | Millisecond | ~$23 | Active data, daily pipelines, ML training |
| Warm | S3 Infrequent Access | Millisecond | ~$12.50 | Monthly queries, staging data |
| Cold | S3 Glacier Instant | Millisecond (instant) | ~$4 | Quarterly access, compliance retention |
| Archive | S3 Glacier Deep Archive | 12 hours | ~$1 | Annual access, legal holds |

#### Tiering Strategy
Hot: last 30-90 days of data. Warm: 90 days to 1 year. Cold: 1-3 years. Archive: 3-7+ years (retention-based). Intelligent tiering (S3) auto-moves objects between hot and warm based on access patterns.

### Step 3: Lifecycle Policies

```json
{
  "Rules": [
    {
      "Id": "TierAndExpire",
      "Status": "Enabled",
      "Filter": { "Prefix": "logs/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" },
        { "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
      ],
      "Expiration": { "Days": 2555 }
    },
    {
      "Id": "ExpireTempFiles",
      "Status": "Enabled",
      "Filter": { "Prefix": "tmp/" },
      "Expiration": { "Days": 7 }
    }
  ]
}
```

### Step 4: Durability and Replication

#### Object Store Durability
S3/ADLS/GCS provide 11 9s of durability via erasure coding across multiple devices/facilities. Replication is handled by the cloud provider. For additional protection: cross-region replication (CRR) for disaster recovery, same-region replication (SRR) for compliance.

#### HDFS Replication

From Kilo-Org/kilo-marketplace, Apache-2.0.

More from Kilo-Org/kilo-marketplace