backend-bulk-import

Files (9)

• SKILL.md
• bulk-export.md · references/
• bulk-import-advanced.md · references/
• bulk-import-fundamentals.md · references/
• csv-parsing.md · references/
• import-monitoring.md · references/
• import-workflow.md · references/
• rollback-recovery.md · references/
• validation-pipeline.md · references/

SKILL.md (first lines)

# Bulk Import Skill

## Purpose
Design robust bulk import systems that handle large CSV/Excel files with validation, progress tracking, error recovery, and audit trails.

## Architecture Decision Trees

### Import Mode Selection

| Criterion | Insert | Upsert | Replace |
|-----------|--------|--------|---------|
| Duplicate handling | Fails on conflict | Updates existing | Truncate all first |
| Performance | Fastest | Moderate (check per row) | Fast (truncate + insert) |
| Idempotent | No | Yes (by dedup field) | No (destructive) |
| Risk level | Low | Low | High (data loss) |
| Audit trail | All inserts | Updates logged | Lost on truncate |
| Rollback complexity | Simple (transaction) | Moderate | Complex (needs backup) |
| Use case | New data ingestion | Sync with external system | Full reimport/replace |

Decision: Upsert for production syncs. Insert for immutable audit data. Replace only with pre-import backup.

### Parsing Strategy

| Criterion | Streaming (CSV) | Full load (Excel) | Hybrid (chunked) |
|-----------|----------------|-------------------|------------------|
| Memory | O(1) rows in memory | Full file in memory | O(batch size) |
| Max file size | Unlimited | ~100MB practical | ~500MB |
| Row validation | Per-batch | All before processing | Per-batch |
| Error collection | Per-batch | Full before process | Per-batch |
| Progress tracking | Yes (real-time) | No (must parse first) | Yes (per chunk) |
| Random access | No | Yes | No |

Decision: Streaming for CSV. Chunked for Excel > 10MB. Full load for Excel < 10MB.

### Deduplication Strategy

| Approach | Precision | Performance | Implementation |
|----------|-----------|-------------|----------------|
| DB unique constraint | Exact | Fast (index) | Schema definition |
| Pre-check with SELECT | Exact | Slow on large tables | Query existing values |
| Row content hash (SHA-256 of normalized row, stored in a unique column) | Exact on normalized content | Fast (index) | Hash + unique index |
| External dedup (Redis set) | Exact within the set's lifetime | Very fast | `SADD` returns 0 for duplicates |
| Bloom filter (RedisBloom `BF.ADD`) | Probabilistic: false positives possible | Very fast, small memory | Only where wrongly skipping a few unique rows is acceptable |

Decision: DB unique constraint (with `ON CONFLICT`) for exact dedup. Bloom filters only as a pre-filter, never as the final decision for business data.

## Agent Protocol

### Trigger
User mentions bulk import, CSV upload, Excel import, data ingestion, batch processing, data migration, import pipeline, file upload processing, import validation, or ETL pipeline.

### Input Context
- Source file format (CSV, Excel, JSON)
- Expected file size and row count
- Data schema and validation rules
- Import mode (insert, upsert, replace)
- Deduplication strategy
- Error handling requirements
- Background processing needs
- Target system and database

### Output Artifact
SKILL.md adherence document plus implemented import pipeline, validation logic, progress tracking, error handling, and rollback mechanisms.

### Response Format
No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output — why use many token when few do trick.

### Completion Criteria
- [ ] File upload endpoint with size/type validation implemented
- [ ] CSV/Excel parser configured with header mapping and encoding detection
- [ ] Validation pipeline with row-level error collection operational
- [ ] Preview step (show sample + errors before confirm) implemented
- [ ] Batch processing with progress tracking and throttling
- [ ] Import template with downloadable sample file ready
- [ ] Deduplication logic implemented (configurable per field)
- [ ] Rollback mechanism for partial failures
- [ ] Import history and audit log stored
- [ ] Webhook notification on import completion
- [ ] Rate limiting on import endpoints

### Max Response Length
4096 tokens

## Workflow

1. **File Upload & Validation**: Accept upload with strict type/size checks.

```typescript
import multer from 'multer';
import { randomUUID } from 'node:crypto';

const ALLOWED_MIME = [
  'text/csv',
  'application/vnd.ms-excel',
  'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
];
const MAX_SIZE = 50 * 1024 * 1024; // 50MB

// Disk (or object storage) instead of memory: the worker streams the file later.
const upload = multer({
  storage: multer.diskStorage({
    destination: process.env.IMPORT_DIR ?? '/var/app/imports',
    filename: (_req, _file, cb) => cb(null, randomUUID()), // never trust client file names
  }),
  limits: { fileSize: MAX_SIZE },
  fileFilter: (_req, file, cb) => {
    // mimetype is client-supplied: treat as a hint and re-validate content when parsing.
    if (!ALLOWED_MIME.includes(file.mimetype)) {
      cb(new Error('Unsupported file type. Only CSV and Excel files are allowed.'));
      return;
    }
    cb(null, true);
  },
});

app.post('/api/imports/upload', upload.single('file'), async (req, res) => {
  const importJob = await ImportJob.create({
    userId: req.user.id,
    fileName: req.file.originalname,

From j4flmao/agent-skills, MIT.

More from j4flmao/agent-skills