memory-optimization

Files (2)

• SKILL.md
• advanced_techniques.md · references/

SKILL.md (first lines)

# Memory Optimization Skill

Transform Python code to minimize memory usage while maintaining functionality.

## Workflow

1. **Profile** to identify memory bottlenecks (largest allocations, leak patterns)
2. **Analyze** data structures and object lifecycles
3. **Select** optimization strategies based on access patterns
4. **Transform** code with memory-efficient alternatives
5. **Verify** memory reduction without correctness loss

## Memory Optimization Decision Tree

```
What's consuming memory?

Large collections:
├── List of objects → __slots__, namedtuple, or dataclass(slots=True)
├── List built all at once → Generator/iterator pattern
├── Storing strings → String interning, categorical encoding
└── Numeric data → NumPy arrays instead of lists

Data processing:
├── Loading full file → Chunked reading, memory-mapped files
├── Intermediate copies → In-place operations, views
├── Keeping processed data → Process-and-discard pattern
└── DataFrame operations → Downcast dtypes, sparse arrays

Object lifecycle:
├── Objects never freed → Check circular refs, use weakref
├── Cache growing unbounded → LRU cache with maxsize
├── Global accumulation → Explicit cleanup, context managers
└── Large temporary objects → Delete explicitly, gc.collect()
```

## Transformation Patterns

### Pattern 1: Class to __slots__

Reduces per-instance memory by 40-60%:

**Before:**
```python
class Point:
    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z
```

**After:**
```python
class Point:
    __slots__ = ('x', 'y', 'z')

    def __init__(self, x, y, z):
        self.x = x
        self.y = y
        self.z = z
```

### Pattern 2: List to Generator

Avoid materializing entire sequences:

**Before:**
```python
def get_all_records(files):
    records = []
    for f in files:
        records.extend(parse_file(f))
    return records

all_data = get_all_records(files)
for record in all_data:
    process(record)
```

**After:**
```python
def get_all_records(files):
    for f in files:
        yield from parse_file(f)

for record in get_all_records(files):
    process(record)
```

### Pattern 3: Downcast Numeric Types

Reduce NumPy/Pandas memory by 2-8x:

**Before:**
```python
df = pd.read_csv('data.csv')  # Default int64, float64
```

**After:**
```python
def optimize_dtypes(df):
    for col in df.select_dtypes(include=['int']):
        df[col] = pd.to_numeric(df[col], downcast='integer')
    for col in df.select_dtypes(include=['float']):
        df[col] = pd.to_numeric(df[col], downcast='float')
    return df

df = optimize_dtypes(pd.read_csv('data.csv'))
```

### Pattern 4: String Deduplication

For repeated strings:

**Before:**
```python
records = [{'status': 'active', 'type': 'user'} for _ in range(1000000)]
```

From benchflow-ai/skillsbench, Apache-2.0.

More from benchflow-ai/skillsbench