data-cleaning

Files (1)

• SKILL.md

SKILL.md (first lines)

# Data Cleaning

Prepares financial time series data for analysis by handling common data quality issues.

## Real Code Reference

- `tradinglearn/utils/data_fetcher.py` — `fetch_stock_data()` normalizes TDX raw data to clean DataFrame
- `tradinglearn/utils/simple_pytdx2.py` — mock data generator with controllable noise
- `tradinglearn/pytdx2/client/quotationClient.py` — raw data source (prices need scaling adjustments)

## Capabilities

- **Missing values**: forward fill (ffill), linear interpolation, drop, or flag
- **Outlier detection**: z-score, IQR, moving average deviation
- **Corporate actions**: adjust prices for splits and dividends using adjustment factors
- **Date alignment**: merge multiple series on common trading dates (not calendar dates)
- **Data quality**: volume anomalies, price gaps > N%, stale data, duplicate rows

## Typical Workflow

1. Load raw data → `pd.DataFrame` with columns: date, open, high, low, close, volume
2. Check nulls → `df.isnull().sum()`, decide fill strategy
3. Detect outliers → flag bars where `abs(zscore(returns)) > 3`
4. Adjust for splits → apply adjustment factor series
5. Align dates → reindex to intersection of all trading calendars
6. Validate → no NaNs, no suspicious jumps, no future dates

## Edge Cases

- Non-trading days: use trading calendar, not calendar days
- IPO dates: trim to actual listing date (data before is invalid)
- Suspended stocks: forward-fill last price or mark as NaN
- Pre/after-hours: decide whether to include or filter

From LisonEvf/finance-skills, MIT.

More from LisonEvf/finance-skills