数据预处理

Files (1)

• SKILL.md

SKILL.md (first lines)

# 数据预处理

## 目的

对原始数据进行系统化预处理，为建模提供清洁规整的输入数据。

## 预处理流程

### Step 1: 数据读取与理解

```python
import pandas as pd
import numpy as np
import chardet

# 自动检测编码（对CSV文件）
with open('文件.csv', 'rb') as f:
    result = chardet.detect(f.read())
    print(f"检测编码: {result['encoding']}")

# 读取数据
df = pd.read_excel('数据/data.xlsx', sheet_name='Sheet1')

# 基础信息
print(f"数据规模: {df.shape}")
print(f"列名: {list(df.columns)}")
print(f"数据类型:\n{df.dtypes}")
print(f"缺失值:\n{df.isnull().sum()}")
print(f"描述统计:\n{df.describe()}")
```

### Step 2: 数据清洗

#### 2.1 缺失值处理

```python
# 统计缺失情况
missing_ratio = df.isnull().sum() / len(df)
print("缺失比例:\n", missing_ratio)

# 策略选择
for col in df.columns:
    ratio = df[col].isnull().sum() / len(df)
    if ratio == 0:
        continue
    elif ratio < 0.05:  # <5%: 直接删除缺失行
        df = df.dropna(subset=[col])
        print(f"  [{col}] 缺失{ratio:.1%}，删除缺失行")
    elif ratio < 0.20:  # 5%-20%: 插补
        if df[col].dtype in ['float64', 'int64']:
            df[col] = df[col].fillna(df[col].median())  # 数值列用中位数
        else:
            df[col] = df[col].fillna(df[col].mode()[0])  # 分类列用众数
        print(f"  [{col}] 缺失{ratio:.1%}，已插补")
    else:  # >20%: 考虑删除该列
        print(f"  ⚠ [{col}] 缺失{ratio:.1%}，比例过高，建议删除或特殊处理")
```

#### 2.2 异常值检测

```python
def detect_outliers(df, method='iqr'):
    """检测异常值"""
    outliers = {}
    for col in df.select_dtypes(include=[np.number]).columns:
        if method == 'iqr':
            Q1 = df[col].quantile(0.25)
            Q3 = df[col].quantile(0.75)
            IQR = Q3 - Q1
            lower = Q1 - 1.5 * IQR
            upper = Q3 + 1.5 * IQR
            mask = (df[col] < lower) | (df[col] > upper)
        elif method == 'zscore':
            from scipy import stats
            z = np.abs(stats.zscore(df[col].dropna()))
            mask = pd.Series(z > 3, index=df[col].dropna().index)

        if mask.sum() > 0:
            outliers[col] = {
                'count': mask.sum(),
                'ratio': mask.sum() / len(df),
                'indices': df[mask].index.tolist()
            }
    return outliers

outliers = detect_outliers(df, method='iqr')
for col, info in outliers.items():
    print(f"  [{col}] 异常值 {info['count']} 个 ({info['ratio']:.1%})")

# 处理异常值：使用中位数替换
for col in outliers:
    median = df[col].median()
    df.loc[df.index.isin(outliers[col]['indices']), col] = median
```

#### 2.3 重复值处理

```python
duplicates = df.duplicated().sum()
print(f"重复行: {duplicates}")
if duplicates > 0:
    df = df.drop_duplicates()
```

### Step 3: 标准化/归一化

#### Min-Max 归一化

\begin{equation}
x' = \frac{x - x_{min}}{x_{max} - x_{min}}
\end{equation}

```python
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols] = scaler.fit_transform(df[numeric_cols])

From hongdotien10-hash/mathmodel-toolkit, MIT.

More from hongdotien10-hash/mathmodel-toolkit