llm-finetuning

Files (1)

• SKILL.md

SKILL.md (first lines)

# LLM Fine-Tuning (PEFT / LoRA)

## Purpose
Fine-tuning changes model behaviour (format, style, task skill, domain language) from examples. It is the wrong tool for injecting frequently changing facts; use retrieval for that. Parameter-efficient methods such as LoRA train small low-rank adapters on frozen base weights, making single-GPU training of 7-8B models routine and letting one base model serve many adapters.

## Agent Protocol

### Trigger
"fine-tune", "LoRA", "QLoRA", "PEFT", "SFT", "instruction tuning", "DPO", "preference tuning", "adapter", "merge_and_unload".

### Input Context
- Goal and why prompting/RAG is insufficient (with failing examples).
- Dataset: size, source, license, format, PII status.
- Base model and license; GPU type/VRAM; serving target.
- Eval set and success metric.

### Output Artifact
Go/no-go decision, dataset spec, training config, eval plan, export/serving instructions.

### Response Format
```
Decision: {fine-tune|RAG|prompting} because {reason}
Base model: {id} (license {x})  Method: {LoRA|QLoRA|full}  Stage: {SFT|DPO}
Data: {n} examples, chat format, split {train/val/test}
LoRA: r={r} alpha={a} dropout={p} targets={modules}
Train: lr={lr} epochs={e} eff_batch={b} max_len={L}
Eval: base vs tuned on {metric}; regression suite {name}
Export: {adapter|merged} -> {engine}
```

### Completion Criteria
- [ ] Baseline (prompted base model, RAG) measured first.
- [ ] Held-out test set never used for tuning decisions.
- [ ] Chat template of the base model applied consistently in training and inference.
- [ ] Tuned model beats baseline on the target metric without regressions on general checks.

## Should You Fine-Tune?
| Goal | Best tool |
|---|---|
| New/changing facts, citations | RAG |
| Output format, tone, strict schema | Prompting + structured outputs first; SFT if still unreliable |
| Narrow task at lower cost/latency | SFT a smaller model (often distilled from a larger one) |
| Preference (helpfulness, style ranking) | DPO/ORPO after SFT |
| Domain vocabulary/style | SFT (or continued pre-training for large corpora) |

## Dataset
- Use the conversational format and let the tokenizer's chat template render it:
  ```json
  {"messages": [{"role": "system", "content": "You extract invoice fields as JSON."},
                {"role": "user", "content": "<invoice text>"},
                {"role": "assistant", "content": "{\"total\": 129.5, \"currency\": \"EUR\"}"}]}
  ```
- Quality over quantity: a few hundred to a few thousand clean, diverse examples often suffice for format/task tuning.
- Deduplicate (exact + near-duplicate), remove PII/secrets, check licenses, balance classes, and keep a stratified held-out test split.
- Train on assistant tokens only (loss masking of prompts) unless you have a reason not to; TRL supports this via its completion/assistant-only loss options.

## LoRA / QLoRA Training (PEFT + TRL)

```python
from __future__ import annotations

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

BASE = "meta-llama/Llama-3.1-8B-Instruct"
USE_QLORA = True

tokenizer = AutoTokenizer.from_pretrained(BASE)
quant = BitsAndBytesConfig(
    load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True,
) if USE_QLORA else None

model = AutoModelForCausalLM.from_pretrained(
    BASE,
    dtype=torch.bfloat16,          # transformers >= 4.56; older versions use torch_dtype=
    quantization_config=quant,
    device_map="auto",
)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules="all-linear",   # attention + MLP projections; better than q/v only
    task_type="CAUSAL_LM",
)

data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

args = SFTConfig(
    output_dir="out/invoice-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,       # effective batch 16
    learning_rate=2e-4,                  # LoRA tolerates higher LR than full FT (~1e-5)
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
    report_to="none",
)

trainer = SFTTrainer(
    model=model, args=args, processing_class=tokenizer,
    train_dataset=data["train"], eval_dataset=data["validation"], peft_config=lora,
)
trainer.train()
trainer.save_model("out/invoice-lora/final")   # saves the adapter only
```

TRL and transformers rename arguments between releases (e.g. `tokenizer` → `processing_class`, `max_seq_length` → `max_length`); pin versions in `pyproject.toml` and check the installed release's docs.

## Hyperparameter Guide

From j4flmao/agent-skills, MIT.

More from j4flmao/agent-skills