Do you need specialized knowledge?

|

+-----------+-----------+

| |

Yes No

| |

Is it static knowledge? Do you need consistent style/format?

| |

+-------+------+ +--------+-------+

| | | |

Yes No Yes No

| | | |

Consider Use RAG Can prompting Use base

Fine-tuning achieve it? model

|

+-------+------+

| |

Yes No

| |

Use Fine-tune

prompts

Data Preparation

Data Requirements

Quantity:

• Minimum: 50-100 examples (often yields some improvement)
• Recommended: 500-1000 examples (meaningful improvement)
• Ideal: 1000-10000 examples (strong performance)
• More may not be better if data quality suffers

Quality:

Quality matters more than quantity. Poor examples teach poor behavior.

Diversity:

Cover the range of inputs you expect:

• Edge cases
• Varying complexity
• Different phrasings
• Multiple scenarios

Format:

Typically JSON with messages format:

{

"messages": [

{"role": "system", "content": "You are a technical support assistant..."},

{"role": "user", "content": "My internet keeps disconnecting..."},

{"role": "assistant", "content": "I understand how frustrating that must be..."}

]

}

Data Collection Strategies

From Human Experts:

Have domain experts write ideal responses:

• Highest quality
• Captures true expertise
• Expensive and slow
• Good for critical examples

From Existing Logs:

Filter and curate existing conversation logs:

• Abundant data source
• Variable quality
• May contain errors or bias
• Requires careful filtering

Synthetic Generation:

Use strong models to generate training data:

• Scalable
• Risk of reinforcing model biases
• Requires validation
• Good for format/structure training

Hybrid Approaches:

Combine methods:

• Expert examples for critical cases
• Synthetic for scale
• Log-based for real-world coverage
• Human validation throughout

Data Cleaning and Curation

Remove Problematic Examples:

• Factually incorrect responses
• Poor grammar or writing quality
• Inconsistent formatting
• Unsafe or biased content
• Off-topic exchanges

Ensure Consistency:

• Consistent terminology
• Uniform formatting
• Aligned tone and style
• Coherent behavior across examples

Balance the Dataset:

• Represent all categories appropriately
• Don't over-index on common cases
• Include edge cases and exceptions
• Avoid spurious correlations

Data Formatting

Chat Format (Recommended):

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

{"messages": [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}

Multi-Turn Conversations:

{

"messages": [

{"role": "system", "content": "You are a helpful coding assistant."},

{"role": "user", "content": "How do I read a file in Python?"},

{"role": "assistant", "content": "Here's how to read a file in Python..."},

{"role": "user", "content": "What if the file doesn't exist?"},

{"role": "assistant", "content": "You should handle the FileNotFoundError..."}

]

}

Completion Format (Older style):

{"prompt": "Translate to French: Hello", "completion": " Bonjour"}

{"prompt": "Translate to French: Goodbye", "completion": " Au revoir"}

Fine-Tuning Techniques

Full Fine-Tuning

What It Is:

Update all model parameters during training.

Advantages:

• Maximum flexibility
• Best performance potential
• Full model adaptation

Disadvantages:

• Extremely resource-intensive
• Requires multiple high-end GPUs
• Expensive for large models
• Risk of catastrophic forgetting
• Storage for full model copies

When to Use:

• Creating significantly specialized models
• When you have abundant compute resources
• Models small enough to be practical

Typical Setup:

• GPT-3.5-scale: 4-8 A100 GPUs
• LLaMA 7B: 2-4 A100 GPUs
• LLaMA 70B: 16+ A100 GPUs

LoRA (Low-Rank Adaptation)

What It Is:

Train small adapter matrices that modify the frozen base model:

W' = W + BA

Where W is frozen original weights, B and A are trainable low-rank matrices.

Advantages:

• Dramatically reduced memory and compute
• Can train on single consumer GPUs
• Multiple LoRAs can share base model
• Easy to store, share, and switch
• Reduced catastrophic forgetting

Disadvantages:

• Slightly less expressive than full fine-tuning
• Adds small inference overhead
• May not capture all adaptations

Typical Settings:

• Rank (r): 8-64 (higher = more parameters)
• Alpha: typically 16-32
• Target modules: query, key, value projections
• Dropout: 0.05-0.1

Example Configuration (PEFT library):

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(

r=16,

lora_alpha=32,

target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],

lora_dropout=0.05,

bias="none",

task_type="CAUSAL_LM"

)

model = get_peft_model(base_model, lora_config)

QLoRA (Quantized LoRA)

What It Is:

Combine LoRA with model quantization for even greater efficiency:

• Base model quantized to 4-bit
• LoRA adapters trained in higher precision
• Dramatically reduced memory requirements

Advantages:

• Fine-tune 7B models on single 24GB GPU
• Fine-tune 70B models on reasonable hardware
• Nearly matches LoRA performance
• Very accessible for practitioners

Disadvantages:

• Slightly lower quality than LoRA
• Quantization overhead at inference
• More complex setup

Example Configuration:

import torch

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(

load_in_4bit=True,

bnb_4bit_quant_type="nf4",

bnb_4bit_compute_dtype=torch.float16,

bnb_4bit_use_double_quant=True,

)

model = AutoModelForCausalLM.from_pretrained(

model_name,

quantization_config=bnb_config,

device_map="auto"

)

Other Techniques

Prefix Tuning:

Train continuous "soft prompts" prepended to inputs:

• Very parameter-efficient
• Good for multiple tasks
• Less common than LoRA

Prompt Tuning:

Similar to prefix tuning, learns task-specific embeddings:

• Extremely lightweight
• Works well for simpler adaptations
• Limited expressiveness

Adapter Layers:

Insert small trainable layers between frozen transformer blocks:

• Modular and composable
• Adds inference latency
• Less popular than LoRA now

RLHF (Reinforcement Learning from Human Feedback):

Train a reward model from human preferences, then optimize:

• Aligns with human preferences
• Complex multi-stage process
• Requires significant resources

DPO (Direct Preference Optimization):

Simpler alternative to RLHF:

• Uses preference pairs directly
• No reward model needed
• More stable training

Implementation Guide

Setting Up the Environment

Requirements:

pip install transformers datasets peft accelerate bitsandbytes

pip install torch # with CUDA support

For advanced training:

pip install trl # for RLHF/DPO

pip install wandb # for experiment tracking

pip install deepspeed # for distributed training

Basic Fine-Tuning with Hugging Face

from transformers import (

AutoModelForCausalLM,

AutoTokenizer,

TrainingArguments,

Trainer,

DataCollatorForLanguageModeling

)

from datasets import load_dataset

# Load model and tokenizer

model_name = "meta-llama/Llama-2-7b-hf"

tokenizer = AutoTokenizer.from_pretrained(model_name)

tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(

model_name,

torch_dtype=torch.float16,

device_map="auto"

)

# Load and prepare dataset

dataset = load_dataset("json", data_files="training_data.jsonl")

def format_example(example):

text = tokenizer.apply_chat_template(

example["messages"],

tokenize=False

)

return {"text": text}

dataset = dataset.map(format_example)

def tokenize(example):

return tokenizer(

example["text"],

truncation=True,

max_length=2048,

padding="max_length"

)

tokenized_dataset = dataset.map(tokenize, remove_columns=["text", "messages"])

# Training arguments

training_args = TrainingArguments(

output_dir="./fine-tuned-model",

num_train_epochs=3,

per_device_train_batch_size=4,

gradient_accumulation_steps=4,

learning_rate=2e-5,

warmup_steps=100,

logging_steps=10,

save_steps=100,

fp16=True,

)

# Trainer

trainer = Trainer(

model=model,

args=training_args,

train_dataset=tokenized_dataset["train"],

data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),

)

# Train

trainer.train()

# Save

trainer.save_model("./fine-tuned-model")

QLoRA Fine-Tuning Example

import torch

from transformers import (

AutoModelForCausalLM,

AutoTokenizer,

BitsAndBytesConfig,

TrainingArguments

)

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

from trl import SFTTrainer

# Quantization config

bnb_config = BitsAndBytesConfig(

load_in_4bit=True,

bnb_4bit_quant_type="nf4",

bnb_4bit_compute_dtype=torch.float16,

bnb_4bit_use_double_quant=True,

)

# Load quantized model

model = AutoModelForCausalLM.from_pretrained(

"meta-llama/Llama-2-7b-hf",

quantization_config=bnb_config,

device_map="auto",

trust_remote_code=True,

)

# Prepare for training

model = prepare_model_for_kbit_training(model)

# LoRA config

lora_config = LoraConfig(

r=16,

lora_alpha=32,

target_modules=[

"q_proj", "k_proj", "v_proj", "o_proj",

"gate_proj", "up_proj", "down_proj"

],

lora_dropout=0.05,

bias="none",

task_type="CAUSAL_LM"

)

# Apply LoRA

model = get_peft_model(model, lora_config)

model.print_trainable_parameters() # Shows ~0.1% of parameters trainable

# Tokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

tokenizer.pad_token = tokenizer.eos_token

# Training arguments

training_args = TrainingArguments(

output_dir="./qlora-model",

num_train_epochs=3,

per_device_train_batch_size=4,

gradient_accumulation_steps=4,

learning_rate=2e-4, # Higher LR typical for LoRA

warmup_ratio=0.03,

logging_steps=10,

save_strategy="epoch",

fp16=True,

optim="paged_adamw_32bit",

)

# SFT Trainer from TRL

trainer = SFTTrainer(

model=model,

args=training_args,

train_dataset=dataset["train"],

tokenizer=tokenizer,

max_seq_length=2048,

dataset_text_field="text",

)

# Train

trainer.train()

# Save LoRA adapters

model.save_pretrained("./qlora-adapters")

Using OpenAI Fine-Tuning API

import openai

from openai import OpenAI

client = OpenAI()

# Upload training file

with open("training_data.jsonl", "rb") as f:

training_file = client.files.create(

file=f,

purpose="fine-tune"

)

# Create fine-tuning job

job = client.fine_tuning.jobs.create(

training_file=training_file.id,

model="gpt-4o-mini-2024-07-18",

hyperparameters={

"n_epochs": 3,

"batch_size": 4,

"learning_rate_multiplier": 1.0

}

)

# Monitor progress

while True:

job_status = client.fine_tuning.jobs.retrieve(job.id)

print(f"Status: {job_status.status}")

if job_status.status in ["succeeded", "failed"]:

break

time.sleep(60)

# Use the fine-tuned model

fine_tuned_model = job_status.fine_tuned_model

response = client.chat.completions.create(

model=fine_tuned_model,

messages=[{"role": "user", "content": "Hello!"}]

)

Hyperparameter Tuning

Learning Rate

Typical Ranges:

• Full fine-tuning: 1e-6 to 5e-5
• LoRA: 1e-4 to 3e-4
• QLoRA: 2e-4 to 5e-4

Guidelines:

• Too high: Loss spikes, unstable training
• Too low: Slow convergence, underfitting
• Use warmup to stabilize early training
• Learning rate schedulers help (cosine, linear decay)

Batch Size and Gradient Accumulation

Effective batch size = per_device_batch_size × num_devices × gradient_accumulation_steps

Guidelines:

• Larger batches: more stable gradients, higher memory
• Smaller batches: noisier updates, may regularize
• Use gradient accumulation to simulate larger batches
• Typical effective batch: 32-128 for fine-tuning

Epochs and Steps

Typical Ranges:

• 1-5 epochs for most fine-tuning
• More epochs with smaller datasets
• Fewer epochs with larger datasets
• Watch for overfitting

Early Stopping:

Monitor validation loss; stop when it increases.

LoRA-Specific Hyperparameters

Rank (r):

• 8: Very lightweight, limited capacity
• 16: Good balance for most tasks
• 32: Higher capacity
• 64+: Approaches full fine-tuning expressiveness

Alpha:

• Often set to 2×r
• Higher alpha: stronger adaptation
• Lower alpha: more conservative

Target Modules:

• Minimum: Query, Value projections
• Recommended: Q, K, V, Output projections
• Full: Add MLP layers (gate, up, down)

Evaluation and Iteration

Evaluation Metrics

Loss-Based:

• Training loss: Should decrease smoothly
• Validation loss: Monitor for overfitting

Task-Specific:

• Accuracy for classification
• ROUGE/BLEU for generation
• Exact match for Q&A
• Custom metrics for your task

Human Evaluation:

• Relevance ratings
• Quality assessments
• Preference comparisons
• Error analysis

Common Issues and Solutions

Overfitting:

• Symptoms: Val loss increases while train loss decreases
• Solutions: More data, regularization, fewer epochs, lower LR

Underfitting:

• Symptoms: Both losses remain high
• Solutions: More epochs, higher LR, larger model, better data

Catastrophic Forgetting:

• Symptoms: General capabilities degrade
• Solutions: Use LoRA, mix in general data, regularization

Mode Collapse:

• Symptoms: Repetitive or templated outputs
• Solutions: More diverse data, temperature adjustment, nucleus sampling

Format Inconsistency:

• Symptoms: Model doesn't follow format reliably
• Solutions: More format examples, clearer formatting in data

Iteration Strategy

1. Start Small:
• Fine-tune on small subset first
• Validate approach works
• Identify data issues early

1. Scale Gradually:
• Increase data size
• Tune hyperparameters
• Monitor metrics carefully

1. Evaluate Comprehensively:
• Test on held-out data
• Check edge cases
• Verify general capabilities preserved

1. Deploy and Monitor:
• A/B test against baseline
• Collect production feedback
• Plan for retraining

Production Considerations

Model Serving

Merging LoRA Adapters:

For inference efficiency, merge adapters into base model:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("base-model")

peft_model = PeftModel.from_pretrained(base_model, "lora-adapters")

merged_model = peft_model.merge_and_unload()

merged_model.save_pretrained("merged-model")