QLoRA made it possible to fine-tune a 7-8B-parameter language model on one consumer GPU, and in 2026 the tooling is stable enough that the whole job fits in one script. This tutorial walks through fine-tuning an open-weight Llama model on an instruction dataset with 4-bit quantization and LoRA adapters, evaluating the result, and exporting merged weights for serving with vLLM.
Budget. An 8B model in 4-bit with LoRA adapters trains comfortably in 16 GB of GPU memory at sequence length 1024 with gradient checkpointing; a 24 GB card gives headroom for longer sequences. Expect one to three hours for a few thousand examples on a single RTX-class GPU.
1. Pick the model and check the license
Open-weight does not mean unrestricted. Before downloading, read the license for the model you intend to use: Llama models ship under the Llama Community License with an acceptable-use policy and a large-user clause; Mistral, Qwen and Gemma each have their own terms. Confirm commercial use is allowed for your case and that you can redistribute fine-tuned weights if you need to. Accept the model terms on Hugging Face and log in with huggingface-cli login.
Install the stack:
pip install torch transformers peft trl datasets bitsandbytes accelerate
2. Prepare the dataset
The model learns the format you show it. Use the model's own chat template so the fine-tune matches how you will prompt it later. Start from a JSONL file of {"instruction": ..., "input": ..., "output": ...} records and hold out a real evaluation split before anything else.
from datasets import load_dataset
raw = load_dataset("json", data_files="data/train.jsonl", split="train")
raw = raw.shuffle(seed=42)
split = raw.train_test_split(test_size=0.05, seed=42)
train_ds, eval_ds = split["train"], split["test"]
def to_messages(example):
user = example["instruction"]
if example.get("input"):
user += "\n\n" + example["input"]
return {
"messages": [
{"role": "system", "content": "You are a concise, accurate assistant."},
{"role": "user", "content": user},
{"role": "assistant", "content": example["output"]},
]
}
train_ds = train_ds.map(to_messages, remove_columns=train_ds.column_names)
eval_ds = eval_ds.map(to_messages, remove_columns=eval_ds.column_names)
Deduplicate and decontaminate first: drop exact and near-duplicate records, and make sure nothing in your evaluation set (or any public benchmark you plan to report) appears in training. This step is unglamorous and it is the one that most often invalidates results.
3. Load the 4-bit base model and attach LoRA
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
MODEL = "meta-llama/Llama-3.1-8B-Instruct"
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL)
tokenizer.pad_token = tokenizer.pad_token or tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
MODEL, quantization_config=bnb, dtype=torch.bfloat16, device_map={"": 0}
)
model = prepare_model_for_kbit_training(model)
lora = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
task_type="CAUSAL_LM",
)
nf4 with double quantization is the QLoRA recipe; the base weights are frozen and stored in 4-bit, and compute happens in BF16. Targeting all linear projections (attention and MLP) gives better quality than attention-only for a modest parameter increase; rank 16 is a sensible default, 8 for small datasets, 32-64 when you have tens of thousands of examples.
4. Train with TRL's SFTTrainer
from trl import SFTConfig, SFTTrainer
config = SFTConfig(
output_dir="out/llama-qlora",
num_train_epochs=2,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # effective batch 16
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
bf16=True,
gradient_checkpointing=True,
max_length=1024,
logging_steps=10,
eval_strategy="steps",
eval_steps=100,
save_strategy="steps",
save_steps=100,
save_total_limit=2,
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=config,
train_dataset=train_ds,
eval_dataset=eval_ds,
processing_class=tokenizer,
peft_config=lora,
)
trainer.train()
trainer.save_model("out/llama-qlora/adapter")
SFTTrainer applies the chat template to the messages field, masks padding, and handles the PEFT wrapping. Gradient checkpointing trades roughly 30% more compute for a large memory saving; it is what makes 16 GB workable. If you see loss collapse to near zero in the first few hundred steps, your evaluation data leaked into training or the dataset is full of duplicates.
5. Evaluate before and after
Training loss is not a result. Evaluate on the held-out set with a task-appropriate metric, and run the same evaluation on the base model so you can show the delta. For structured tasks (classification, extraction) compute exact-match or F1; for open-ended generation use an LLM-as-judge rubric and spot-check a sample by hand.
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
MODEL, quantization_config=bnb, dtype=torch.bfloat16, device_map={"": 0})
tuned = PeftModel.from_pretrained(base, "out/llama-qlora/adapter")
tuned.eval()
def generate(m, messages, max_new_tokens=256):
ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt").to(m.device)
out = m.generate(ids, max_new_tokens=max_new_tokens, do_sample=False)
return tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)
Loop over eval_ds, generate with the user turn only, and score against the reference assistant turn. Keep the base-model scores in the same table; a fine-tune that does not beat the base on your task should not ship.
6. Merge and export for vLLM
Adapters can be served directly (vLLM supports LoRA at runtime, useful for multi-tenant setups), but the simplest production path is merged weights. Merging requires loading the base in BF16 rather than 4-bit:
base_bf16 = AutoModelForCausalLM.from_pretrained(
MODEL, dtype=torch.bfloat16, device_map="cpu")
merged = PeftModel.from_pretrained(base_bf16, "out/llama-qlora/adapter").merge_and_unload()
merged.save_pretrained("out/llama-qlora/merged", safe_serialization=True)
tokenizer.save_pretrained("out/llama-qlora/merged")
Then serve:
pip install vllm
vllm serve out/llama-qlora/merged --dtype bfloat16 --max-model-len 4096
vLLM exposes an OpenAI-compatible endpoint on port 8000. Re-run your evaluation against the served endpoint; a small numeric drift from the 4-bit training-time model is expected, a large one is a bug. For quantizing the merged model for cheaper serving, see Serving an LLM with vLLM.
What to try next
Increase rank and data before increasing epochs; two epochs is usually plenty and a third often overfits. Add a DPO stage with TRL when you need to shape tone or format preference. And if the model is too big to serve economically, distill it into a smaller one using its own outputs as training data.
We run this pipeline for clients as part of our LLM fine-tuning and customization service: data preparation, training, evaluation and serving-ready weights on your infrastructure. Contact us if you would like help with yours.