You fine-tuned three adapters this quarter: one for support-ticket classification, one for SQL generation, one for the tone your brand team signed off on. Each is good. Serving all three means three checkpoints, three GPUs or a multi-adapter router, and three eval pipelines. Model merging asks a different question: can you get one set of weights that does all three, for the price of a few minutes of CPU arithmetic and no gradient steps at all?
Often, yes. Merging has quietly become standard practice — most of the strongest open-weight checkpoints on public leaderboards are merges, and every serious post-training pipeline now ends with a merge step of some kind. This tutorial covers what actually works in PyTorch: weight averaging, task arithmetic, TIES, DARE, SLERP, and the evaluation discipline that separates a real merge from a checkpoint that looks fine and fails in production.
When merging is the right tool
Merging combines the parameters of several models, not their outputs. That only makes sense when the models share an architecture and a common ancestor — all fine-tunes of the same base checkpoint. Two independently pretrained models cannot be merged meaningfully; their weight spaces are unrelated.
| Situation | Use |
|---|---|
| N fine-tunes of one base, need one model | Merging |
| N specialists, need routing and isolation | Multi-LoRA serving |
| Need a smaller model, not a broader one | Distillation |
| Several checkpoints from one training run | Model soup / EMA |
| Combining models of different families | Ensembling at inference, not merging |
The economics are why people care. A merge takes minutes on a CPU box with enough RAM, costs nothing in GPU hours, and produces a single artifact you serve exactly like any other. Fine-tuning on the union of three datasets costs a training run and usually needs data you cannot legally or practically pool.
The four methods worth knowing
1. Linear averaging (model soup)
The simplest thing: average the weights elementwise. For checkpoints from the same run, or fine-tunes with identical hyperparameters, this is remarkably robust and often beats any individual member.
import torch
from safetensors.torch import load_file, save_file
paths = ["ft_a/model.safetensors", "ft_b/model.safetensors", "ft_c/model.safetensors"]
weights = [0.4, 0.3, 0.3]
merged = {}
for w, p in zip(weights, paths):
sd = load_file(p)
for k, v in sd.items():
t = v.to(torch.float32) * w
merged[k] = t if k not in merged else merged[k] + t
merged = {k: v.to(torch.bfloat16) for k, v in merged.items()}
save_file(merged, "merged/model.safetensors")
Two details that bite people: accumulate in FP32 even when the checkpoints are BF16 (averaging in BF16 loses low bits on every add), and copy the tokenizer, config and generation config from the base model into the output directory or the merge will not load.
2. Task arithmetic
The key insight is that a fine-tune's delta from base — the task vector τ = θ_ft − θ_base — behaves like a portable, composable object. Add task vectors to combine skills; subtract one to remove a behaviour.
def task_vector(ft_sd, base_sd):
return {k: (ft_sd[k].float() - base_sd[k].float()) for k in base_sd}
def apply_task_vectors(base_sd, tvs, scale=0.5):
out = {}
for k, v in base_sd.items():
acc = v.float().clone()
for tv in tvs:
acc += scale * tv[k]
out[k] = acc.to(v.dtype)
return out
scale (usually called lambda) is the one hyperparameter that matters. Below ~0.3 you barely move off the base; above ~1.0 you typically get degeneration — repetition loops, broken formatting. Sweep {0.3, 0.5, 0.7, 1.0} divided by the number of vectors and evaluate each; do not guess.
3. TIES: resolve the interference
Naive task-vector addition fails when vectors disagree: two fine-tunes that move the same parameter in opposite directions cancel out, and the tiny-magnitude noise in each delta accumulates into drift. TIES-Merging fixes both with three steps — trim, elect sign, disjoint merge:
def ties_merge(base_sd, tvs, density=0.2, scale=1.0):
out = {}
for k, base in base_sd.items():
stack = torch.stack([tv[k].float() for tv in tvs]) # [N, ...]
# 1. TRIM: keep only the top-`density` fraction by magnitude, per vector
flat = stack.flatten(1)
keep = max(1, int(flat.shape[1] * density))
thresh = flat.abs().kthvalue(flat.shape[1] - keep + 1, dim=1).values
flat = torch.where(flat.abs() >= thresh[:, None], flat, torch.zeros_like(flat))
stack = flat.view_as(stack)
# 2. ELECT: majority sign, weighted by total magnitude
elected = torch.sign(stack.sum(0))
# 3. DISJOINT MERGE: average only entries agreeing with the elected sign
agree = (torch.sign(stack) == elected) & (stack != 0)
summed = (stack * agree).sum(0)
count = agree.sum(0).clamp(min=1)
out[k] = (base.float() + scale * summed / count).to(base.dtype)
return out
A density of 0.1-0.3 is normal: most of a fine-tune's delta is redundant. TIES is the default we reach for when merging three or more genuinely different task fine-tunes.
4. DARE: drop and rescale
DARE (Drop And REscale) makes the same observation more aggressively — randomly zero 90-99% of each delta, then rescale the survivors by 1/(1-p) to preserve the expected value. It is usually applied before TIES or task arithmetic rather than instead of them.
def dare(tv, p=0.9, generator=None):
return {k: (torch.bernoulli((1 - p) * torch.ones_like(v), generator=generator) * v) / (1 - p)
for k, v in tv.items()}
That 90%+ of a fine-tune's parameter delta can be discarded with negligible quality loss says something real about how much of fine-tuning is redundant. In practice dare_ties — DARE-sparsify then TIES-elect — is the strongest general-purpose recipe for merging many task fine-tunes.
SLERP, briefly
For exactly two models, spherical linear interpolation preserves the norm of the weight vectors instead of shrinking it the way linear averaging can. It tends to produce smoother merges for two strong chat models. It does not generalise past two models, which limits its usefulness in multi-task pipelines.
Merging LoRA adapters
Most consulting work involves LoRA fine-tunes, not full ones, and adapters bring their own rules.
Merging an adapter into its base is exact and always safe:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "adapters/sql-v3")
model = model.merge_and_unload() # writes BA*alpha/r into the base weights
model.save_pretrained("merged/sql-v3")
Merging two adapters with each other is not exact. A₁B₁ + A₂B₂ is generally not representable as a single rank-r product, so concatenation (peft's add_weighted_adapter(..., combination_type="cat")) doubles the rank, while "ties" / "dare_ties" / "svd" compress back to rank r and lose something. Rules of thumb: merge each adapter down into the base first and merge in full-weight space when you can afford the RAM; if you must stay in adapter space, use cat with a rank budget you are willing to serve; and never merge adapters trained against different base checkpoints or different target-module sets.
If the specialists genuinely need to stay separable — different customers, different compliance boundaries — do not merge at all. Serve many adapters from one GPU instead.
Using mergekit
Hand-rolled merges are worth writing once to understand the mechanics; in production, mergekit is the tool. It handles sharded checkpoints, out-of-core merging on machines with modest RAM, tokenizer reconciliation and every method above.
# merge.yaml
base_model: meta-llama/Llama-3.1-8B-Instruct
merge_method: dare_ties
dtype: bfloat16
parameters:
int8_mask: true
models:
- model: ./ft-support-classifier
parameters: { weight: 0.4, density: 0.6 }
- model: ./ft-sql-generation
parameters: { weight: 0.35, density: 0.6 }
- model: ./ft-brand-tone
parameters: { weight: 0.25, density: 0.5 }
mergekit-yaml merge.yaml ./merged-model --cuda --lazy-unpickle --allow-crimes
Weights need not sum to 1 (with normalize: true, the default for several methods, they are renormalized for you), and density is per-model: give the task you care about most the higher density.
Evaluating a merge — the part teams skip
A merge has no training loss to watch, which makes it dangerously easy to ship. Every merge is a hypothesis, and merging is cheap enough that you should treat it as a small search, not a single attempt.
Build a held-out suite before you merge, containing: a task set per contributing fine-tune (does each skill survive?); a general-capability regression set such as instruction following and basic reasoning (merges frequently degrade general ability while preserving the narrow tasks); a format-compliance check (JSON validity, tool-call schema — the first thing to break); and a degeneration probe of long generations scanned for repetition loops.
Then sweep. Vary the weights and density over a grid of 8-20 configurations, score each on the same suite, and pick on the frontier rather than on a single average. Watch for the characteristic failure signature: task scores look fine but the model has lost its chat template or started emitting the EOS token late. That means the delta scale was too high — lower scale, or lower density, and rerun.
Log the recipe with the artifact. A merge is fully reproducible from its YAML plus input checkpoint hashes, and six months later that provenance is the only thing standing between you and re-deriving a model nobody remembers building. Pin the input revisions.
A practical workflow
- Fine-tune each capability separately, with its own eval set. Keep the base checkpoint pinned and identical.
- Merge each LoRA into the base to get full-weight fine-tunes.
- Start with
dare_ties, density 0.5-0.7, weights proportional to how much you care about each task. - Sweep weight and density on your held-out suite; 10-20 merges is an hour of CPU.
- Compare the best merge against the multi-adapter serving baseline on both quality and serving cost.
- Ship the merge only if it wins on both. If one task collapses, keep that one as a separate adapter and merge the rest.
Merging will not create a capability that is not in any input. It is a cheap consolidation and regularization tool, not an alternative to good data. But when you have several fine-tunes of one base and one serving budget, an afternoon of merging and evaluation routinely replaces a training run you were about to pay for.
IntelliSensei's PyTorch consultants build and consolidate fine-tuning pipelines for enterprise teams — post-training, merging, evaluation harnesses and the serving stack underneath them. If you are running more fine-tunes than you can afford to serve, get in touch.