One spine, borrowed experts.
Take a strong model and freeze it. Lift expert layers out of other mixture-of-experts families and wire them in through small learned adapters. Train only the joins. The spine never forgets what it knew, because at the start it is exactly itself.
Frozen spine
The base model's weights do not move. It keeps its tokenizer, its embeddings and its behaviour.
Donor experts
Expert layers come from other MoE families, each kept as its donor trained it.
Zero-start adapters
Learned adapters map in and out of donor space. The output side starts at zero.
Donor routers
Each donor's own router picks its own experts, in its own space.
A fresh merge is exactly the spine.
The adapter that writes back into the spine is initialised to zero. Before any training, every transplanted expert contributes nothing, so the merged model's outputs are identical to the spine's.
That gives a clean starting line. Anything the merge gains, it gains in training, and anything it loses can be measured against a known baseline rather than guessed at.
# spine hidden state h (frozen path) z = A(h) # into donor space idx = donor_router(z) # donor picks its own experts y = donor_experts[idx](z) h_out = h + B(y) # B starts at zero # at init: B == 0 => h_out == h
Why this design
Tokenizers never meet
Different model families split text differently. Reconciling two vocabularies is a research project of its own and usually costs quality. A transplant avoids it entirely: text enters through the spine's tokenizer and embeddings and leaves through the spine's head. Donor experts only ever see hidden states, translated into their space by the input adapter. No donor token is ever produced or consumed.
Donors route in their own space
A donor's router was trained alongside its experts. It knows which expert suits which input — but only for inputs that look like the donor's own hidden states. So the router sits after the input adapter and decides in donor space, where its training still applies. We do not train a new router from scratch to guess at experts it has never seen.
Only the joins train
The spine is frozen and the donor experts are kept as their donors trained them. What learns is the adapters. That keeps the trainable surface small, keeps the spine's behaviour as a floor, and makes it clear what the experiment changed.
Controls before claims
A bigger model with extra parameters can look better for reasons that have nothing to do with the donor's knowledge. So each experiment is designed with a control arm: the same adapters and routing wired to experts that carry no donor training, matched in scale. If the real transplant does not beat its control, the donor contributed nothing, and we will say so.
Where it stands.
| Item | Status |
|---|---|
| MethodFrozen spine, zero-initialised adapters, donor routing | In the lab |
| Small-scale experimentDesigned against a control arm | In the lab |
| ResultsPublished with losses alongside any wins | — |
| Released model | Nothing released |
New results land on the evaluation page and in the changelog.
Work on transplants.
If you merge models and want to compare notes, write to us.