01

Separate learning rates

LM head transformer body input embeddings LR LR LR or the reverse
No better than a single learning rate.
Following Unsloth
02

Protecting conflicting layers

DNA ENG
DNA scores did not change, and English improved by only about one point.
 
03

Merging two specialists

DNA specialist English specialist merged base + a·ΔDNA + b·ΔEnglish
The merged models traded DNA for English and never kept both.