When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
📡 Tech & Science When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging arXiv:2608.27518v1 Announce Type: new Abstract: Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as \textit{task interference} and reduce it to a common layer-wise Frobenius inner product $\langle \Delta W_\ell, J_\ell(x)\rangle_F$. This quantity, in turn, is utilized to expose th...
.jpg)
.jpg)
