Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
arXiv:2608.13596v1 Announce Type: new Abstract: Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.
📖 Cet article provient d'une source externe.
🔗 Lire l'article complet sur la source →243 mots extraits · Source originale
Ordinateur portable N5095 15,6 pouces, 8 Go + 256 Go, Windows 11 Intel Celeron N5095 Quad Core, prise en charge du déverrouillage par empreinte digitale, prise britannique (argent)
✅ Découvrez toutes les caractéristiques.
✅ Vérifiez la disponibilité actuelle.
✅ Consultez les avis des acheteurs.
