There are some mad lads making different sizes of llama by "grafting" attention heads from one model onto another and finetuning a bit to stablize the transplant. For instance:
Any place people doing such grafting are congregating?
I've often pondered if taking some random chunk of weights from the middle of a trained model, and dumping it into some totally different model might perform better than random initialization when the scale gets big enough.
Being in the Discord age, lots of the discussion about cool llm stuff is fragmented and buried. I am in these discords, and I only know because I ran into it on HuggingFace.
Is it something other than the standard meaning of "slerp" as spherical linear interpolation? That's not exactly new, it's been around since the mid-1980s.
https://huggingface.co/models?sort=modified&search=20B
Its very experimental, but apparently the 20B models are actually improving on 13B.