This indeed seems to have been a heavy focus of their research team in the past year, eg. "Efficient Large Language Model Inference with Limited Memory" [1] and OpenELM [2]

[1] https://arxiv.org/pdf/2312.11514

[2] https://arxiv.org/pdf/2404.14619 (with 1.1B parameters, this appears to be their attempt at building a lightweight LLM)