This is not a direct response to your experience but a little off topic but lazyvim uses lazy.vim which is a modern plugin manager that lazy loads plugins/extensions, it also loads plugins on demand instead of loading them all together at startup from your config. It also can lazy-load plugins based on events, keymaps, filetypes etc