None of the small LLMs are good enough yet. You could certainly build a system around local VLMs but it would require much more task specific programming baked in.

I’m certainly interested in building a product (not entirely controlled by an LLM but I see lots of utility in building interfaces with them) but not really sure what this would be useful for. Looking into some spaces now but there has to be a clear ROI to get any sort of funding for robotics.