I wonder if you could send the embeddings or some higher level compressed latent vector across the cloud you couldn't get the best of both worlds.

GPS, phone orientation, last 5 apps you were in, etc. --> embedding

you might even have like "what time is it?" compressed as it's own embedding.