Well, to be fair you will lose it if you had it in first place. People who don't bulk up by training, only cary the muscle that... carries them. As long as you keep your protein intake as recommended, you will keep the muscle you need in your regular physical activity level.
But yes, diet is much more important than exercise.