Several years of ML research for CNNs indicate at least that one can do very well with 8 bit integers. Such quantization is basically standard now for any deployment outside of GPU (where 8bit isn't any faster anyway due to the hardware).