Afaik it's not that clear cut. If some kind of fault occurs in the device causing it to draw more current than designed then the lower power charger could be safer assuming it has some kind of current limiter or fuse. Ie high power charger would happily give faulty device more current possibly causing more damage (heating up, exploding batteries etc), while a low power charger could prevent that by tripping a fuse or limiting current.

Of course I'm not sure if this is applicable to USB chargers which all are relatively low power.