I agree with you these arguments can be used to justify atrocities. Actually they can be used to justify atrocities arbitrarily large as long as you find enough people that you think are being benefitted.

However I don't think they should be entirely dismissed. Specially if at some point the weight on each side of the balance differs by enough orders of magnitude.