I think there is a bit stronger version of your argument:
The wealth that I have is intrinsically tied to me personally: It's a business that I'm running, an idea I'm pursuing, etc. Taking it away from me destroys it.
The discrepancy comes from confusing money with wealth. You can take my $100 and then you will have $100. But if you take my business away from me, you will end up with very little. And so will everybody else that was benefitting from the business that I was running.
This is a compelling argument but there is an upper limit of how much wealth can be intrinsically tied to a person. When we start talking about 8, 9, 10, 11, even 12 figures it's really hard to argue that the value of the wealth would diminish if it was distributed among more people. If anything I would expect people with more modest amounts of wealth to be more motivated to manage it (rather than hiring someone else to do it, at which point there is no intrinsic personal connection).
The discrepancy comes from confusing money with wealth. You can take my $100 and then you will have $100. But if you take my business away from me, you will end up with very little. And so will everybody else that was benefitting from the business that I was running.