Mailboxes live in this weird property-right limbo (that I don’t know really anything about) where the homeowner buys it but it is “property of” the US Postal Service. I found this[1] Supreme Court case that sort of touches on how the property rights work.
I don't think that's what they're saying though. They're talking about what the mailbox while it's in use. I didn't find anything mentioning boxes being property of the USPS, they use the term "authorized repository." When it's in use, it's still property of the private entity, but it's usage is subject to the rules of USPS. (I'm not a lawyer)
My understanding it they can remove the USPS box (looking at what postal workers said on Quora), but they wouldn't be part of the mail system anymore. I guess they could have mail forwarded to a new box, or else it'd be undeliverable.
> Writing for the majority, Associate Justice William H. Rehnquist said that no one was required to receive mail or to have a mailbox but that once a postal customer provides a box for receiving mail he implicitly agrees to abide by postal regulations on its use.
It's your property. Do whatever you please.