Oh I'm with you, you've gotta put other controls in place. Still in my basic acl for every network, because it's one if the first things users will do to circumvent controls.
HTTP 1.1 servers need the host name in the request, so that a single IP can host multiple domains that resolve to it. If you just go to the IP address, you get an error or a default host. It should work fine with most other protocols, though.
Adding to what others say here: if you have/know the ip address, you probably also know the host name. There's nothing magical about:
# from memory, syntax might not quite work
telnet 1.2.3.4 80
Http/1.1
Host: example.com
Get /
Which is indeed why you can put the ip and host name(s) in /etc/hosts - and without other network level blocks - browsers etc will just work.
With http 1.0 blocking/filtering ips was enough, with 1.1 you need a proxy. With tls/ssl you have the choice between (having the capability to) decrypt everything or filter nothing. (obviously ip level filtering works, but it's a little crude in a Http 1/1 world. Ditto for http2 etc).
Especially with things like Google DNS over HTTPS and https://github.com/pforemski/dingo ...