I think the point is just that "memory safety is a magic bullet" is a straw man version of the actual argument, "memory safety improves security significantly over not having it".
I've seem people believe that memory leaks can't happen in programs written in garbage-collected programs. So, yeah, the belief that garbage collection and/or bounds checking is a magic bullet really needs to be addressed.