People argue against the "more goods" part, on the basis that we have way more stuff than we know what to do with anyway. They're sort of right, but it's throwing baby out with the bathwater. Economic growth also encompasses doing same stuff better, and being more efficient with resource use. From what I've seen, the motivation behind criticism is usually twofold: criticizing greed, and worrying that ever-increasing use will exhaust available resources.