I was going to come in here announcing some random algorithm book, but you know, you're right. When I'm writing code for speed, I have to start thinking about how the data is laid out in memory (even in languages that abstract much of this away from you, like Java); and I start thinking about how long different operations take in their assembly form. These are both pretty important things to know, especially as you get close to the metal in languages like C. Discrete Math is useful, too; as is at least a basic understanding of networking and compilers; and, nowadays, knowing how to work with ~at least~ SQL databases is greatly beneficial. Most of that whole list is useful :)