You don’t even solve the problem by moving the memory on-die, as the majority of time is not spent sending the signal on traces, it’s waiting for the relatively slow memory to find, read, and send the data on the bus.

Expanding the cache so everything fits in it is one way to achieve the performance uplift of cache hits, but cache is expensive compared to memory, and current cache sizes are tiny compared to RAM. If cache gets 50x bigger tomorrow, chips will get at least 10x as hot, power-hungry, and expensive.

Judging by the replies, it seems no one is looking at Apple's "Unified Memory." RAM is on the same silicon with the CPU.

Apple RAM is not on the same silicon[1] as the CPU, just on the same package. Also it doesn't have significant latency advantages. It has higher throughput, possibly because of a custom controller. The biggest advantage is that the GPU doesn't have to go through the PCI bus to access it.

[1] my understanding is that RAM and CPU processes are very different and it is hard to produce a chip with both features while remaining optimal.

think about it, L1, L2 and L3 are already on die, yet they are progressively slower. Why do you think it is the case?

Distance from the CPU. And traditionally, RAM sits waaaaaay across the motherboard. If you put it on the CPU die, it’s much closer and much faster. Still slower than L1 and L2 cache, sure, but much faster than waiting on the fetch across the motherboard.

Look into Apple’s unified memory.

Apple machines are already pretty much not RAM-upgradeable. And they design their own chips too. So if there were benefits to putting RAM on the die (I don't know myself, others in this thread suggest there are in fact not), Apple would seem like a possible first mover, they don't seem subject to the downsides.

This is exactly what "unified memory" is on Apple systems, the RAM is on the silicon with the CPU.

> ... the available RAM is on the M1 system-on-a-chip (SoC).