FPGA latency is also much more predictable. Since you have cycle level control, you can know exactly how long any given operation will take. In a multi-threaded CPU, it's likely that the operation may be interrupted or delayed because of another process using the same resourt.
It's also worth noting that FPGAs are often much closer to the I/O than a CPU on a motherboard. Pin to pin latencies are much reduced in a situation where direct access to I/O is available. An Ethernet stack can sit on the same silicon without having to serialize data between the CPU and the north bridge.
>In a multi-threaded CPU, it's likely that the operation may be interrupted or delayed because of another process using the same resourt.
If you're considering an FPGA, isn't the alternative likely bare-metal CPU programming? In that case, you have just as much control over thread scheduling as you do over FPGA timing don't you?
Throughput and latency are opposite ends of the same tradeoff. FPGAs enable cycle perfect timing control, while for bare-metal CPU programming.. even if you just have an infinite loop running bare-metal on one CPU, looking at the assembly can't tell you anything about the timing. Modern CPUs have multiple layers of caches with penalties for a miss coming in at hundreds or thousands of cycles. They do parallel and speculative execution of instructions. Instructions cost variable amount of cycles and take variably long depending on instructions before and after.
And then, with the SoCs that are everywhere now.. you are poking some memory mapped register, say you toggle a GPIO output value.. how long does it take to change on the actual physical pin? Is the GPIO peripheral part of the processor or some IP they bought in and then interconnected over an AXI bus? Is it buffered? It's all entirely impossible to say.
I feel like people are taking the hybrid approach of having a dedicated microcontroller on the same die as their CPU, like the PRU on TI's ARM chips: http://processors.wiki.ti.com/index.php/PRU-ICSS
This by no means replaces an FPGA, but if you just want your pins to flip and code to run at a predictable interval, this gets you that. (As for reading pins, they have another peripheral on the die that will just timestamp when an event occurs. This is much more consistent than dealing with your OS's interrupt handler, but again, it's no FPGA.)
Yeah, the TI PRUs are what you get approaching this topic from a processor manufacturer standpoint, the XILINX Zynq is how a FPGA manufacturers would do it.
It's also worth noting that FPGAs are often much closer to the I/O than a CPU on a motherboard. Pin to pin latencies are much reduced in a situation where direct access to I/O is available. An Ethernet stack can sit on the same silicon without having to serialize data between the CPU and the north bridge.