But it's worse than that. Even if in theory the system could be fixed, we don't actually know how to fix it for real, the way we can fix a normal computer program.
The reason we can't fix them is because we have no idea how they work; and the reason we have no idea how they work is this:
1. The "normal" computer program, which we do understand, implement a neural network
2. This neural network is essentially a different kind of processor. The "actual" computer program for modern deep learning systems is the weights. That is, weights : neural net :: machine language : normal cpu
3. We don't program these weights; we literally summon them out of the mathematical aether by the magic of back-propagation and gradient descent.
This summoning is possible because the "processor" (the neural network architecture) has been designed to be differentiable: for every node we can calculate the slope of the curve with respect to the result we wanted, so we know "The final output for this particular bit was 0.7, but we wanted it to be 1. If this weight in the middle of the network were just a little bit lower, then that particular output would have been a little bit higher, so we'll bump it down a bit."
And that's fundamentally why we can't verify their properties or "fix" them the way we can fix normal computer programs: Because what we program is the neural network; the real program, which runs on top of that network, is summoned and not written.
That’s a very poetic description. Mine is simpler. It’s a generator. You train it to give it the parameters (weights) of the formula thag generate stuff (the formula is known). Then you give it some input data, and it will gives you an output.
Both the weights and the formula is known. But the weight are meaningless in a human fashion. This is unlike traditional software where everything from encoding (the meaning of the bits) to how the state machine (the cpu) was codified by humans.
The only ways to fix it (somewhat) is to come up with better training data (hopeless), a better formula, or tacking something on top to smooth the worst errors (kinda hopeless).
The correct way to fix it would be to build a decompiler to normal code, that would explain what it does, but this is akin to building the everything machine.
The reason we can't fix them is because we have no idea how they work; and the reason we have no idea how they work is this:
1. The "normal" computer program, which we do understand, implement a neural network
2. This neural network is essentially a different kind of processor. The "actual" computer program for modern deep learning systems is the weights. That is, weights : neural net :: machine language : normal cpu
3. We don't program these weights; we literally summon them out of the mathematical aether by the magic of back-propagation and gradient descent.
This summoning is possible because the "processor" (the neural network architecture) has been designed to be differentiable: for every node we can calculate the slope of the curve with respect to the result we wanted, so we know "The final output for this particular bit was 0.7, but we wanted it to be 1. If this weight in the middle of the network were just a little bit lower, then that particular output would have been a little bit higher, so we'll bump it down a bit."
And that's fundamentally why we can't verify their properties or "fix" them the way we can fix normal computer programs: Because what we program is the neural network; the real program, which runs on top of that network, is summoned and not written.