Chris, I think we didn't communicate this well. When we were first building RethinkDB we looked at other similar products and noticed that they don't perform well in split-brain scenarios, and allow all kinds of errors and issues to creep in. For examples of this, just read Aphyr's posts that review various distributed systems.
So we thought, first, let's build a system based on manual failover, make that really solid, learn as much as possible about real-world use cases, and then build automatic failover on top of that.
This process took a couple of years, and we're finally at a point where we can ship automatic failover that we're confident will behave correctly in real-world (and theoretical) scenarios (modulo bugs that may not have been uncovered in testing).
To answer your specific question, if a netsplit occurs, two primaries could never be elected. There will be a primary either one side, or the other side of the split, so rejoining will not be a problem.
In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client.
These questions can get really, really subtle -- (for example, what happens if there are multiple cascading netsplits in your cluster?) We wanted to take the time to understand these problems much better before we build automated failover, so we went with a manual failover system first.
> In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client.
That's the particular case that I was curious about, that solution makes a lot of sense. I suppose that would also mean that if an outage takes out more than half of your replicas then you will loose all write-abilities (unless I'm mistaken), but that's probably better than the potential mess that could ensue.
Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?
> That's the particular case that I was curious about, that solution makes a lot of sense. I suppose that would also mean that if an outage takes out more than half of your replicas then you will loose all write-abilities
That's correct. We've built in an "emergency repair" provisions into the product to handle cases like this, should they happen, but that requires manual intervention. (In general, if you lose more than half of your servers, you want to intervene manually anyway)
> Do these write requests to nodes that have no route to a primary simply "hang", or are they rejected by the daemon?
They time out via normal TCP mechanisms and get rejected.
So we thought, first, let's build a system based on manual failover, make that really solid, learn as much as possible about real-world use cases, and then build automatic failover on top of that.
This process took a couple of years, and we're finally at a point where we can ship automatic failover that we're confident will behave correctly in real-world (and theoretical) scenarios (modulo bugs that may not have been uncovered in testing).
To answer your specific question, if a netsplit occurs, two primaries could never be elected. There will be a primary either one side, or the other side of the split, so rejoining will not be a problem.
In practice, however, this is more subtle -- what if there is a netsplit, one side maintains its primary, and the other side elects a new one? What if the user writes to both primaries on either side? What happens after the cluster rejoins? In this specific case, we solve this by requiring the majority of the replicas to acknowledge writes by default before the write acknowledgement is sent to the client.
These questions can get really, really subtle -- (for example, what happens if there are multiple cascading netsplits in your cluster?) We wanted to take the time to understand these problems much better before we build automated failover, so we went with a manual failover system first.