The reasoning looks solid: two servers, clustered, and if one dies the other takes over. It is the first instinct when you want redundancy without a production budget.
Then the day comes when a node genuinely fails. And the survivor, instead of picking up the load, freezes: nothing starts, nothing changes, the interface is read-only. This is not a malfunction. It is exactly what was designed to happen.
Quorum, and why 2 is the worst number
A cluster must know at all times who decides. The mechanism is a vote: each node has one, and you need a strict majority for the group to be considered legitimate.
Run the numbers for different sizes:
- 3 nodes: majority is 2. One node can fail, the other two carry on.
- 5 nodes: majority is 3. Two can fail.
- 2 nodes: majority is 2. None can fail.
With two nodes, losing one puts the survivor in the minority. It loses quorum, refuses to write the shared configuration, and steps back. A two-node cluster is therefore, in practice, less available than a single server for administrative operations.
Why the survivor cannot simply continue
The natural question: why not let the remaining node get on with it? Because it cannot distinguish two radically different situations:
- the other node is genuinely stopped;
- the other node is running perfectly, but the network link between them is cut.
From node A's point of view these are strictly identical: silence on the network. In the second case, node B reasons symmetrically and also concludes it is the survivor.
If both then start working, they each start the same virtual machines on the same shared storage. Two systems write in parallel to the same files with no coordination. That is split-brain, and its result is not a conflict to resolve: it is filesystem corruption, usually permanent.
The fix: a third vote, without a third server
You do not need to add a real node. You need to add a vote.
An external witness is a participant that votes without running anything: no shared storage, no virtual machines, no load. The cluster then counts three votes, majority becomes two, and losing a node leaves the survivor in the majority together with the witness. It keeps quorum and carries on.
Two conditions apply to that witness:
- It must be independent of both nodes. Hosted on one of them, it dies with its host and is strictly useless.
- It must be reachable from both. A witness behind the same network equipment as one node reintroduces a common point of failure.
Beyond that it can be tiny: a single-board computer, a small container on a NAS, a machine hosted elsewhere. What it brings is arbitration, not capacity.
Manual override: useful, but understand it
For recovery, you can tell the surviving node that one vote is enough. Sometimes that is the only way to restart a critical service on a Sunday evening.
But this operation disables split-brain protection. It is only acceptable after physically verifying the other node is off — not "it doesn't answer pings", but genuinely powered down. And it must be reverted once things are back to normal. It is never a permanent configuration.
Quorum is not enough if you automate recovery
The last step of the reasoning, often skipped: with a witness, the survivor keeps quorum and can automatically restart the lost node's machines. But it still does not know whether that node is dead or merely isolated.
Hence fencing: the cluster forces the suspect node off — controlled power cut, management card, switched outlet — before restarting its machines elsewhere. You stop assuming it is out of service and make sure of it.
For a homelab, the decision is simple:
- Centralised administration and manual migration: a witness is enough, fencing is unnecessary.
- Automatic restart enabled: without fencing, you can recreate precisely the duplicate quorum was protecting against.
What to remember
Two nodes without a witness is a cluster that freezes at the first incident — the opposite of the goal. Add a third vote: it costs little and transforms how failure day plays out. And only enable automatic recovery if you have a way to guarantee the suspect node is genuinely out of service.
Comments