MongoDB arbiters can stall writes and weaken durability
MongoDB arbiters can help a replica set elect a primary, but they cannot store, replicate or acknowledge data. Percona says its support cases show that this distinction can turn an apparently healthy production deployment into one that throttles writes, times out or rolls back acknowledged records.
A replica set is a group of MongoDB servers that maintain copies of the same data. MongoDB introduced the arbiter in its early releases in 2010 so operators could create an odd number of votes without paying for another data-bearing server. During an election, the process that selects which member becomes primary, the arbiter casts a vote. It does not apply the oplog, MongoDB's ordered record of database changes, and has never improved data durability.
That limitation became more consequential during the MongoDB 3.x and 4.x eras. WiredTiger became the default storage engine, while the replica-set protocol moved to pv1. MongoDB also made readConcern and writeConcern, settings that control how strongly reads and writes must be confirmed, central to durability. Change streams, multi-document transactions, flow control and cluster-wide default read and write concerns all rely on forms of majority agreement. An arbiter counts toward an election majority, but not toward the data-bearing majority required to acknowledge writes or advance the majority commit point, the latest change durably confirmed by enough replicas.
One Percona support case involved a sharded cluster, which divides data among multiple replica sets. It had three shards and a config-server replica set. Every shard placed three data-bearing members in one data center, three in a second, and an arbiter at a third site. That produced seven voting members per shard, six of which held data. The config servers instead used six data-bearing members, with two at each of three sites and no arbiter.
After one data center failed, elections completed and every shard selected a new primary. Write throughput then fell from roughly 3,000 to 4,000 operations per second on each shard to about 50 to 75. Applications connected through mongos, MongoDB's query router, and used `{ w: 2, j: true }`, requiring confirmation from two members with journal persistence. Three data-bearing members and the arbiter remained reachable on each shard.
The problem was an internal operation rather than the application's selected write concern. Some sharded-cluster tasks, including collection-routing refreshes, issue a local no-op, an operation that changes no user data, with majority write concern. With seven voters and one arbiter, `writeMajorityCount` was four. The arbiter could vote but could not acknowledge that write, so the three surviving data-bearing members could not satisfy the requirement. The majority commit point stopped moving, and routing refreshes waited on a hardcoded timeout of about 60 seconds. The config servers avoided the same failure because four of their six data-bearing members survived, enough for their majority.
A second case showed why an arbiter cannot relieve flow control, MongoDB's mechanism for slowing the primary when replicas fall too far behind. The replica set had four data-bearing members split evenly across two data centers, plus an arbiter at a third site. Two secondaries at one location lagged by as much as about 1,000 seconds during heavy writes that generated a large oplog on the primary. The primary and its local secondary remained current, but advancing the majority commit point required three of the four data-bearing members. With only two caught up, flow control throttled primary writes even though the arbiter remained available.
The most serious example used a Primary-Secondary-Arbiter topology, commonly shortened to PSA. The primary, N1, had election priority 2, and the application used `w: 1`, meaning a write could be acknowledged after reaching only one member. When N1 crashed, the secondary, N2, became primary and accepted acknowledged writes for about an hour. N1 then returned, but before it had caught up, N2 went down. N1 and the arbiter formed a voting majority, allowing N1 to become primary despite lacking that hour of changes.
When N2 rejoined, its oplog contained a newer but conflicting history beyond the last point shared with N1. MongoDB rolled those changes back to match the elected primary. The documents were recoverable only because MongoDB had retained them as BSON files, its binary document format, in the rollback directory under `dbPath`, the database storage location. They had never been replicated elsewhere. A third data-bearing member that remained current could instead have been ahead of N1 and eligible to become primary, although any final writes that had not reached it could still have rolled back.
MongoDB's documented deployment guidance reflects these risks. For three-member sets, Primary-Secondary-Secondary, or PSS, is recommended. PSA is presented as a lower-cost fallback with weaker redundancy and fault tolerance because only two complete data copies exist. An arbiter should not share a host with a primary or secondary, since one host failure would then remove both data and a vote. Multiple arbiters also undermine reliable majority writes. Starting with MongoDB 5.3, support for multiple arbiters in one replica set is disabled by default because a set could accept writes without enough secondaries to replicate them.
Arbiters also restrict transactions. A shard key cannot be changed through a transaction when the replica set contains an arbiter. Because arbiters cannot perform the data operations needed for multi-shard transactions, a transaction spanning multiple shards will error and abort if any operation reads from or writes to a shard containing one. MongoDB Atlas does not support arbiters in any tier.
The practical lesson is to evaluate voting availability and data durability separately. A status screen can show a successful election while the database cannot advance majority commits, keep up with replication or preserve acknowledged writes. Teams should monitor replication lag, write latency and commit-point movement, test complete site failures, and choose write concerns that reflect the importance of the data. For organisations deciding where to operate these systems, AEU Data provides European cloud and data infrastructure, while sound replica-set design remains an operational responsibility.
Replacing an arbiter with a data-bearing member costs more capacity, but it adds another copy capable of acknowledging and preserving writes. Percona's cases show that the cheaper voting member can impose a larger operational cost when writes stall, transactions fail or records survive only in rollback files.