Linux large folios can trigger Galera disk write spikes
Galera disk write spikes can occur when modern Linux kernels flush large sections of a memory-mapped cache file even though the database changed only a small amount of data. Percona traced this behaviour in PXC, a cluster built on the relational database used by many web applications and using Galera synchronous replication, after an upgrade from PXC 5.7 to 8.0.
The workload and database server settings had not changed, yet every cluster member began producing occasional but regular bursts of heavy disk writes. Percona used pidstat, which reports activity by process or thread, and iostat, which measures storage activity, to establish that the database server process was the only process responsible. The main sources were Galera's replication receiver, connection threads on the writer node and replication applier threads on secondary members. Writes from the transactional storage engine and the binary log remained far too small to explain the total.
During one burst lasting a few seconds, Galera applier threads accounted for 13.0 GB, or 61% of writes. The galera_recv-0 receiver produced 5.2 GB, or 24%. Transactional storage-engine page cleaners wrote 2.0 GB, the redo-log writer 0.76 GB and the Galera service thread 0.29 GB. Despite what looked like several gigabytes of replicated write sets, meaning packaged transaction changes sent between nodes, transactional storage-engine page activity stayed flat.
One initial theory concerned binlog_row_image=full. That setting can make row events include large BLOB, TEXT or JSON columns even when a transaction changes only one column, and the row image format applies to Galera write sets as well as binary logs. Analysis with binlogsum did not support that explanation: binary-log volume was moderate, steady and unrelated to the bursts. A search of changes to the Galera plugin and wsrep replication extensions also found nothing that accounted for the behaviour.
The significant change was the operating system. Debian 11 used the Linux 5.10 LTS kernel series, while Debian 12 brought Linux 6.1 LTS. The affected system also stored Galera's cache on XFS. Unlike most database server data and log files, apart from TempTable storage, Galera cache files use mmap(), which maps a file into memory so an application can modify it as memory rather than calling write() or pwrite() for each change.
Linux introduced page folios in 5.16 to manage groups of memory pages more efficiently. Large folios reached the XFS page cache in kernel 5.18, making them relevant to the move from Linux 5.10 to 6.1. Page cache is memory that Linux uses to retain file data before reading it again or writing changes to storage. Reports of unexpectedly heavy I/O from the systemd journal, another mmap() user, describe related symptoms; a systemd issue covers the problem, although Percona says the underlying cause is still often confused in that community.
With large folios enabled, cached file pages can be grouped into units ranging from 128 kB to as much as 2 MB. In kernels 5.18 through 6.5, write() and pwrite() paths could also be affected. The transactional storage engine commonly opens tablespaces with O_DIRECT, which bypasses page cache, while redo logs and binary logs use small appends that create order-0, 4 kB folios. Linux 6.6 added filesystem-block dirty tracking for write() and pwrite(), allowing the kernel to record overlapping blocks at filesystem-block granularity. That improvement does not cover mmap() updates, which still mark an entire folio. When writeback later copies dirty cached data to disk, a 4 kB modification can therefore cause as much as 2 MB to be written.
The timing also fits Linux writeback rules. Every dirty_writeback_centisecs interval, five seconds by default, the kernel checks whether dirty data is old enough under dirty_expire_centisecs, which defaults to 30 seconds. The observed bursts occurred roughly every 33 seconds, although those settings do not guarantee a fixed schedule. Readahead is another important variable: a device's optimal_io_size and read_ahead_kb settings influence the folio size. Random changes spread across many folios create much more amplification than multiple changes within one folio.
Galera's design makes this particularly visible. Its gcache has always been memory-mapped and defaults to 128 MB, although administrators often enlarge it to retain enough history for Incremental State Transfer (IST), which lets a returning node receive only the transactions it missed. New write sets are appended sequentially, but numerous small, scattered updates mark applied transactions as RELEASED and available for reuse. As the ring buffer wraps, its allocator also walks forward and discards released areas. At startup, recovery scans the entire cache sequentially by default, potentially filling large areas of page cache.
Storage layout can magnify the problem. Linux may raise readahead automatically for software RAID according to the array geometry and advertised optimal I/O size; hardware RAID and device-mapper or Logical Volume Manager stacks may behave similarly. That happened in the investigated case. A system using the usual 128 kB readahead may experience enough amplification to mistake it for normal database traffic, while folios reaching 2 MB can produce conspicuous bursts involving galera.cache and gcache.page.X files.
Percona says there is currently no kernel-side solution for the mmap() path and lists several workarounds. EXT4 can avoid large-folio amplification on applicable older kernels because its support arrived in 6.16, much later than XFS support in 5.18; changing to EXT4 resolved the investigated case. Administrators can lower device readahead, accepting possible performance costs, or developers can apply MADV_RANDOM through madvise() to disable readahead for random mmap access. An application can replace mmap() with pwrite() on Linux 6.6 or later. A custom modern kernel built without CONFIG_TRANSPARENT_HUGEPAGE also prevents creation of large folios, but merely disabling Transparent Huge Pages in GRUB is not sufficient. Percona recorded the PXC-specific issue as PXC-5340.
Checks should cover XFS on Linux 5.18 or later and ext4 on 6.16 or later. Administrators can inspect /sys/block//queue/read_ahead_kb and optimal_io_size, look for FilePmdMapped in /proc/meminfo or the gcache entry in smaps, and watch thp_file_mapped in /proc/vmstat. iostat may reveal bursts about every 30 to 35 seconds, while pidstat -d can attribute writes to Galera threads in multiples as large as 2 MB. Percona also noted a separate case in which combining page-cache writes with O_DIRECT caused stalls in a key-value database engine, showing that large-folio interactions extend beyond Galera.