You set up ZFS, everything works, then you look at memory usage: 90% consumed on a machine that is not doing much. The reflex is to hunt for a leak. There isn't one — but the usual reassurance, "it's just cache, ignore it", is incomplete. On a hypervisor, that cache can genuinely cause you trouble.
The ARC in one minute
The ARC (Adaptive Replacement Cache) is the ZFS read cache. It keeps the most useful blocks in memory using two combined criteria: what was read recently and what is read frequently. That dual logic is what sets it apart from a plain cache: a large sequential scan is not enough to evict genuinely hot data.
By default it targets roughly half the machine's memory, and it gives that memory back when an application asks. Unused RAM serves nobody: filling it with cache is desirable behaviour, not a defect.
On a dedicated file server, then, there is nothing to do. Let it work.
Why a hypervisor is different
The "the ARC gives memory back on demand" reasoning is accurate, but it skips a decisive variable: speed.
A hypervisor does not request memory gradually. When a VM boots, it asks for several gigabytes at once and immediately. The ARC then has to free memory — which takes time, all the more so when it is large and memory is fragmented. If the allocation fails before that completes, the boot is refused.
The symptom is confusing: a VM refuses to start while the machine appears to have headroom, and the problem vanishes a few minutes later. This is not a memory shortage, it is a release latency problem.
Sizing without wrecking performance
The reasoning is simple: start from total memory, subtract what VMs and containers can claim simultaneously at worst, keep headroom for the host, and give the rest to the ARC.
Two symmetrical mistakes to avoid:
- Capping far too low "to be safe". The cache hit rate collapses, every read goes back to the disks, and performance drops visibly. An ARC of a few hundred megabytes on a busy hypervisor is counter-productive.
- Not capping at all on a machine where VMs start and stop regularly. You expose yourself to the allocation problem above, intermittently and in a way that is hard to diagnose.
The setting is verified, not guessed. Kernel statistics expose current ARC size, its target, and above all the cache hit rate. That last value should drive your adjustment: as long as it stays high, the cap is fine. If it collapses after tightening, raise it.
The gigabyte-per-terabyte myth
"ZFS needs 1 GB of RAM per terabyte of storage" comes up constantly. It is false in the general case, and the belief discourages perfectly viable deployments.
The rule comes from deduplication, which maintains a hash table that must stay in memory and whose size grows with stored data. Once it no longer fits in RAM, performance collapses abruptly.
But deduplication is discouraged for almost every workload: its cost is permanent, its gains marginal except on massively redundant data, and compression — enabled by default, nearly free in CPU — usually delivers a better result.
Without deduplication, ZFS runs fine with a few gigabytes of RAM across several terabytes of data. More memory improves reads; it is not a prerequisite.
SSD cache: consider it last
Since RAM is short, why not add an SSD as a second-level cache? This is often a false good idea.
That cache needs an index held in RAM. It therefore consumes the very resource that was missing, and that RAM is no longer available to the ARC, which is orders of magnitude faster than an SSD. On a machine already tight on memory, adding one frequently degrades performance rather than improving it.
The correct order is: max out RAM first, measure the ARC hit rate second, and only consider an SSD cache if the active data set durably exceeds installable memory.
Takeaway
RAM at 90% on a ZFS server is not a problem: it is the expected behaviour. On a hypervisor, though, the question is not "how much does the ARC consume" but "how fast can it give it back". Cap according to your allocation peaks, verify with the cache hit rate, and forget the gigabyte-per-terabyte rule.
Frequently asked questions
Does ZFS really need 1 GB of RAM per terabyte?
Should I cap the ARC on a plain ZFS server?
Why does a VM refuse to start when there is still free memory?
How do I check what the ARC is actually using?
/proc/spl/kstat/zfs/arcstats. The values that matter are current size, target size and maximum allowed size. A high cache hit rate means the ARC is working effectively; if it collapses after you cap, you have squeezed too hard. That measurement, not the global memory display, should drive the setting.
Comments