OOM killer: why your process dies with no error message

Key takeaways

• A process killed by the kernel for lack of memory cannot write anything: the signal it receives cannot be caught.
• The evidence is in the kernel logs, not the application's — that is the first place to look.
• The kernel picks its victim by score: the greediest gets hit, not necessarily the guilty one.
• Linux overcommits memory: it promises more than it has, so the shortage only surfaces on actual use.
• Real protection comes from per-service memory limits, which isolate the offender instead of sacrificing a neighbour.

The service stopped responding. You check its logs: nothing. No error, no warning, no shutdown message. The last line is a perfectly ordinary entry, as if the process had evaporated mid-sentence.

That is exactly what happened.

Why the application could not write anything

When the kernel runs so short of memory that it can no longer satisfy an allocation, it picks a process and removes it.

The signal it uses cannot be caught. The process can neither trap it, nor ignore it, nor run the slightest shutdown routine. It does not close its files, does not flush its buffers, does not write a farewell line. It stops, full stop.

Hence the characteristic symptom: application logs ending on an unremarkable line, with no trace whatsoever of what happened.

The evidence is elsewhere — in the kernel logs, where the event is systematically recorded along with the targeted process, its score, and a snapshot of memory at the time. That is the first place to look when a service disappears without explanation.

The victim is almost never the culprit

The kernel gives each candidate a score based mainly on memory consumed, then kills the one whose score is highest.

The logic is defensible: killing the biggest frees the most memory, so it resolves the crisis with a single victim.

But it produces a misleading result. The process that dies is the biggest consumer, not the one that triggered the shortage.

The typical scenario: a badly written script loads a multi-gigabyte file into memory in one go. Memory saturates. The kernel looks at who is consuming the most — and finds the database, which has legitimately been holding a large share of the machine for weeks. It gets killed.

So you go looking for what is wrong with the database. Nothing is: it was simply the most visible.

Overcommit, or why the error comes too late

A natural question: why does the system accept allocations it cannot honour, instead of refusing them cleanly?

Because Linux deliberately overcommits memory. Programs almost always reserve far more than they will use, and refusing those reservations would waste resources on a massive scale.

So the kernel promises beyond its capacity, betting that not everyone will consume at once. That bet pays off the overwhelming majority of the time.

The downside is that the shortage is only observed at the moment of actual use — when a page is genuinely written. By then it is too late to return an allocation error to the offending program: the only way out is to kill someone.

This behaviour is tunable, but making it strict comes at a high price: plenty of programs treat overcommit as a given and fail to start in strict mode. It is not a setting to change without measuring.

Swap: time, not memory

Adding swap is the reflex advice. It is partly relevant.

Swap provides headroom: rarely used pages migrate to disk, which absorbs a one-off spike without an abrupt kill. On a properly sized machine, it prevents pointless incidents.

But if the load exceeds physical memory for any length of time, the system spends its life moving pages between disk and memory. Performance collapses — the machine can end up less responsive than a clean kill would have been — and the process gets killed anyway, after a long agony.

Swap buys time. It does not create memory.

The right protection: per-service limits

You can adjust a process score to make it less likely to be picked. That is a blunt fix: you do not remove the problem, you simply nominate someone else.

The sound approach is to impose a per-service memory limit through control groups. The service manager on most distributions lets you declare it directly in the service definition.

The change is fundamental: a service that exceeds its limit triggers handling inside its own group. The offender is killed, and the other services are never touched.

You move from a model where the shortage randomly hits the biggest, to one where every service owns its own excesses. On a machine hosting several services, it is the only approach that makes incidents predictable.

What to do

1. Confirm the cause in the kernel logs: without that confirmation, you may be chasing a bug that does not exist.
2. Identify what suddenly demanded memory, not what got killed.
3. Set per-service limits so that the next incident hits the offender.
4. Alert on available memory, not just on used memory — cache distorts the reading of the latter.

And if the incident keeps repeating despite correct limits, the conclusion is simple: the machine is undersized for what is being asked of it. No setting durably makes up for a genuine lack of memory.