博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Disabling OOM killer on Ubuntu 14.04
阅读量:6867 次
发布时间:2019-06-26

本文共 6906 字,大约阅读时间需要 23 分钟。

hot3.png

This is a XY problem; that is, you do have a problem (X), whether you know it or not, and believe you can solve things by solving a different problem, which is problem Y.

Why is it that you want to disable the oomkiller? That's the X problem you need to solve. What you should do is investigate and detail problem X. Give the details of that, and it is possible we can help you.

Going after problem Y and disabling the oomkiller is generally a very bad idea which would surelycreate much bigger problems that it might solve.

Invoking the oomkiller is a desperate, last-ditch resort when things have gone awry memory-wise, and doom is imminent. Your system has six passengers, one hour to go, and only five man-hours of oxygen supply. You might think that killing one passenger at random (not at random - ) is a terrible thing to do, but the alternative is having all six passengers dead by suffocation ten long minutes before docking.

Suppose you do that

You succeed in disabling the oomkiller. The system restarts and goes its merry way. Now two things can happen:

  • the oomkiller never requires invocation. You did it all for naught.
  • the oomkiller does require invocation.

But why was the oomkiller required? That was because some process needed memory, and the system discovered that it could not serve that memory.

There is a process which we call "memoryhog", which would have been killed by oomkiller, and now this is not what happens. What does happen?

Option 1: OOM means death

You issued these commands at boot

sysctl vm.panic_on_oom=1sysctl kernel.panic=5

or added this to /etc/sysctl.conf and rebooted

vm.panic_on_oom=1kernel.panic=5

and as soon as the system is hogged, it will panic and reboot after 5 seconds.

Option 2: kill someone else if possible

You classify at boot some processes as "expendable" and some others as "not expendable". For example Apache could be expendable, not so MySQL. So when Apache encounters some absurdly large request (on LAMP systems, usually ensuing from the "easy fix" of setting PHP memory allocation to ludicrously large values instead of changing some memory-bound or leaky approach), the Apache child is killed even if MySQL is a larger hog. This way, the customer will simply see a blank page, hit RELOAD, get another... and finally desist.

You do so by recording MySQL's PID at startup and adjusting its OOM rating:

echo -15 > /proc/$( pidof mysql )/oom_adj

In this situation however I'd also adjust .

Note that if this option solves your problem, it means you have a problem in the other processes (for example, a too-large PHP process. Which derives from a sub-optimal algorithm or data selection stage. You follow the chain until you arrive at the one problem which is really the one to solve).

Probably, installing more RAM would have had the same effect.

In this option, if MySQL itself grows too much (e.g. you botched the innodb_pool_size value), then MySQL will be killed anyway.

Option 3: never ever kill the $MYPRECIOUSSS process

As above, but now the adjustment is set to -17.

I'm now talking about MySQL, but most programs will behave in the same way. I know for sure that both PostgreSQL and Oracle do (and they're entitled to. You abuse RDBMS's at your peril).

This option is dangerous in that oomkiller will kill someone else, possibly repeatedly, but above all, possibly because of $MYPRECIOUSSS itself.

So, a runaway process will evict everyone else from memory until the system is unusable and you can't even log in to check or do anything. Soon after, MySQL will corrupt its own status information and crash horrifically, and you'll be left with a long fsck session, followed by painstaking table repair, followed by careful transaction reconstruction.

I can tell you from my own experience that management will be extremely unhappy with the one that set up what is, in effect, a ticking time bomb. There'll be a lot of blame flying around (MySQL was misconfigured, Apache was leaky, OOMKiller was crazy, it was a hacker...), so "the one" might be "the many". However, for this third option, I cannot stress heavily enough the need of full, adequate and constantly updated and monitored backups, with recovery tests.

Option 4 - "turning off" OOMkiller

This option might look like a . Like the old Windows XP trick to "How to avoid blue screens forever!" - which actually really worked... by setting a  that turned them magenta.

But you can make it so oomkiller is never invoked, by editing the /etc/sysctl.conf file and adding these lines:

vm.overcommit_memory = 2vm.overcommit_kbytes = 0

Then also run the commands sysctl vm.overcommit_memory=2 and sysctl vm.overcommit_kbytes=0 to avoid the need to reboot.

What happens now? If the system has memory enough, nothing happens - just as if you did nothing.

When the system is low on memory, new processes will be unable to start or will crash upon starting. They need memory, and the memory is not there. So they will not be "oomkilled", but instead "killed by out of memory error". The results are absolutely identical, and that's why some might think this a joke.

My own recommendation

Investigate why the system runs low on memory. This can be either inherently unavoidable, remediable, or avoidable entirely. If inherently unavoidable, bump the problem higher and require more memory to be installed.

If it is caused by a runaway process, due to a bug or a faulty input or a condition that can be recognized before criticality, it is remediable. You need to have the process reengineered, and to build a case for that.

It could also happen that the problem is entirely avoidable by splitting services between different servers, or by tuning more carefully one or more processes, or by (again) reengineering the faulty process(es).

In some scenarios, the more efficient solution might be "throw some money at it" - if you've troubles with a server with 256GB RAM and you know you'll need at most 280 for the foreseeable future, an upgrade to 320GB might cost as little as 3,000 USD up front, all inclusive, and be done in the weekend - which, compared to a 150,000 USD/day downtime cost, or four programming man-months at 14,000 USD each, is a real sweet deal, and will likely improve performances even in everyday conditions.

An alternative could be that of setting up a really large swap area, and activate it on demand. Supposing we have 32 GB free in /var/tmp (and that it is not memory-backed):

dd if=/dev/zero of=/var/tmp/oomswap bs=1M count=32768# or: fallocate -l 32g /var/tmp/oomswapchmod 600 /var/tmp/oomswapmkswap /var/tmp/oomswapswapon /var/tmp/oomswap

This will create a 32Gb file useful to give some breath to the various processes. You may also want to inspect and possibly increase swappiness:

cat /proc/sys/vm/swappiness

Ubuntu 14.04 has a default swappiness of 60 which is okay for most users.

sysctl vm.swappiness=70

转载于:https://my.oschina.net/wdyoschina/blog/1456766

你可能感兴趣的文章
优化Android Studio
查看>>
zabbix二次开发-flask-获取告警
查看>>
我的友情链接
查看>>
java实现MD5加密处理
查看>>
实用JVM参数总结
查看>>
oracle 11g R2 64位 安装详细步骤
查看>>
Jpeg 库的解码OpenCL优化
查看>>
码易应用商城入驻流程
查看>>
正则表达式
查看>>
『中级篇』docker之虚拟机创建vagrant技巧(番外篇)(81)
查看>>
交换机SPAN功能配置
查看>>
MySQL 架构组成—存储引擎
查看>>
基于数值分析思想对多项式求值的原理和应用进行探究
查看>>
vue-devtools vue开发调试神器
查看>>
PHP扩展模块的安装
查看>>
BGP基础操作
查看>>
selenium系列->Actions命令实例整理->goBack()
查看>>
CentOS 7上构建squid传统代理,透明代理(squid3.4.6)
查看>>
SimpleXml项目
查看>>
php下使用PDO创建sqlite3数据库
查看>>