lkml.org 
[lkml]   [2019]   [Apr]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: [PATCH] net: add big honking pfmemalloc OOM warning
On Wed, Apr 10, 2019 at 3:20 AM Juha-Matti Tilli
<juha-matti.tilli@foreca.com> wrote:
>
> A system administrator is not notified (except via an obscure SNMP
> counter that most sysadmins don't know to look for) if packets are
> dropped due to out-of-memory condition when SKBs use pfmemalloc
> reserves. This can for example lead to NFS connections hanging on
> high-volume systems.
>
> Implement a ratelimited big honking out of memory warning that directs
> the sysadmin to bump up vm.min_free_kbytes in case this problem happens.
> Our experience shows that with default vm.min_free_kbytes (90112, or
> about 90 megabytes), NFS connections hang approximately once per day,
> whereas with 901120 (default multiplied by 10) NFS connections never
> hang.
>

If NFS sessions hang, then there is a bug to eventually root cause and fix.

Just telling the user : Increase the limit is the same thing than admitting :

Our limit system or TCP or NFS stacks are broken and unable to
recover, so lets disable the limit system and work around a more
serious bug.

Maybe the bug is in a NIC driver, please share more details before
adding yet another noisy signal in syslog

SNMP counters are per netns, and more useful in the modern computing
era, where a host is shared by many different containers.



>
> Signed-off-by: Juha-Matti Tilli <juha-matti.tilli@foreca.com>
> Cc: Eric Dumazet <edumazet@google.com>
> Cc: Rafael Aquini <aquini@redhat.com>
> Cc: Murphy Zhou <xzhou@redhat.com>
> Cc: Yongcheng Yang <yoyang@redhat.com>
> Cc: Jianhong Yin <jiyin@redhat.com>
> ---
> net/core/filter.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index fc92ebc4e200..7d8ef239b9af 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -99,6 +99,8 @@ int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap)
> * helping free memory
> */
> if (skb_pfmemalloc(skb) && !sock_flag(sk, SOCK_MEMALLOC)) {
> + printk_ratelimited(KERN_WARNING
> + "dropped packet due to out-of-memory condition, please bump up vm.min_free_kbytes\n");
> NET_INC_STATS(sock_net(sk), LINUX_MIB_PFMEMALLOCDROP);
> return -ENOMEM;
> }
> --
> 2.17.1
>

\
 
 \ /
  Last update: 2019-04-10 16:17    [W:0.131 / U:0.528 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site