lkml.org 
[lkml]   [2019]   [May]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
Date
SubjectRe: 5.1 kernel: khugepaged stuck at 100%
On Thu, May 9, 2019 at 5:54 AM Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
> Hello,
>
> Kernel: 5.1 (self-compiled, no modules)
> Arch: x86_64
> Distro: Debian Testing
>
> Issue: I was performing a dump of ext3 and ext4 filesystems and then
> restoring them to a separate volume (testing)-- afterwards I noticed that
> khugepaged is stuck at 100% CPU. It is currently still stuck at 100% CPU, is
> there anything I can do to debug what is happening prior to a reboot to work
> around the issue? I have never seen this behavior prior to 5.1.
>
> $ cat /proc/cmdline
> auto BOOT_IMAGE=5.1.0-2 ro root=901 3w-sas.use_msi=1 nohugeiomap
> page_poison=1 pcie_aspm=off pti=on rootfstype=ext4 slub_debug=P
> zram.enabled=1 zram.num_devices=12 zswap.enabled=1 zswap.compressor=lz4
> zswap.zpool=z3fold
>
> $ 5.1 .config attached: config.txt.gz
>
> $ attachment: graphic.JPG -> graph of the processes, dark green ->
> khugepaged
>
> $ top
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 77 root 39 19 0 0 0 R 100.0 0.0 23:29.27
> khugepaged
> 1 root 20 0 171800 7832 4948 S 0.0 0.0 1:25.59 systemd
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02
> kthreadd
> 3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
> 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> rcu_par_gp
> 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> kworker/0+
> 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
> mm_percpu+
>

Will disable zcache/zswap for now, but FWIW

From perf top:

PerfTop: 1181 irqs/sec kernel:83.1% exact: 96.9% lost: 73/137
drop: 0/0 [4000Hz cycles], (all, 12 CPUs)
-------------------------------------------------------------------------------

54.56% [kernel] [k] compaction_alloc
29.46% [kernel] [k] __pageblock_pfn_to_page
10.66% [kernel] [k] nmi
0.61% [kernel] [k] _cond_resched
0.32% [kernel] [k] format_decode
0.25% [kernel] [k] __lock_text_start
0.20% [kernel] [k] __vma_adjust
0.16% perf [.] __symbols__insert
0.13% perf [.] rb_insert_color_cached

I tried dropping caches, etc, I will disable huge pages for now.

Some other posts:
https://bugzilla.redhat.com/show_bug.cgi?id=879801
https://lkml.org/lkml/2012/6/27/565
https://lkml.org/lkml/2011/11/9/252

\
 
 \ /
  Last update: 2019-05-09 13:18    [W:0.090 / U:1.048 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site