[lkml]   [2020]   [Mar]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [PATCH v3] virtio-balloon: Switch back to OOM handler for VIRTIO_BALLOON_F_DEFLATE_ON_OOM
On 10.03.20 20:05, David Rientjes wrote:
> On Tue, 10 Mar 2020, David Hildenbrand wrote:
>> Commit 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker")
>> changed the behavior when deflation happens automatically. Instead of
>> deflating when called by the OOM handler, the shrinker is used.
>> However, the balloon is not simply some other slab cache that should be
>> shrunk when under memory pressure. The shrinker does not have a concept of
>> priorities yet, so this behavior cannot be configured. Eventually once
>> that is in place, we might want to switch back after doing proper
>> testing.
>> There was a report that this results in undesired side effects when
>> inflating the balloon to shrink the page cache. [1]
>> "When inflating the balloon against page cache (i.e. no free memory
>> remains) vmscan.c will both shrink page cache, but also invoke the
>> shrinkers -- including the balloon's shrinker. So the balloon
>> driver allocates memory which requires reclaim, vmscan gets this
>> memory by shrinking the balloon, and then the driver adds the
>> memory back to the balloon. Basically a busy no-op."
>> The name "deflate on OOM" makes it pretty clear when deflation should
>> happen - after other approaches to reclaim memory failed, not while
>> reclaiming. This allows to minimize the footprint of a guest - memory
>> will only be taken out of the balloon when really needed.
>> Keep using the shrinker for VIRTIO_BALLOON_F_FREE_PAGE_HINT, because
>> this has no such side effects. Always register the shrinker with
>> VIRTIO_BALLOON_F_FREE_PAGE_HINT now. We are always allowed to reuse free
>> pages that are still to be processed by the guest. The hypervisor takes
>> care of identifying and resolving possible races between processing a
>> hinting request and the guest reusing a page.
>> In contrast to pre commit 71994620bb25 ("virtio_balloon: replace oom
>> notifier with shrinker"), don't add a moodule parameter to configure the
>> number of pages to deflate on OOM. Can be re-added if really needed.
>> Also, pay attention that leak_balloon() returns the number of 4k pages -
>> convert it properly in virtio_balloon_oom_notify().
>> Testing done by Tyler for future reference:
>> Test setup: VM with 16 CPU, 64GB RAM. Running Debian 10. We have a 42
>> GB file full of random bytes that we continually cat to /dev/null.
>> This fills the page cache as the file is read. Meanwhile we trigger
>> the balloon to inflate, with a target size of 53 GB. This setup causes
>> the balloon inflation to pressure the page cache as the page cache is
>> also trying to grow. Afterwards we shrink the balloon back to zero (so
>> total deflate = total inflate).
>> Without patch (kernel 4.19.0-5):
>> Inflation never reaches the target until we stop the "cat file >
>> /dev/null" process. Total inflation time was 542 seconds. The longest
>> period that made no net forward progress was 315 seconds (see attached
>> graph).
>> Result of "grep balloon /proc/vmstat" after the test:
>> balloon_inflate 154828377
>> balloon_deflate 154828377
>> With patch (kernel 5.6.0-rc4+):
>> Total inflation duration was 63 seconds. No deflate-queue activity
>> occurs when pressuring the page-cache.
>> Result of "grep balloon /proc/vmstat" after the test:
>> balloon_inflate 12968539
>> balloon_deflate 12968539
>> Conclusion: This patch fixes the issue. In the test it reduced
>> inflate/deflate activity by 12x, and reduced inflation time by 8.6x.
>> But more importantly, if we hadn't killed the "grep balloon
>> /proc/vmstat" process then, without the patch, the inflation process
>> would never reach the target.
>> [1]
>> Reported-by: Tyler Sanderson <>
>> Tested-by: Tyler Sanderson <>
>> Fixes: 71994620bb25 ("virtio_balloon: replace oom notifier with shrinker")
>> Cc: Michael S. Tsirkin <>
>> Cc: Wei Wang <>
>> Cc: Alexander Duyck <>
>> Cc: David Rientjes <>
>> Cc: Nadav Amit <>
>> Cc: Michal Hocko <>
>> Cc: Andrew Morton <>
>> Signed-off-by: David Hildenbrand <>
> Acked-by: David Rientjes <>


> Should this have:
> Cc: # 4.19+

I guess as nothing will actually "crash" it's not worth stable.


David / dhildenb

 \ /
  Last update: 2020-03-10 20:13    [W:0.069 / U:0.960 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site