[lkml]   [2019]   [Jan]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[LSF/MM TOPIC]: userfaultfd (was: [LSF/MM TOPIC] NUMA remote THP vs NUMA local non-THP under MADV_HUGEPAGE)

(changed the subject and added CRIU folks)

On Tue, Jan 29, 2019 at 06:40:58PM -0500, Andrea Arcangeli wrote:
> Hello,
> --
> In addition to the above "NUMA remote THP vs NUMA local non-THP
> tradeoff" topic, there are other developments in "userfaultfd" land that
> are approaching merge readiness and that would be possible to provide a
> short overview about:
> - Peter Xu made significant progress in finalizing the userfaultfd-WP
> support over the last few months. That feature was planned from the
> start and it will allow userland to do some new things that weren't
> possible to achieve before. In addition to synchronously blocking
> write faults to be resolved by an userland manager, it has also the
> ability to obsolete the softdirty feature, because it can provide
> the same information, but with O(1) complexity (as opposed of the
> current softdirty O(N) complexity) similarly to what the Page
> Modification Logging (PML) does in hardware for EPT write accesses.

We (CRIU) have some concerns about obsoleting soft-dirty in favor of
uffd-wp. If there are other soft-dirty users these concerns would be
relevant to them as well.

With soft-dirty we collect the information about the changed memory every
pre-dump iteration in the following manner:
* freeze the tasks
* find entries in /proc/pid/pagemap with SOFT_DIRTY set
* unfreeze the tasks
* dump the modified pages to disk/remote host

While we do need to traverse the /proc/pid/pagemap to identify dirty pages,
in between the pre-dump iterations and during the actual memory dump the
tasks are running freely.

If we are to switch to uffd-wp, every write by the snapshotted/migrated
task will incur latency of uffd-wp processing by the monitor.

We'd need to see how this affects overall slowdown of the workload under
migration before moving forward with obsoleting soft-dirty.

> - Blake Caldwell maintained the UFFDIO_REMAP support to atomically
> remove memory from a mapping with userfaultfd (which can't be done
> with a copy as in UFFDIO_COPY and it requires a slow TLB flush to be
> safe) as an alternative to host swapping (which of course also
> requires a TLB flush for similar reasons). Notably UFFDIO_REMAP was
> rightfully naked early on and quickly replaced by UFFDIO_COPY which
> is more optimal to add memory to a mapping is small chunks, but we
> can't remove memory with UFFDIO_COPY and UFFDIO_REMAP should be as
> efficient as it gets when it comes to removing memory from a
> mapping.

If we are to discuss userfaultfd, I'd like also to bring the subject of COW
The pages populated with UFFDIO_COPY cannot be COW-shared between related
processes which unnecessarily increases memory footprint of a migrated
process tree.
I've posted a patch [1] a (real) while ago, but nobody reacted and I've put
this aside.
Maybe it's time to discuss it again :)

> Thank you,
> Andrea


Sincerely yours,

 \ /
  Last update: 2019-01-30 09:14    [W:0.089 / U:4.728 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site