[lkml]   [2019]   [Jan]   [30]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: [LSF/MM TOPIC]: userfaultfd (was: [LSF/MM TOPIC] NUMA remote THP vs NUMA local non-THP under MADV_HUGEPAGE)
On Wed, Jan 30, 2019 at 10:13:36AM +0200, Mike Rapoport wrote:
> Hi,
> (changed the subject and added CRIU folks)
> On Tue, Jan 29, 2019 at 06:40:58PM -0500, Andrea Arcangeli wrote:
> > Hello,
> >
> > --
> >
> > In addition to the above "NUMA remote THP vs NUMA local non-THP
> > tradeoff" topic, there are other developments in "userfaultfd" land that
> > are approaching merge readiness and that would be possible to provide a
> > short overview about:
> >
> > - Peter Xu made significant progress in finalizing the userfaultfd-WP
> > support over the last few months. That feature was planned from the
> > start and it will allow userland to do some new things that weren't
> > possible to achieve before. In addition to synchronously blocking
> > write faults to be resolved by an userland manager, it has also the
> > ability to obsolete the softdirty feature, because it can provide
> > the same information, but with O(1) complexity (as opposed of the
> > current softdirty O(N) complexity) similarly to what the Page
> > Modification Logging (PML) does in hardware for EPT write accesses.
> We (CRIU) have some concerns about obsoleting soft-dirty in favor of
> uffd-wp. If there are other soft-dirty users these concerns would be
> relevant to them as well.
> With soft-dirty we collect the information about the changed memory every
> pre-dump iteration in the following manner:
> * freeze the tasks
> * find entries in /proc/pid/pagemap with SOFT_DIRTY set
> * unfreeze the tasks
> * dump the modified pages to disk/remote host
> While we do need to traverse the /proc/pid/pagemap to identify dirty pages,
> in between the pre-dump iterations and during the actual memory dump the
> tasks are running freely.
> If we are to switch to uffd-wp, every write by the snapshotted/migrated
> task will incur latency of uffd-wp processing by the monitor.
> We'd need to see how this affects overall slowdown of the workload under
> migration before moving forward with obsoleting soft-dirty.
> > - Blake Caldwell maintained the UFFDIO_REMAP support to atomically
> > remove memory from a mapping with userfaultfd (which can't be done
> > with a copy as in UFFDIO_COPY and it requires a slow TLB flush to be
> > safe) as an alternative to host swapping (which of course also
> > requires a TLB flush for similar reasons). Notably UFFDIO_REMAP was
> > rightfully naked early on and quickly replaced by UFFDIO_COPY which
> > is more optimal to add memory to a mapping is small chunks, but we
> > can't remove memory with UFFDIO_COPY and UFFDIO_REMAP should be as
> > efficient as it gets when it comes to removing memory from a
> > mapping.
> If we are to discuss userfaultfd, I'd like also to bring the subject of COW
> mappings.
> The pages populated with UFFDIO_COPY cannot be COW-shared between related
> processes which unnecessarily increases memory footprint of a migrated
> process tree.
> I've posted a patch [1] a (real) while ago, but nobody reacted and I've put
> this aside.
> Maybe it's time to discuss it again :)

Hi, Mike,

It's interesting to know such a work...

Since I really don't have much context on this, so sorry if I'm going
to ask a silly question... but I'd say when reading this I'm thinking
of KSM. I think KSM does not suite in this case since when doing
UFFDIO_COPY_COW it'll contain hinting information while KSM was only
scanning over the pages between processes which seems to be O(N*N) if
assuming there're two processes. However, would it make any sense to
provide a general interface to scan for same pages between any two
processes within specific range and merge them if found (rather than a
specific interface for userfaultfd only)? Then it might even be used
by KSM admins (just as an example) when the admin knows exactly that
memory range (addr1, len) of process A should very probably has many
same contents as the memory range (addr2, len) of process B?


Peter Xu

 \ /
  Last update: 2019-01-30 10:24    [W:0.097 / U:2.672 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site