lkml.org 
[lkml]   [2018]   [Jan]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
Patch in this message
/
From
Subject[RFC PATCH v1 12/13] mm: split up release_pages into non-sentinel and sentinel passes
Date
A common case in release_pages is for the 'pages' list to be in roughly
the same order as they are in their LRU. With LRU batch locking, when a
sentinel page is removed, an adjacent non-sentinel page must be promoted
to a sentinel page to follow the locking scheme. So we can get behavior
where nearly every page in the 'pages' array is treated as a sentinel
page, hurting the scalability of this approach.

To address this, split up release_pages into non-sentinel and sentinel
passes so that the non-sentinel pages can be locked with an LRU batch
lock before the sentinel pages are removed.

For the prototype, just use a bitmap and a temporary outer loop to
implement this.

Performance numbers from a single microbenchmark at this point in the
series are included in the next patch.

Signed-off-by: Daniel Jordan <daniel.m.jordan@oracle.com>
---
mm/swap.c | 20 +++++++++++++++++++-
1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/mm/swap.c b/mm/swap.c
index fae766e035a4..a302224293ad 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -731,6 +731,7 @@ void lru_add_drain_all(void)
put_online_cpus();
}

+#define LRU_BITMAP_SIZE 512
/**
* release_pages - batched put_page()
* @pages: array of pages to release
@@ -742,16 +743,32 @@ void lru_add_drain_all(void)
*/
void release_pages(struct page **pages, int nr)
{
- int i;
+ int h, i;
LIST_HEAD(pages_to_free);
struct pglist_data *locked_pgdat = NULL;
spinlock_t *locked_lru_batch = NULL;
struct lruvec *lruvec;
unsigned long uninitialized_var(flags);
+ DECLARE_BITMAP(lru_bitmap, LRU_BITMAP_SIZE);
+
+ VM_BUG_ON(nr > LRU_BITMAP_SIZE);

+ bitmap_zero(lru_bitmap, nr);
+
+ for (h = 0; h < 2; h++) {
for (i = 0; i < nr; i++) {
struct page *page = pages[i];

+ if (h == 0) {
+ if (PageLRU(page) && page->lru_sentinel) {
+ bitmap_set(lru_bitmap, i, 1);
+ continue;
+ }
+ } else {
+ if (!test_bit(i, lru_bitmap))
+ continue;
+ }
+
if (is_huge_zero_page(page))
continue;

@@ -798,6 +815,7 @@ void release_pages(struct page **pages, int nr)

list_add(&page->lru, &pages_to_free);
}
+ }
if (locked_lru_batch) {
lru_batch_unlock(NULL, &locked_lru_batch, &locked_pgdat,
&flags);
--
2.16.1
\
 
 \ /
  Last update: 2018-02-01 00:06    [W:0.262 / U:0.288 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site