lkml.org 
[lkml]   [2015]   [Mar]   [31]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [PATCH RFC] nohz,blk-mq: do not create blk-mq workqueues on nohz dedicated CPUs
On Tue, Mar 31, 2015 at 10:27:26AM -0400, Rik van Riel wrote:
> CPUs with nohz_full do not want disruption from timer interrupts,
> or other random system things. This includes block mq work.
>
> There is another issue with block mq vs. realtime tasks that run
> 100% of the time, which is not uncommon on systems that have CPUs
> dedicated to real time use with isolcpus= and nohz_full=
>
> Specifically, on systems like that, a block work item may never
> get to run, which could lead to filesystems getting stuck forever.
>
> We can avoid both issues by not scheduling blk-mq workqueues on
> cpus in nohz_full mode.
>
> Question for Jens: should we try to spread out the load for
> currently offline and nohz CPUs across the remaining CPUs in
> the system, to get the full benefit of blk-mq in these situations?
>
> If so, do you have any preference on how I should implement that?
>
> Cc: Frederic Weisbecker <fweisbec@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Jens Axboe <axboe@kernel.org>
> Signed-off-by: Rik van Riel <riel@redhat.com>
> ---
> block/blk-mq.c | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 4f4bea21052e..1004d6817fa4 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -21,6 +21,7 @@
> #include <linux/sched/sysctl.h>
> #include <linux/delay.h>
> #include <linux/crash_dump.h>
> +#include <linux/tick.h>
>
> #include <trace/events/block.h>
>
> @@ -1760,6 +1761,10 @@ static void blk_mq_init_cpu_queues(struct request_queue *q,
> if (!cpu_online(i))
> continue;
>
> + /* Do not schedule work on nohz full dedicated CPUs. */
> + if (tick_nohz_full_cpu(i))
> + continue;

I guess in this case, the queue for this CPU will be handled by another CPU?
Is this an unbound workqueue? I guess it's not but if it is, we should wait for
the workqueue affinity patchset.

Also, since we are doing a lot of kernel pre-setting behind nohz full, it would
be nice to warn the user about each of them in dmesg.

> +
> hctx = q->mq_ops->map_queue(q, i);
> cpumask_set_cpu(i, hctx->cpumask);
> hctx->nr_ctx++;
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/


\
 
 \ /
  Last update: 2015-03-31 17:41    [W:0.064 / U:0.940 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site