lkml.org 
[lkml]   [2019]   [Mar]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
From
SubjectBUG-RT: scheduling while in atomic in the watchdog's hrtimer
Date
Hi,

In the 4.19/5.0-rt configured !SMP, the following chain of events can
take place:

------------------------------ %< -------------------------------------
smp_apic_timer_interrupt(){
hrtimer_interrupt() {
__hrtimer_run_queues() {
watchdog_timer_fn() {
stop_one_cpu_nowait() {
#ifdef !CONFIG_SMP
schedule_work() {
queue_work() {
queue_work_on() {
/* phew, long, ah!?.... */
local_lock_irqsave() {
__local_lock_irqsave() {
__local_lock_irq() {
spin_lock_irqsave() {
rt_spin_lock()...
/* from here on you already know...*/
/* a lot of } */
------------------------------ >% -------------------------------------

And this can cause a scheduling while in atomic.

Tracking down, the commit that introduced the stop_one_cpu_nowait()->
queue_work_on() to the path was:

------------------------------ %< -------------------------------------
commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra
Date: Thu Jun 7 10:52:03 2018 +0200

watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work

Oleg suggested to replace the "watchdog/%u" threads with
cpu_stop_work. That removes one thread per CPU while at the same time
fixes softlockup vs SCHED_DEADLINE.

But more importantly, it does away with the single
smpboot_update_cpumask_percpu_thread() user, which allows
cleanups/shrinkage of the smpboot interface.

Suggested-by: Oleg Nesterov
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------

Later modified by:

------------------------------ %< -------------------------------------
commit be45bf5395e0886a93fc816bbe41a008ec2e42e2
Author: Peter Zijlstra
Date: Fri Jul 13 12:42:08 2018 +0200

watchdog/softlockup: Fix cpu_stop_queue_work() double-queue bug

When scheduling is delayed for longer than the softlockup interrupt
period it is possible to double-queue the cpu_stop_work, causing list
corruption.

Cure this by adding a completion to track the cpu_stop_work's
progress.

Reported-by: kernel test robot
Tested-by: Rong Chen
Signed-off-by: Peter Zijlstra (Intel)
Cc: Linus Torvalds
Cc: Peter Zijlstra
Cc: Thomas Gleixner
Fixes: 9cf57731b63e ("watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work")
Link: http://lkml.kernel.org/r/20180713104208.GW2494@hirez.programming.kicks-ass.net
Signed-off-by: Ingo Molnar
------------------------------ >% -------------------------------------

I hit it while trying the automata model against the 4.19-rt/5.0-rt
kernels.

You can find more about it here:
http://bristot.me/another-bug-found-with-the-model/

Thanks
-- Daniel

\
 
 \ /
  Last update: 2019-03-21 22:28    [W:0.040 / U:20.720 seconds]
©2003-2018 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site