[lkml]   [2020]   [Feb]   [14]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    Patch in this message
    Subject[PATCH AUTOSEL 5.5 011/542] rcu: Fix missed wakeup of exp_wq waiters
    From: Neeraj Upadhyay <>

    [ Upstream commit fd6bc19d7676a060a171d1cf3dcbf6fd797eb05f ]

    Tasks waiting within exp_funnel_lock() for an expedited grace period to
    elapse can be starved due to the following sequence of events:

    1. Tasks A and B both attempt to start an expedited grace
    period at about the same time. This grace period will have
    completed when the lower four bits of the rcu_state structure's
    ->expedited_sequence field are 0b'0100', for example, when the
    initial value of this counter is zero. Task A wins, and thus
    does the actual work of starting the grace period, including
    acquiring the rcu_state structure's .exp_mutex and sets the
    counter to 0b'0001'.

    2. Because task B lost the race to start the grace period, it
    waits on ->expedited_sequence to reach 0b'0100' inside of
    exp_funnel_lock(). This task therefore blocks on the rcu_node
    structure's ->exp_wq[1] field, keeping in mind that the
    end-of-grace-period value of ->expedited_sequence (0b'0100')
    is shifted down two bits before indexing the ->exp_wq[] field.

    3. Task C attempts to start another expedited grace period,
    but blocks on ->exp_mutex, which is still held by Task A.

    4. The aforementioned expedited grace period completes, so that
    ->expedited_sequence now has the value 0b'0100'. A kworker task
    therefore acquires the rcu_state structure's ->exp_wake_mutex
    and starts awakening any tasks waiting for this grace period.

    5. One of the first tasks awakened happens to be Task A. Task A
    therefore releases the rcu_state structure's ->exp_mutex,
    which allows Task C to start the next expedited grace period,
    which causes the lower four bits of the rcu_state structure's
    ->expedited_sequence field to become 0b'0101'.

    6. Task C's expedited grace period completes, so that the lower four
    bits of the rcu_state structure's ->expedited_sequence field now
    become 0b'1000'.

    7. The kworker task from step 4 above continues its wakeups.
    Unfortunately, the wake_up_all() refetches the rcu_state
    structure's .expedited_sequence field:

    wake_up_all(&rnp->exp_wq[rcu_seq_ctr(rcu_state.expedited_sequence) & 0x3]);

    This results in the wakeup being applied to the rcu_node
    structure's ->exp_wq[2] field, which is unfortunate given that
    Task B is instead waiting on ->exp_wq[1].

    On a busy system, no harm is done (or at least no permanent harm is done).
    Some later expedited grace period will redo the wakeup. But on a quiet
    system, such as many embedded systems, it might be a good long time before
    there was another expedited grace period. On such embedded systems,
    this situation could therefore result in a system hang.

    This issue manifested as DPM device timeout during suspend (which
    usually qualifies as a quiet time) due to a SCSI device being stuck in
    _synchronize_rcu_expedited(), with the following stack trace:


    This commit therefore prevents such delays, timeouts, and hangs by
    making rcu_exp_wait_wake() use its "s" argument consistently instead of
    refetching from rcu_state.expedited_sequence.

    Fixes: 3b5f668e715b ("rcu: Overlap wakeups with next expedited grace period")
    Signed-off-by: Neeraj Upadhyay <>
    Signed-off-by: Paul E. McKenney <>
    Signed-off-by: Sasha Levin <>
    kernel/rcu/tree_exp.h | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
    index 69c5aa64fcfd6..f504ac8317797 100644
    --- a/kernel/rcu/tree_exp.h
    +++ b/kernel/rcu/tree_exp.h
    @@ -558,7 +558,7 @@ static void rcu_exp_wait_wake(unsigned long s)
    smp_mb(); /* All above changes before wakeup. */
    - wake_up_all(&rnp->exp_wq[rcu_seq_ctr(rcu_state.expedited_sequence) & 0x3]);
    + wake_up_all(&rnp->exp_wq[rcu_seq_ctr(s) & 0x3]);
    trace_rcu_exp_grace_period(, s, TPS("endwake"));
     \ /
      Last update: 2020-02-14 19:26    [W:2.253 / U:3.812 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site