[lkml]   [2019]   [Dec]   [6]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
SubjectRe: Injecting delays into block layer

> Il giorno 21 nov 2019, alle ore 09:00, Paolo Valente <> ha scritto:
>> Il giorno 21 nov 2019, alle ore 08:13, Oleksandr Natalenko <> ha scritto:
>> Hi Paolo et al.
> Hi
>> I have a strong suspect that something is going wrong when the underlying block device responds with a large delay. What makes me thinking so is that I use a VM on some cloud provider, and they have substantial block device latency resulting in permanently high (~20%) iowait. It spikes occasionally when their cluster is overloaded, and when that happens, the I/O in my VM may stop and never recover. This is a rare occasion, but it really happens.
>> What's worse, so far I've seen such a behaviour with BFQ only. I'm still testing other schedulers though.
>> Important note: I have no strict evidences that this is *the* case, thus I'm asking for some suggestions. My idea is to fire up a local VM and inject delays to a block device while performing some I/O from within the VM.
>> So the question is: how can those delays be injected? Using dm-delay? Can those delays be random?
> So far I have used scsi_debug [1] for this kind of tests. In my S
> suite [2], it boils down to setting SCSI_DEBUG=yes in the S config
> file, and then launching any of the benchmarks. Unfortunately, AFAIK
> scsi_debug gives you only constant delays; but you can emulate delay
> spikes very easily, by changing the delay parameter manually during
> the test.
> If this option sounds reasonable to you, then I'm willing to help you
> for every step.

Hi Oleksandr,
Simone (in CC) and I have worked a little bit on reproducing the I/O
freeze you report. Simone made a small change in SCSI_debug, which
makes the latter serve I/O with a highly varying random delay (100ms -
1s), about twice a second.

Then, to generate some fluctuating and heavy I/O, he ran the script of my S suite with SCSI_debug a few times.
Unfortunately, he didn't succeed in reproducing the problem. If you
want, we can send you a patch with his change for SCSI_debug.

Any news on your side?


> Thanks,
> Paolo
> [1]
> [2]
>> Thanks in advance.
>> --
>> Oleksandr Natalenko (post-factum)

 \ /
  Last update: 2019-12-06 17:18    [W:0.082 / U:6.128 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site