[lkml]   [2020]   [Feb]   [26]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[PATCH v3 0/2] kstats: kernel metric collector

This patchset introduces a small library to collect per-cpu samples and
accumulate distributions to be exported through debugfs.

This v3 series addresses some initial comments (mostly style fixes in the
code) and revises commit logs.

Note that use case and performance are orthogonal to those of the
perf/tracing architecture, which is why this is proposed as a standalone

I have used this to trace the execution times of small sections of code
(network stack functions, drivers and xdp processing, ...), whole functions,
or notification latencies (e.g. IPI dispatch, etc.). The fine granularity
of the aggregation helps identifying outliers, multimodal distributions,
caching effects and general performance-related issues.

Samples are 64-bit values that are aggregated into per-cpu logarithmic buckets
with configurable density (typical sample collectors use one buckets for
each power of 2, powers of 2, but that is very coarse and corresponds
to 1 significant bit per value; in quickstats one can specify the number
of significant bits, up to 5, which is useful for finer grain measurements).

There are two ways to trace a block of code: manual annotations has the best
performance at runtime and is done as follows:

// create metric and entry in debugfs
struct kstats *key = kstats_new("foo", frac_bits);

// instrument the code
u64 t0 = ktime_get_ns(); // about 20ns
<section of code to measure>
t0 = ktime_get_ns() - t0; // about 20ns
kstats_record(key, t0); // 5ns hot cache, 300ns cold

This method has an accuracy of about 20-30ns (inherited from the clock)
and an overhead with hot/cold cache as show above.

Values can be read from debugfs in an easy to parse format

# cat /sys/kernel/debug/kstats/foo
slot 55 CPU 0 count 589 avg 480 p 0.027613
slot 55 CPU 1 count 18 avg 480 p 0.002572
slot 55 CPU 2 count 25 avg 480 p 0.003325
slot 55 CPUS 28 count 814 avg 480 p 0.002474
slot 97 CPU 13 count 1150 avg 20130 p 0.447442
slot 97 CPUS 28 count 152585 avg 19809 p 0.651747

Writing STOP, START, RESET to the file executes the corresponding action

echo RESET > /sys/kernel/debug/kstats/foo

The second instrumentation mechanism uses kretrprobes or tracepoints and
lets tracing be enabled or removed at runtime from the command line for
any globally visible function

echo trace some_function bits 3 > /sys/kernel/debug/kstats/_control

Data are exported or controlled in the same way as above. Accuracy is
worse due to the presence of kretprobe trampolines: 90ns with hot cache,
500ns with cold cache. The overhead on the traced function is 250ns hot,
1500ns cold.

Hope you find this useful.

Luigi Rizzo (2):
kstats: kernel metric collector
kstats: kretprobe and tracepoint support

include/linux/kstats.h | 62 ++++
lib/Kconfig.debug | 7 +
lib/Makefile | 1 +
lib/kstats.c | 654 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 724 insertions(+)
create mode 100644 include/linux/kstats.h
create mode 100644 lib/kstats.c


 \ /
  Last update: 2020-02-26 14:51    [W:0.114 / U:1.088 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site