Messages in this thread |  | | From | Ivan Babrou <> | Date | Wed, 4 Dec 2019 19:46:10 -0800 | Subject | perf not picking up symbols for namespaced processes |
| |
We have a service that forks a child process in a namespace-based sandbox where the mount namespace is intentionally designed to reflect a totally empty filesystem. Our use case is very similar to Chrome's sandbox, for example, but on a server. Within the sandbox, not even the service's own binary is present in the mount namespace.
Process tree looks like this:
$ sudo pstree -psc 63989 edgeworker(63989)─┬─edgeworker/sbox(255716)─┬─edgeworker/zygt(255718) │ ├─{edgeworker/sbox}(255719) │ ├─{edgeworker/sbox}(255720) │ ├─{edgeworker/sbox}(255721) ├─edgeworker/stry(5803) ├─edgeworker/stry(63990) ├─edgeworker/stry(106218) ├─edgeworker/stry(191905) ├─edgeworker/stry(255695) ├─edgeworker/supr(255717)
Here sbox processes do actual work living in an empty mount namespaces and stry is a helper process for error reporting. All tasks come from the same binary that lives in the root mount namespace, launched by systemd.
During "perf script" run on a trace obtained from the system there are these possible outcomes:
1. The first pid to be processed is a non-namespaced helper and symbols are present. 2. The first pid is not found and symbols are present. 3. The first pid is a sandboxed task and symbols are missing.
Symbols are missing, because "perf script" tries to jump into an empty sandbox and find a binary there, when in fact it lives outside:
getcwd("/state/home/ivan", 4096) = 17 open("/proc/self/ns/mnt", O_RDONLY) = 5 open("/proc/255719/ns/mnt", O_RDONLY) = 6 setns(6, CLONE_NEWNS) = 0 stat("/usr/local/bin/edgeworker", 0x7ffedb9b3ca0) = -1 ENOENT (No such file or directory)
In the second outcome we don't have a PID to figure out the namespace to jump into, so this doesn't happen. It's a good fallback, but it was a bit confusing during debugging.
It's not entirely clear to me why sometimes a helper PID is picked, even though it's not the first sample in the recorded trace (at least not in the output). This happens deterministically, or at least appears so. In my process tree it's 255695.
I think perf should try to fallback to the default namespace to look up symbols if they are not found inside to cover our case. Relevant piece of logic is here:
* https://elixir.free-electrons.com/linux/v5.4.1/source/tools/perf/util/dso.c#L520
|  |