Messages in this thread | | | Date | Sat, 4 Dec 2021 11:40:56 +0100 | From | Peter Zijlstra <> | Subject | Re: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs |
| |
On Wed, Dec 01, 2021 at 03:18:44PM +0000, Mel Gorman wrote: > + /* Calculate allowed NUMA imbalance */ > + for_each_cpu(i, cpu_map) { > + int imb_numa_nr = 0; > + > + for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) { > + struct sched_domain *child = sd->child; > + > + if (!(sd->flags & SD_SHARE_PKG_RESOURCES) && child && > + (child->flags & SD_SHARE_PKG_RESOURCES)) { > + int nr_groups; > + > + nr_groups = sd->span_weight / child->span_weight; > + imb_numa_nr = max(1U, ((child->span_weight) >> 1) / > + (nr_groups * num_online_nodes())); > + } > + > + sd->imb_numa_nr = imb_numa_nr; > + }
OK, so let's see. All domains with SHARE_PKG_RESOURCES set will have imb_numa_nr = 0, all domains above it will have the same value calculated here.
So far so good I suppose :-)
Then nr_groups is what it says on the tin; we could've equally well iterated sd->groups and gotten the same number, but this is simpler.
Now, imb_numa_nr is where the magic happens, the way it's written doesn't help, but it's something like:
(child->span_weight / 2) / (nr_groups * num_online_nodes())
With a minimum value of 1. So the larger the system is, or the smaller the LLCs, the smaller this number gets, right?
So my ivb-ep that has 20 cpus in a LLC and 2 nodes, will get: (20 / 2) / (1 * 2) = 10, while the ivb-ex will get: (20/2) / (1*4) = 5.
But a Zen box that has only like 4 CPUs per LLC will have 1, regardless of how many nodes it has.
Now, I'm thinking this assumes (fairly reasonable) that the level above LLC is a node, but I don't think we need to assume this, while also not assuming the balance domain spans the whole machine (yay paritions!).
for (top = sd; top->parent; top = top->parent) ;
nr_llcs = top->span_weight / child->span_weight; imb_numa_nr = max(1, child->span_weight / nr_llcs);
which for my ivb-ep gets me: 20 / (40 / 20) = 10 and the Zen system will have: 4 / (huge number) = 1
Now, the exp: a / (b / a) is equivalent to a * (a / b) or a^2/b, so we can also write the above as:
(child->span_weight * child->span_weight) / top->span_weight;
Hmm?
> + } > + > /* Calculate CPU capacity for physical packages and nodes */ > for (i = nr_cpumask_bits-1; i >= 0; i--) { > if (!cpumask_test_cpu(i, cpu_map)) > -- > 2.31.1 >
| |