lkml.org 
[lkml]   [2021]   [Dec]   [28]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
SubjectRe: [PATCH v3 0/9] Parallel CPU bringup for x86_64
From
Dear David,


Am 27.12.21 um 17:57 schrieb Paul Menzel:

> Am 15.12.21 um 15:56 schrieb David Woodhouse:
>> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for
>> them shaves about 80% off the AP bringup time on a 96-thread socket
>> Skylake box (EC2 c5.metal) — from about 500ms to 100ms.
>>
>> There are more wins to be had with further parallelisation, but this is
>> the simple part.
>>
>> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
>> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
>>      in preparation for more parallelisation.
>>
>>
>> David Woodhouse (8):
>>        x86/apic/x2apic: Fix parallel handling of cluster_mask
>>        cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
>>        cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU
>>        x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
>>        x86/smpboot: Split up native_cpu_up into separate phases and document them
>>        x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
>>        x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup
>>        x86/smpboot: Serialize topology updates for secondary bringup
>>
>> Thomas Gleixner (1):
>>        x86/smpboot: Support parallel startup of secondary CPUs
>>
>>   arch/x86/include/asm/realmode.h       |   3 +
>>   arch/x86/include/asm/smp.h            |  13 +-
>>   arch/x86/include/asm/topology.h       |   2 -
>>   arch/x86/kernel/acpi/sleep.c          |   1 +
>>   arch/x86/kernel/apic/apic.c           |   2 +-
>>   arch/x86/kernel/apic/x2apic_cluster.c | 108 +++++++-----
>>   arch/x86/kernel/cpu/common.c          |   6 +-
>>   arch/x86/kernel/cpu/mtrr/mtrr.c       |   9 +
>>   arch/x86/kernel/head_64.S             |  71 ++++++++
>>   arch/x86/kernel/smpboot.c             | 324 ++++++++++++++++++++++++----------
>>   arch/x86/realmode/init.c              |   3 +
>>   arch/x86/realmode/rm/trampoline_64.S  |  14 ++
>>   arch/x86/xen/smp_pv.c                 |   4 +-
>>   include/linux/cpuhotplug.h            |   2 +
>>   include/linux/smpboot.h               |   7 +
>>   kernel/cpu.c                          |  27 ++-
>>   kernel/smpboot.c                      |   2 +-
>>   kernel/smpboot.h                      |   2 -
>>   18 files changed, 441 insertions(+), 159 deletions(-)
>
> Thank you for working on this. I tested this on a MSI MS-7A37/B350M
> MORTAR (BIOS 1.MW 11/01/2021) with a Ryzen 3 2200G, but nothing was
> printed to the screen after the GRUB loading messages, so it crashed or
> hung somewhere. Unfortunately, this device is used by others, and no
> serial console is connected and I do not know how to capture the Linux
> log with other means.

Same on the ASUS F2A85-M PRO with AMD A6-6400K. Without serial console,
the messages below are printed below to the monitor after nine seconds.

[ 1.078879] smp: Bringing up secondary CPUs ...
[ 1.080950] x86: Booting SMP configuration:

Please find the serial log attached.


Kind regards,

Paul[ 0.000000] Linux version 5.16.0-rc7-00106-gcc498e0c43be (root@45e877da5b3e) (gcc (Debian 11.2.0-12) 11.2.0, GNU ld (GNU Binutils for Debian) 2.37) #245 SMP PREEMPT Tue Dec 28 10:00:33 UTC 2021
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be root=/dev/sda3 rw debug noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 console=ttyS0,115200 console=tty1 earlyprintk=serial,ttyS0,115200,keep
[ 0.000000] random: get_random_u32 called from bsp_init_amd+0x142/0x210 with crng_init=0
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] signal: max sigframe size: 1776
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000005fe45fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000005fe46000-0x000000007fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000017effffff] usable
[ 0.000000] printk: console [earlyser0] enabled
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 3.0.0 present.
[ 0.000000] DMI: ASUS F2A85-M_PRO/F2A85-M_PRO, BIOS 4.15-676-g90cfb8f5ef 12/28/2021
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] tsc: Initial usec timer 20439600
[ 0.000000] tsc: Detected 3900.178 MHz processor
[ 0.000588] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.007106] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.012655] last_pfn = 0x17f000 max_arch_pfn = 0x400000000
[ 0.018249] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
Memory KASLR using RDTSC...
[ 0.027700] last_pfn = 0x5fe46 max_arch_pfn = 0x400000000
[ 0.036861] Using GB pages for direct mapping
[ 0.041210] ACPI: Early table checksum verification disabled
[ 0.046691] ACPI: RSDP 0x00000000000F6250 000024 (v02 COREv4)
[ 0.052409] ACPI: XSDT 0x000000005FE4C0E0 000074 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.060905] ACPI: FACP 0x000000005FE4DBC0 000114 (v06 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.069398] ACPI: DSDT 0x000000005FE4C280 00193A (v02 COREv4 COREBOOT 00010001 INTL 20200925)
[ 0.077890] ACPI: FACS 0x000000005FE4C240 000040
[ 0.082483] ACPI: FACS 0x000000005FE4C240 000040
[ 0.087077] ACPI: SSDT 0x000000005FE4DCE0 00008A (v02 COREv4 COREBOOT 0000002A CORE 20200925)
[ 0.095570] ACPI: MCFG 0x000000005FE4DD70 00003C (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.104064] ACPI: APIC 0x000000005FE4DDB0 000062 (v03 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.112557] ACPI: HPET 0x000000005FE4DE20 000038 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.121051] ACPI: HEST 0x000000005FE4DE60 0001D0 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.129544] ACPI: IVRS 0x000000005FE4E030 000070 (v02 AMD AMDIOMMU 00000001 AMD 00000000)
[ 0.138037] ACPI: SSDT 0x000000005FE4E0A0 00051F (v02 AMD ALIB 00000001 MSFT 04000000)
[ 0.146531] ACPI: SSDT 0x000000005FE4E5C0 0006B2 (v01 AMD POWERNOW 00000001 AMD 00000001)
[ 0.155025] ACPI: VFCT 0x000000005FE4EC80 00F269 (v01 COREv4 COREBOOT 00000000 CORE 20200925)
[ 0.163517] ACPI: Reserving FACP table memory at [mem 0x5fe4dbc0-0x5fe4dcd3]
[ 0.170537] ACPI: Reserving DSDT table memory at [mem 0x5fe4c280-0x5fe4dbb9]
[ 0.177558] ACPI: Reserving FACS table memory at [mem 0x5fe4c240-0x5fe4c27f]
[ 0.184578] ACPI: Reserving FACS table memory at [mem 0x5fe4c240-0x5fe4c27f]
[ 0.191598] ACPI: Reserving SSDT table memory at [mem 0x5fe4dce0-0x5fe4dd69]
[ 0.198619] ACPI: Reserving MCFG table memory at [mem 0x5fe4dd70-0x5fe4ddab]
[ 0.205638] ACPI: Reserving APIC table memory at [mem 0x5fe4ddb0-0x5fe4de11]
[ 0.212659] ACPI: Reserving HPET table memory at [mem 0x5fe4de20-0x5fe4de57]
[ 0.219679] ACPI: Reserving HEST table memory at [mem 0x5fe4de60-0x5fe4e02f]
[ 0.226699] ACPI: Reserving IVRS table memory at [mem 0x5fe4e030-0x5fe4e09f]
[ 0.233719] ACPI: Reserving SSDT table memory at [mem 0x5fe4e0a0-0x5fe4e5be]
[ 0.240740] ACPI: Reserving SSDT table memory at [mem 0x5fe4e5c0-0x5fe4ec71]
[ 0.247760] ACPI: Reserving VFCT table memory at [mem 0x5fe4ec80-0x5fe5dee8]
[ 0.254835] No NUMA configuration found
[ 0.258593] Faking a node at [mem 0x0000000000000000-0x000000017effffff]
[ 0.265273] NODE_DATA(0) allocated [mem 0x17efe7000-0x17effdfff]
[ 0.283316] Zone ranges:
[ 0.285678] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.291830] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[ 0.297984] Normal [mem 0x0000000100000000-0x000000017effffff]
[ 0.304138] Device empty
[ 0.306998] Movable zone start for each node
[ 0.311245] Early memory node ranges
[ 0.314798] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.321039] node 0: [mem 0x0000000000100000-0x000000005fe45fff]
[ 0.327278] node 0: [mem 0x0000000100000000-0x000000017effffff]
[ 0.333520] Initmem setup node 0 [mem 0x0000000000001000-0x000000017effffff]
[ 0.340544] On node 0, zone DMA: 1 pages in unavailable ranges
[ 0.340602] On node 0, zone DMA: 97 pages in unavailable ranges
[ 0.359000] On node 0, zone Normal: 442 pages in unavailable ranges
[ 0.364808] On node 0, zone Normal: 4096 pages in unavailable ranges
[ 0.371106] ACPI: PM-Timer IO Port: 0x818
[ 0.381304] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
[ 0.387198] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[ 0.394039] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.400367] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[ 0.406868] ACPI: Using ACPI (MADT) for SMP configuration information
[ 0.413280] ACPI: HPET id: 0x10228210 base: 0xfed00000
[ 0.418398] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[ 0.423333] smpboot: smpboot: XXX end of prefill_possible_map
[ 0.429053] After prefill_possible_map
[ 0.432781] After init_cpu_to_node
[ 0.436160] After init_gi_nodes
[ 0.439281] After io_apic_init_mappings
[ 0.443094] After x86_init.hyper.guest_late_init
[ 0.447696] [mem 0x80000000-0xf7ffffff] available for PCI devices
[ 0.453754] After e820
[ 0.456096] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[ 0.470502] After unwind_init
[ 0.473298] After setup_arch
[ 0.476169] After setup_command_line
[ 0.479711] After setup_nr_cpu_ids
[ 0.483091] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:2 nr_node_ids:1
[ 0.491127] percpu: Embedded 54 pages/cpu s182040 r8192 d30952 u1048576
[ 0.497579] pcpu-alloc: s182040 r8192 d30952 u1048576 alloc=1*2097152
[ 0.503978] pcpu-alloc: [0] 0 1
[ 0.507209] After setup_per_cpu_areas
[ 0.510826] After smp_perpare_boot_cpu
[ 0.514553] After boot_cpu_hotplug_init
[ 0.518368] Fallback order for Node 0: 0
[ 0.522352] Built 1 zonelists, mobility grouping on. Total pages: 898444
[ 0.529113] Policy zone: Normal
[ 0.532233] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be root=/dev/sda3 rw debug noisapnp cryptomgr.notests ipv6.disable_ipv6=1 selinux=0 console=ttyS0,115200 console=tty1 earlyprintk=serial,ttyS0,115200,keep
[ 0.553561] Unknown kernel command line parameters "noisapnp BOOT_IMAGE=/boot/vmlinuz-5.16.0-rc7-00106-gcc498e0c43be", will be passed to user space.
[ 0.567513] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[ 0.575640] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[ 0.583229] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.629839] Memory: 3483292K/3651472K available (14344K kernel code, 2321K rwdata, 4212K rodata, 1692K init, 6332K bss, 167920K reserved, 0K cma-reserved)
[ 0.643901] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.650282] After mm_init
[ 0.652850] ftrace: allocating 35324 entries in 138 pages
[ 0.670177] ftrace: allocated 138 pages with 3 groups
[ 0.675169] Dynamic Preempt: full
[ 0.678348] After sched_init
[ 0.681282] rcu: Preemptible hierarchical RCU implementation.
[ 0.686928] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2.
[ 0.693515] Trampoline variant of Tasks RCU enabled.
[ 0.698541] Rude variant of Tasks RCU enabled.
[ 0.703048] Tracing variant of Tasks RCU enabled.
[ 0.707815] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[ 0.715441] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[ 0.722125] After rcu_init
[ 0.734046] NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16
[ 0.740046] rcu: Offload RCU callbacks from CPUs: (none).
[ 0.745386] random: crng_init_try_arch_early failed with i = 4, X86_FEATURE_RDRAND = no
[ 0.745388] random: crng_init_try_arch_early failed with i = 5, X86_FEATURE_RDRAND = no
[ 0.753324] random: crng_init_try_arch_early failed with i = 6, X86_FEATURE_RDRAND = no
[ 0.761299] random: crng_init_try_arch_early failed with i = 7, X86_FEATURE_RDRAND = no
[ 0.769272] random: crng_init_try_arch_early failed with i = 8, X86_FEATURE_RDRAND = no
[ 0.777245] random: crng_init_try_arch_early failed with i = 9, X86_FEATURE_RDRAND = no
[ 0.785218] random: crng_init_try_arch_early failed with i = 10, X86_FEATURE_RDRAND = no
[ 0.793192] random: crng_init_try_arch_early failed with i = 11, X86_FEATURE_RDRAND = no
[ 0.801252] random: crng_init_try_arch_early failed with i = 12, X86_FEATURE_RDRAND = no
[ 0.809313] random: crng_init_try_arch_early failed with i = 13, X86_FEATURE_RDRAND = no
[ 0.817372] random: crng_init_try_arch_early failed with i = 14, X86_FEATURE_RDRAND = no
[ 0.825432] random: crng_init_try_arch_early failed with i = 15, X86_FEATURE_RDRAND = no
[ 0.833494] After add_latent_entropy
[ 0.845109] After add_device_randomness
[ 0.848921] After boot_init_stack_canary
[ 0.852875] spurious 8259A interrupt: IRQ7.
[ 0.854860] Console: colour VGA+ 80x25
[ 0.866354] printk: console [tty1] enabled
[ 0.870342] ACPI: Core revision 20210930
[ 0.874423] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484873504 ns
[ 0.883411] APIC: Switch to symmetric I/O mode setup
[ 0.923446] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.933411] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x7070070e77e, max_idle_ns: 881591209168 ns
[ 0.943779] Calibrating delay loop (skipped), value calculated using timer frequency.. 7800.35 BogoMIPS (lpj=3900178)
[ 0.944776] pid_max: default: 32768 minimum: 301
[ 0.945884] LSM: Security Framework initializing
[ 0.946890] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.947791] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
Poking KASLR using RDTSC...
[ 0.952654] Bit 30 in CPUID ECX not set.
[ 0.952681] Last level iTLB entries: 4KB 512, 2MB 1024, 4MB 512
[ 0.953775] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 512, 1GB 0
[ 0.954780] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 0.955776] Spectre V2 : Mitigation: Full AMD retpoline
[ 0.956775] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[ 0.957776] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[ 0.958776] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl
[ 0.963298] Freeing SMP alternatives memory: 40K
[ 0.963777] After check_bugs
[ 0.964776] After acpi_subsystem_init
[ 0.965776] After arch_post_acpi_subsys_init
[ 0.966776] After rcu_scheduler_starting
[ 0.967851] After find_task_by_pid_ns and PF_NO_SETAFFINITY
[ 0.968781] After numa_default_policy
[ 0.969801] After rcu_read_lock
[ 0.970775] After rcu_read_unlock
[ 0.971776] After kthreadd_done
[ 0.972786] smpboot: Start of smp_prepare_cpus_common
[ 0.973777] smpboot: smpboot: zalloc 0
[ 0.974776] smpboot: smpboot: zalloc 1
[ 0.975775] smpboot: smpboot: After set_sched_topology()
[ 0.976777] smpboot: smpboot: After smp_sanity_check()
[ 0.977775] smpboot: smpboot: Before x86_init.timers.setup_percpu_clockev()
[ 0.997775] random: random: 1
[ 0.998775] random: random: 2
[ 0.998775] random: random: 3
[ 0.998775] random: random: 4
[ 1.061775] random: random: 1
[ 1.062775] random: random: 2
[ 1.062775] random: random: 3
[ 1.062775] random: random: 4
[ 1.062808] APIC calibration not consistent with PM-Timer: 102ms instead of 100ms
[ 1.063775] APIC delta adjusted to PM-Timer: 625036 (640760)
[ 1.063780] smpboot: smpboot: After x86_init.timers.setup_percpu_clockev()
[ 1.064775] smpboot: smp_get_logical_apicid()
[ 1.065775] smpboot: CPU0: AMD A6-6400K APU with Radeon(tm) HD Graphics (family: 0x15, model: 0x13, stepping: 0x1)
[ 1.067103] Performance Events: Fam15h core perfctr, AMD PMU driver.
[ 1.067777] ... version: 0
[ 1.068775] ... bit width: 48
[ 1.069775] ... generic registers: 6
[ 1.070777] ... value mask: 0000ffffffffffff
[ 1.071775] ... max period: 00007fffffffffff
[ 1.072775] ... fixed-purpose events: 0
[ 1.073775] ... event mask: 000000000000003f
[ 1.075812] rcu: Hierarchical SRCU implementation.
[ 1.078397] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[ 1.078879] smp: Bringing up secondary CPUs ...
[ 1.080950] x86: Booting SMP configuration:
\
 
 \ /
  Last update: 2021-12-28 12:35    [W:0.145 / U:0.116 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site