[lkml]   [2022]   [Mar]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
Subject[PATCH v1 0/2] x86: Avoid using INC and DEC instructions on hot paths

In order to take maximum advantage of out-of-order execution,
avoid using INC/DEC instructions when appropriate. INC/DEC only
writes to part of the flags register, which can cause a partial
flag register stall. This series replaces INC/DEC with ADD/SUB.

Agner Fog's optimization manual says [1]:
The INC and DEC instructions are inefficient on some CPUs because they
write to only part of the flags register (excluding the carry flag).
Use ADD or SUB instead to avoid false dependences or inefficient
splitting of the flags register, especially if they are followed by
an instruction that reads the flags.

Intel's optimization manual says [2]:
The INC and DEC instructions modify only a subset of the bits in the
flag register. This creates a dependence on all previous writes of
the flag register. This is especially problematic when these
instructions are on the critical path because they are used to change
an address for a load on which many other instructions depend.

Assembly/Compiler Coding Rule 33. (M impact, H generality) INC and DEC
instructions should be replaced with ADD or SUB instructions, because
ADD and SUB overwrite all flags, whereas INC and DEC do not, therefore
creating false dependencies on earlier instructions that set the flags.

Newer compilers also do it for generic x86-64 CPU (
# C code:

int fy_inc(int a, int b, int c)
a++; b++; c++;
return a * b * c;

## GCC 4.1.2 and older use INC (old).
incl %edi
incl %esi
leal 1(%rdx), %eax
imull %esi, %edi
imull %edi, %eax

## GCC 4.4.7 to GCC 11.2 use ADD (new).
addl $1, %edi
addl $1, %esi
addl $1, %edx
imull %esi, %edi
movl %edi, %eax
imull %edx, %eax

## Clang 5.0.2 and older use INC (old).
incl %edi
leal 1(%rsi), %eax
imull %edi, %eax
incl %edx
imull %edx, %eax

## Clang 6.0.0 to Clang 13.0.1 use ADD (new).
addl $1, %edi
leal 1(%rsi), %eax
imull %edi, %eax
addl $1, %edx
imull %edx, %eax


Signed-off-by: Ammar Faizi <>
Ammar Faizi (2):
x86/include/asm: Avoid using INC and DEC instructions on hot paths
x86/lib: Avoid using INC and DEC instructions on hot paths

arch/x86/include/asm/xor_32.h | 16 ++++++++--------
arch/x86/lib/copy_mc_64.S | 14 +++++++-------
arch/x86/lib/copy_user_64.S | 26 +++++++++++++-------------
arch/x86/lib/memset_64.S | 6 +++---
arch/x86/lib/string_32.c | 20 ++++++++++----------
arch/x86/lib/strstr_32.c | 4 ++--
arch/x86/lib/usercopy_64.c | 12 ++++++------
7 files changed, 49 insertions(+), 49 deletions(-)

base-commit: ffb217a13a2eaf6d5bd974fc83036a53ca69f1e2
Ammar Faizi

 \ /
  Last update: 2022-03-07 12:48    [W:0.432 / U:0.044 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site