mirror-linux/arch/x86/kernel
Linus Torvalds 94a855111e - Add the call depth tracking mitigation for Retbleed which has
been long in the making. It is a lighterweight software-only fix for
 Skylake-based cores where enabling IBRS is a big hammer and causes a
 significant performance impact.
 
 What it basically does is, it aligns all kernel functions to 16 bytes
 boundary and adds a 16-byte padding before the function, objtool
 collects all functions' locations and when the mitigation gets applied,
 it patches a call accounting thunk which is used to track the call depth
 of the stack at any time.
 
 When that call depth reaches a magical, microarchitecture-specific value
 for the Return Stack Buffer, the code stuffs that RSB and avoids its
 underflow which could otherwise lead to the Intel variant of Retbleed.
 
 This software-only solution brings a lot of the lost performance back,
 as benchmarks suggest:
 
   https://lore.kernel.org/all/20220915111039.092790446@infradead.org/
 
 That page above also contains a lot more detailed explanation of the
 whole mechanism
 
 - Implement a new control flow integrity scheme called FineIBT which is
 based on the software kCFI implementation and uses hardware IBT support
 where present to annotate and track indirect branches using a hash to
 validate them
 
 - Other misc fixes and cleanups
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEEzv7L6UO9uDPlPSfHEsHwGGHeVUoFAmOZp5EACgkQEsHwGGHe
 VUrZFxAAvi/+8L0IYSK4mKJvixGbTFjxN/Swo2JVOfs34LqGUT6JaBc+VUMwZxdb
 VMTFIZ3ttkKEodjhxGI7oGev6V8UfhI37SmO2lYKXpQVjXXnMlv/M+Vw3teE38CN
 gopi+xtGnT1IeWQ3tc/Tv18pleJ0mh5HKWiW+9KoqgXj0wgF9x4eRYDz1TDCDA/A
 iaBzs56j8m/FSykZHnrWZ/MvjKNPdGlfJASUCPeTM2dcrXQGJ93+X2hJctzDte0y
 Nuiw6Y0htfFBE7xoJn+sqm5Okr+McoUM18/CCprbgSKYk18iMYm3ZtAi6FUQZS1A
 ua4wQCf49loGp15PO61AS5d3OBf5D3q/WihQRbCaJvTVgPp9sWYnWwtcVUuhMllh
 ZQtBU9REcVJ/22bH09Q9CjBW0VpKpXHveqQdqRDViLJ6v/iI6EFGmD24SW/VxyRd
 73k9MBGrL/dOf1SbEzdsnvcSB3LGzp0Om8o/KzJWOomrVKjBCJy16bwTEsCZEJmP
 i406m92GPXeaN1GhTko7vmF0GnkEdJs1GVCZPluCAxxbhHukyxHnrjlQjI4vC80n
 Ylc0B3Kvitw7LGJsPqu+/jfNHADC/zhx1qz/30wb5cFmFbN1aRdp3pm8JYUkn+l/
 zri2Y6+O89gvE/9/xUhMohzHsWUO7xITiBavewKeTP9GSWybWUs=
 =cRy1
 -----END PGP SIGNATURE-----

Merge tag 'x86_core_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 core updates from Borislav Petkov:

 - Add the call depth tracking mitigation for Retbleed which has been
   long in the making. It is a lighterweight software-only fix for
   Skylake-based cores where enabling IBRS is a big hammer and causes a
   significant performance impact.

   What it basically does is, it aligns all kernel functions to 16 bytes
   boundary and adds a 16-byte padding before the function, objtool
   collects all functions' locations and when the mitigation gets
   applied, it patches a call accounting thunk which is used to track
   the call depth of the stack at any time.

   When that call depth reaches a magical, microarchitecture-specific
   value for the Return Stack Buffer, the code stuffs that RSB and
   avoids its underflow which could otherwise lead to the Intel variant
   of Retbleed.

   This software-only solution brings a lot of the lost performance
   back, as benchmarks suggest:

       https://lore.kernel.org/all/20220915111039.092790446@infradead.org/

   That page above also contains a lot more detailed explanation of the
   whole mechanism

 - Implement a new control flow integrity scheme called FineIBT which is
   based on the software kCFI implementation and uses hardware IBT
   support where present to annotate and track indirect branches using a
   hash to validate them

 - Other misc fixes and cleanups

* tag 'x86_core_for_v6.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (80 commits)
  x86/paravirt: Use common macro for creating simple asm paravirt functions
  x86/paravirt: Remove clobber bitmask from .parainstructions
  x86/debug: Include percpu.h in debugreg.h to get DECLARE_PER_CPU() et al
  x86/cpufeatures: Move X86_FEATURE_CALL_DEPTH from bit 18 to bit 19 of word 11, to leave space for WIP X86_FEATURE_SGX_EDECCSSA bit
  x86/Kconfig: Enable kernel IBT by default
  x86,pm: Force out-of-line memcpy()
  objtool: Fix weak hole vs prefix symbol
  objtool: Optimize elf_dirty_reloc_sym()
  x86/cfi: Add boot time hash randomization
  x86/cfi: Boot time selection of CFI scheme
  x86/ibt: Implement FineIBT
  objtool: Add --cfi to generate the .cfi_sites section
  x86: Add prefix symbols for function padding
  objtool: Add option to generate prefix symbols
  objtool: Avoid O(bloody terrible) behaviour -- an ode to libelf
  objtool: Slice up elf_create_section_symbol()
  kallsyms: Revert "Take callthunks into account"
  x86: Unconfuse CONFIG_ and X86_FEATURE_ namespaces
  x86/retpoline: Fix crash printing warning
  x86/paravirt: Fix a !PARAVIRT build warning
  ...
2022-12-14 15:03:00 -08:00
..
acpi x86/acpi/cstate: Optimize ARB_DISABLE on Centaur CPUs 2022-11-11 09:42:05 -08:00
apic A set of changes for the x86 APIC code: 2022-12-12 12:30:31 -08:00
cpu - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
fpu * Clarify XSAVE consistency warnings 2022-12-12 14:41:57 -08:00
kprobes x86/modules: Set VM_FLUSH_RESET_PERMS in module_alloc() 2022-10-17 16:40:57 +02:00
.gitignore
Makefile - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
alternative.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
amd_gart_64.c x86: Fix various duplicate-word comment typos 2022-08-15 19:17:52 +02:00
amd_nb.c x86/amd_nb: Add AMD PCI IDs for SMN communication 2022-07-20 17:35:40 +02:00
aperture_64.c x86: Fix various duplicate-word comment typos 2022-08-15 19:17:52 +02:00
apm_32.c x86/ibt: Disable IBT around firmware 2022-03-15 10:32:40 +01:00
asm-offsets.c Linux 6.1-rc6 2022-11-21 23:01:51 +01:00
asm-offsets_32.c
asm-offsets_64.c x86: Fixup asm-offsets duplicate 2022-10-17 16:41:06 +02:00
audit_64.c audit: add support for the openat2 syscall 2021-10-01 16:52:48 -04:00
bootflag.c
callthunks.c x86/ftrace: Make it call depth tracking aware 2022-10-17 16:41:19 +02:00
cfi.c x86: Add support for CONFIG_CFI_CLANG 2022-09-26 10:13:16 -07:00
check.c
cpuid.c
crash.c x86/crash: Fix minor typo/bug in debug message 2022-04-13 19:39:54 +02:00
crash_core_32.c
crash_core_64.c
crash_dump_32.c vmcore: convert copy_oldmem_page() to take an iov_iter 2022-04-29 14:37:59 -07:00
crash_dump_64.c use less confusing names for iov_iter direction initializers 2022-11-25 13:01:55 -05:00
devicetree.c x86/of: Add support for boot time interrupt delivery mode configuration 2022-12-02 14:57:14 +01:00
doublefault_32.c exit/doublefault: Remove apparently bogus comment about rewind_stack_do_exit 2021-10-20 13:09:43 -05:00
dumpstack.c - Yu Zhao's Multi-Gen LRU patches are here. They've been under test in 2022-10-10 17:53:04 -07:00
dumpstack_32.c x86/percpu: Move irq_stack variables next to current_task 2022-10-17 16:41:05 +02:00
dumpstack_64.c x86/percpu: Move irq_stack variables next to current_task 2022-10-17 16:41:05 +02:00
e820.c x86/kexec: Carry forward IMA measurement log on kexec 2022-07-01 15:22:16 +02:00
early-quirks.c drm/i915/rpl-p: Add PCI IDs 2022-04-19 17:14:09 -07:00
early_printk.c x86/earlyprintk: Clean up pciserial 2022-08-29 12:19:25 +02:00
ebda.c
eisa.c
espfix_64.c x86/espfix: Use get_random_long() rather than archrandom 2022-10-31 20:12:50 +01:00
ftrace.c x86/ftrace: Make it call depth tracking aware 2022-10-17 16:41:19 +02:00
ftrace_32.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
ftrace_64.S Merge branch 'x86/urgent' into x86/core, to resolve conflict 2022-10-22 10:06:18 +02:00
head32.c
head64.c x86: Clear .brk area at early boot 2022-07-01 11:11:34 +02:00
head_32.S x86/asm/32: Remove setup_once() 2022-12-02 14:06:34 +01:00
head_64.S x86/callthunks: Add call patching for call depth tracking 2022-10-17 16:41:13 +02:00
hpet.c rtc: Check return value from mc146818_get_time() 2021-12-16 21:50:06 +01:00
hw_breakpoint.c
i8237.c
i8253.c
i8259.c x86/i8259: Make default_legacy_pic static 2022-10-31 10:20:32 +01:00
idt.c x86/traps: Add #VE support for TDX guest 2022-04-07 08:27:51 -07:00
io_delay.c
ioport.c
irq.c x86/irq: Ensure PI wakeup handler is unregistered before module unload 2021-10-22 12:45:35 -04:00
irq_32.c x86/percpu: Move irq_stack variables next to current_task 2022-10-17 16:41:05 +02:00
irq_64.c x86/percpu: Move irq_stack variables next to current_task 2022-10-17 16:41:05 +02:00
irq_work.c
irqflags.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
irqinit.c
itmt.c x86/sched: Decrease further the priorities of SMT siblings 2021-10-05 15:51:59 +02:00
jailhouse.c
jump_label.c jump_label: make initial NOP patching the special case 2022-06-24 09:48:55 +02:00
kdebugfs.c x86/boot: Fix memremap of setup_indirect structures 2022-03-09 12:49:44 +01:00
kexec-bzimage64.c integrity-v6.0 2022-08-02 15:21:18 -07:00
kgdb.c
ksysfs.c x86/boot: Fix memremap of setup_indirect structures 2022-03-09 12:49:44 +01:00
kvm.c x86/paravirt: Use common macro for creating simple asm paravirt functions 2022-11-24 13:56:44 +01:00
kvmclock.c x86/kvm: Don't waste kvmclock memory if there is nopv parameter 2022-04-13 13:37:19 -04:00
ldt.c memcg: enable accounting for ldt_struct objects 2021-09-03 09:58:13 -07:00
machine_kexec_32.c
machine_kexec_64.c x86/kexec: fix memory leak of elf header buffer 2022-06-01 15:57:16 -07:00
mmconf-fam10h_64.c
module.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
mpparse.c
msr.c
nmi.c x86/nmi: Make register_nmi_handler() more robust 2022-05-17 09:25:25 +02:00
nmi_selftest.c
paravirt-spinlocks.c
paravirt.c x86/paravirt: Use common macro for creating simple asm paravirt functions 2022-11-24 13:56:44 +01:00
pci-dma.c swiotlb: merge swiotlb-xen initialization into swiotlb 2022-04-18 07:21:13 +02:00
pcspeaker.c
perf_regs.c
platform-quirks.c
pmem.c x86/pmem: Fix platform-device leak in error path 2022-06-20 18:01:16 +02:00
probe_roms.c x86/kernel: Validate ROM memory before accessing when SEV-SNP is active 2022-04-06 13:23:09 +02:00
process.c Random number generator updates for Linux 6.2-rc1. 2022-12-12 16:22:22 -08:00
process.h x86: Snapshot thread flags 2021-12-01 00:06:43 +01:00
process_32.c x86/percpu: Move current_top_of_stack next to current_task 2022-10-17 16:41:05 +02:00
process_64.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
ptrace.c x86: Improve formatting of user_regset arrays 2022-11-01 15:36:52 -07:00
pvclock.c
quirks.c
reboot.c x86: Use do_kernel_power_off() 2022-05-19 19:30:31 +02:00
reboot_fixups_32.c
relocate_kernel_32.S x86/kexec: Disable RET on kexec 2022-07-09 13:12:32 +02:00
relocate_kernel_64.S x86/callthunks: Add call patching for call depth tracking 2022-10-17 16:41:13 +02:00
resource.c x86/PCI: Tidy E820 removal messages 2022-12-10 10:33:11 -06:00
rethook.c x86,rethook: Fix arch_rethook_trampoline() to generate a complete pt_regs 2022-03-28 19:38:51 -07:00
rtc.c x86/rtc: Rename mach_set_rtc_mmss() to mach_set_cmos_time() 2022-08-14 11:24:29 +02:00
setup.c - Split MTRR and PAT init code to accomodate at least Xen PV and TDX 2022-12-13 14:56:56 -08:00
setup_percpu.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
sev-shared.c Revert "x86/sev: Expose sev_es_ghcb_hv_call() for use by HyperV" 2022-07-27 18:09:13 +02:00
sev.c x86/sev: Mark snp_abort() noreturn 2022-08-25 15:54:03 +02:00
sev_verify_cbit.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
signal.c x86/signal/64: Move 64-bit signal code to its own file 2022-10-19 09:58:49 +02:00
signal_32.c x86/signal/32: Merge native and compat 32-bit signal code 2022-10-19 09:58:49 +02:00
signal_64.c x86/signal/64: Move 64-bit signal code to its own file 2022-10-19 09:58:49 +02:00
signal_compat.c signal: Deliver SIGTRAP on perf event asynchronously if blocked 2022-04-22 12:14:05 +02:00
smp.c
smpboot.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
stacktrace.c x86: remove __range_not_ok() 2022-02-25 09:36:05 +01:00
static_call.c static_call: Add call depth tracking support 2022-10-17 16:41:16 +02:00
step.c ptrace: Reimplement PTRACE_KILL by always sending SIGKILL 2022-05-11 14:34:28 -05:00
sys_ia32.c
sys_x86_64.c x86/mm: Cleanup the control_va_addr_alignment() __setup handler 2022-05-04 18:20:42 +02:00
tboot.c mm: remove rb tree. 2022-09-26 19:46:16 -07:00
time.c
tls.c
tls.h
topology.c x86/cpu: Switch to cpu_feature_enabled() for X86_FEATURE_XENPV 2022-11-22 16:18:19 +01:00
trace.c trace/osnoise: Fix an ifdef comment 2021-10-25 23:02:36 -04:00
trace_clock.c
tracepoint.c x86/traceponit: Fix comment about irq vector tracepoints 2022-05-26 22:03:52 -04:00
traps.c - Add the call depth tracking mitigation for Retbleed which has 2022-12-14 15:03:00 -08:00
tsc.c x86/tsc: Make art_related_clocksource static 2022-10-17 16:20:48 +02:00
tsc_msr.c
tsc_sync.c x86/tsc: Add a timer to make sure TSC_adjust is always checked 2021-12-02 00:40:35 +01:00
umip.c x86/umip: Downgrade warning messages to debug loglevel 2021-09-25 13:23:28 +02:00
unwind_frame.c x86: kmsan: don't instrument stack walking functions 2022-10-03 14:03:25 -07:00
unwind_guess.c x86/unwind: Recover kretprobe trampoline entry 2021-09-30 21:24:07 -04:00
unwind_orc.c Linux 6.1-rc6 2022-11-21 23:01:51 +01:00
uprobes.c uprobes/x86: Allow to probe a NOP instruction with 0x66 prefix 2022-12-05 11:55:18 +01:00
verify_cpu.S x86: Prepare asm files for straight-line-speculation 2021-12-08 12:25:37 +01:00
vm86_32.c x86/32: Remove lazy GS macros 2022-04-14 14:09:43 +02:00
vmlinux.lds.S x86/ibt: Implement FineIBT 2022-11-01 13:44:10 +01:00
vsmp_64.c
x86_init.c x86/boot: Skip realmode init code when running as Xen PV guest 2022-11-25 12:05:22 +01:00