superrob1500
/
mirror-linux
mirror of https://github.com/torvalds/linux
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Crypto library updates for 7.1 

- Migrate more hash algorithms from the traditional crypto subsystem
   to lib/crypto/.
 
   Like the algorithms migrated earlier (e.g. SHA-*), this simplifies
   the implementations, improves performance, enables further
   simplifications in calling code, and solves various other issues:
 
     - AES CBC-based MACs (AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC)
 
         - Support these algorithms in lib/crypto/ using the AES
           library and the existing arm64 assembly code
 
         - Reimplement the traditional crypto API's "cmac(aes)",
           "xcbc(aes)", and "cbcmac(aes)" on top of the library
 
         - Convert mac80211 to use the AES-CMAC library. Note: several
           other subsystems can use it too and will be converted later
 
         - Drop the broken, nonstandard, and likely unused support for
           "xcbc(aes)" with key lengths other than 128 bits
 
         - Enable optimizations by default
 
     - GHASH
 
         - Migrate the standalone GHASH code into lib/crypto/
 
         - Integrate the GHASH code more closely with the very similar
           POLYVAL code, and improve the generic GHASH implementation
           to resist cache-timing attacks and use much less memory
 
         - Reimplement the AES-GCM library and the "gcm" crypto_aead
           template on top of the GHASH library. Remove "ghash" from
           the crypto_shash API, as it's no longer needed
 
         - Enable optimizations by default
 
     - SM3
 
         - Migrate the kernel's existing SM3 code into lib/crypto/, and
           reimplement the traditional crypto API's "sm3" on top of it
 
         - I don't recommend using SM3, but this cleanup is worthwhile
           to organize the code the same way as other algorithms
 
 - Testing improvements
 
     - Add a KUnit test suite for each of the new library APIs
 
     - Migrate the existing ChaCha20Poly1305 test to KUnit
 
     - Make the KUnit all_tests.config enable all crypto library tests
 
     - Move the test kconfig options to the Runtime Testing menu
 
 - Other updates to arch-optimized crypto code
 
     - Optimize SHA-256 for Zhaoxin CPUs using the Padlock Hash Engine
 
     - Remove some MD5 implementations that are no longer worth keeping
 
     - Drop big endian and voluntary preemption support from the arm64
       code, as those configurations are no longer supported on arm64
 
 - Make jitterentropy and samples/tsm-mr use the crypto library APIs
 
 Note: the overall diffstat is neutral, but when the test code is
 excluded it is significantly negative:
 
     Tests:     13 files changed, 1982 insertions(+),  888 deletions(-)
     Non-test: 141 files changed, 2897 insertions(+), 3987 deletions(-)
     All:      154 files changed, 4879 insertions(+), 4875 deletions(-)
 -----BEGIN PGP SIGNATURE-----
 
 iIoEABYIADIWIQSacvsUNc7UX4ntmEPzXCl4vpKOKwUCadWPyxQcZWJpZ2dlcnNA
 a2VybmVsLm9yZwAKCRDzXCl4vpKOK8QCAQD0i98miI1mu01RKuEwrBzmn7L/2sUH
 ReYV/dFDtnN0GwD+KMCiNAM2XTVLRKq5t3OxPHpKZ4y+gZwRowAJeFA02Q8=
 =5rip
 -----END PGP SIGNATURE-----

Merge tag 'libcrypto-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux

Pull crypto library updates from Eric Biggers:

 - Migrate more hash algorithms from the traditional crypto subsystem to
   lib/crypto/

   Like the algorithms migrated earlier (e.g. SHA-*), this simplifies
   the implementations, improves performance, enables further
   simplifications in calling code, and solves various other issues:

     - AES CBC-based MACs (AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC)

         - Support these algorithms in lib/crypto/ using the AES library
           and the existing arm64 assembly code

         - Reimplement the traditional crypto API's "cmac(aes)",
           "xcbc(aes)", and "cbcmac(aes)" on top of the library

         - Convert mac80211 to use the AES-CMAC library. Note: several
           other subsystems can use it too and will be converted later

         - Drop the broken, nonstandard, and likely unused support for
           "xcbc(aes)" with key lengths other than 128 bits

         - Enable optimizations by default

     - GHASH

         - Migrate the standalone GHASH code into lib/crypto/

         - Integrate the GHASH code more closely with the very similar
           POLYVAL code, and improve the generic GHASH implementation to
           resist cache-timing attacks and use much less memory

         - Reimplement the AES-GCM library and the "gcm" crypto_aead
           template on top of the GHASH library. Remove "ghash" from the
           crypto_shash API, as it's no longer needed

         - Enable optimizations by default

     - SM3

         - Migrate the kernel's existing SM3 code into lib/crypto/, and
           reimplement the traditional crypto API's "sm3" on top of it

         - I don't recommend using SM3, but this cleanup is worthwhile
           to organize the code the same way as other algorithms

 - Testing improvements:

     - Add a KUnit test suite for each of the new library APIs

     - Migrate the existing ChaCha20Poly1305 test to KUnit

     - Make the KUnit all_tests.config enable all crypto library tests

     - Move the test kconfig options to the Runtime Testing menu

 - Other updates to arch-optimized crypto code:

     - Optimize SHA-256 for Zhaoxin CPUs using the Padlock Hash Engine

     - Remove some MD5 implementations that are no longer worth keeping

     - Drop big endian and voluntary preemption support from the arm64
       code, as those configurations are no longer supported on arm64

 - Make jitterentropy and samples/tsm-mr use the crypto library APIs

* tag 'libcrypto-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux: (66 commits)
  lib/crypto: arm64: Assume a little-endian kernel
  arm64: fpsimd: Remove obsolete cond_yield macro
  lib/crypto: arm64/sha3: Remove obsolete chunking logic
  lib/crypto: arm64/sha512: Remove obsolete chunking logic
  lib/crypto: arm64/sha256: Remove obsolete chunking logic
  lib/crypto: arm64/sha1: Remove obsolete chunking logic
  lib/crypto: arm64/poly1305: Remove obsolete chunking logic
  lib/crypto: arm64/gf128hash: Remove obsolete chunking logic
  lib/crypto: arm64/chacha: Remove obsolete chunking logic
  lib/crypto: arm64/aes: Remove obsolete chunking logic
  lib/crypto: Include <crypto/utils.h> instead of <crypto/algapi.h>
  lib/crypto: aesgcm: Don't disable IRQs during AES block encryption
  lib/crypto: aescfb: Don't disable IRQs during AES block encryption
  lib/crypto: tests: Migrate ChaCha20Poly1305 self-test to KUnit
  lib/crypto: sparc: Drop optimized MD5 code
  lib/crypto: mips: Drop optimized MD5 code
  lib: Move crypto library tests to Runtime Testing menu
  crypto: sm3 - Remove 'struct sm3_state'
  crypto: sm3 - Remove the original "sm3_block_generic()"
  crypto: sm3 - Remove sm3_base.h
  ...
master
Linus Torvalds 2026-04-13 17:31:39 -07:00
parent 9932f00bf4 12b11e47f1
commit 370c388319
154 changed files with 4888 additions and 4884 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
MAINTAINERS
View File

@ -12279,10 +12279,10 @@ F: arch/powerpc/crypto/aes_cbc.c
F: arch/powerpc/crypto/aes_ctr.c
F: arch/powerpc/crypto/aes_xts.c
F: arch/powerpc/crypto/aesp8-ppc.*
F: arch/powerpc/crypto/ghash.c
F: arch/powerpc/crypto/ghashp8-ppc.pl
F: arch/powerpc/crypto/ppc-xlate.pl
F: arch/powerpc/crypto/vmx.c
F: lib/crypto/powerpc/gf128hash.h
F: lib/crypto/powerpc/ghashp8-ppc.pl
IBM ServeRAID RAID DRIVER
S: Orphan

13
arch/arm/crypto/Kconfig
View File

@ -3,26 +3,17 @@
menu "Accelerated Cryptographic Algorithms for CPU (arm)"
config CRYPTO_GHASH_ARM_CE
tristate "Hash functions: GHASH (PMULL/NEON/ARMv8 Crypto Extensions)"
tristate "AEAD cipher: AES in GCM mode (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_AEAD
select CRYPTO_HASH
select CRYPTO_CRYPTD
select CRYPTO_LIB_AES
select CRYPTO_LIB_GF128MUL
help
GCM GHASH function (NIST SP800-38D)
AEAD cipher: AES-GCM
Architecture: arm using
- PMULL (Polynomial Multiply Long) instructions
- NEON (Advanced SIMD) extensions
- ARMv8 Crypto Extensions
Use an implementation of GHASH (used by the GCM AEAD chaining mode)
that uses the 64x64 to 128 bit polynomial multiplication (vmull.p64)
that is part of the ARMv8 Crypto Extensions, or a slower variant that
uses the vmull.p8 instruction that is part of the basic NEON ISA.
config CRYPTO_AES_ARM_BS
tristate "Ciphers: AES, modes: ECB/CBC/CTR/XTS (bit-sliced NEON)"
depends on KERNEL_MODE_NEON

171
arch/arm/crypto/ghash-ce-core.S
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Accelerated GHASH implementation with NEON/ARMv8 vmull.p8/64 instructions.
* Accelerated AES-GCM implementation with ARMv8 Crypto Extensions.
*
* Copyright (C) 2015 - 2017 Linaro Ltd.
* Copyright (C) 2023 Google LLC. <ardb@google.com>
@ -29,39 +29,10 @@
XM_H .req d7
XH_L .req d8
t0l .req d10
t0h .req d11
t1l .req d12
t1h .req d13
t2l .req d14
t2h .req d15
t3l .req d16
t3h .req d17
t4l .req d18
t4h .req d19
t0q .req q5
t1q .req q6
t2q .req q7
t3q .req q8
t4q .req q9
XH2 .req q9
s1l .req d20
s1h .req d21
s2l .req d22
s2h .req d23
s3l .req d24
s3h .req d25
s4l .req d26
s4h .req d27
MASK .req d28
SHASH2_p8 .req d28
k16 .req d29
k32 .req d30
k48 .req d31
SHASH2_p64 .req d31
HH .req q10
@ -93,72 +64,6 @@
.text
.macro __pmull_p64, rd, rn, rm, b1, b2, b3, b4
vmull.p64 \rd, \rn, \rm
.endm
/*
* This implementation of 64x64 -> 128 bit polynomial multiplication
* using vmull.p8 instructions (8x8 -> 16) is taken from the paper
* "Fast Software Polynomial Multiplication on ARM Processors Using
* the NEON Engine" by Danilo Camara, Conrado Gouvea, Julio Lopez and
* Ricardo Dahab (https://hal.inria.fr/hal-01506572)
*
* It has been slightly tweaked for in-order performance, and to allow
* 'rq' to overlap with 'ad' or 'bd'.
*/
.macro __pmull_p8, rq, ad, bd, b1=t4l, b2=t3l, b3=t4l, b4=t3l
vext.8 t0l, \ad, \ad, #1 @ A1
.ifc \b1, t4l
vext.8 t4l, \bd, \bd, #1 @ B1
.endif
vmull.p8 t0q, t0l, \bd @ F = A1*B
vext.8 t1l, \ad, \ad, #2 @ A2
vmull.p8 t4q, \ad, \b1 @ E = A*B1
.ifc \b2, t3l
vext.8 t3l, \bd, \bd, #2 @ B2
.endif
vmull.p8 t1q, t1l, \bd @ H = A2*B
vext.8 t2l, \ad, \ad, #3 @ A3
vmull.p8 t3q, \ad, \b2 @ G = A*B2
veor t0q, t0q, t4q @ L = E + F
.ifc \b3, t4l
vext.8 t4l, \bd, \bd, #3 @ B3
.endif
vmull.p8 t2q, t2l, \bd @ J = A3*B
veor t0l, t0l, t0h @ t0 = (L) (P0 + P1) << 8
veor t1q, t1q, t3q @ M = G + H
.ifc \b4, t3l
vext.8 t3l, \bd, \bd, #4 @ B4
.endif
vmull.p8 t4q, \ad, \b3 @ I = A*B3
veor t1l, t1l, t1h @ t1 = (M) (P2 + P3) << 16
vmull.p8 t3q, \ad, \b4 @ K = A*B4
vand t0h, t0h, k48
vand t1h, t1h, k32
veor t2q, t2q, t4q @ N = I + J
veor t0l, t0l, t0h
veor t1l, t1l, t1h
veor t2l, t2l, t2h @ t2 = (N) (P4 + P5) << 24
vand t2h, t2h, k16
veor t3l, t3l, t3h @ t3 = (K) (P6 + P7) << 32
vmov.i64 t3h, #0
vext.8 t0q, t0q, t0q, #15
veor t2l, t2l, t2h
vext.8 t1q, t1q, t1q, #14
vmull.p8 \rq, \ad, \bd @ D = A*B
vext.8 t2q, t2q, t2q, #13
vext.8 t3q, t3q, t3q, #12
veor t0q, t0q, t1q
veor t2q, t2q, t3q
veor \rq, \rq, t0q
veor \rq, \rq, t2q
.endm
//
// PMULL (64x64->128) based reduction for CPUs that can do
// it in a single instruction.
//
.macro __pmull_reduce_p64
vmull.p64 T1, XL_L, MASK
@ -170,30 +75,7 @@
vmull.p64 XL, T1_H, MASK
.endm
//
// Alternative reduction for CPUs that lack support for the
// 64x64->128 PMULL instruction
//
.macro __pmull_reduce_p8
veor XL_H, XL_H, XM_L
veor XH_L, XH_L, XM_H
vshl.i64 T1, XL, #57
vshl.i64 T2, XL, #62
veor T1, T1, T2
vshl.i64 T2, XL, #63
veor T1, T1, T2
veor XL_H, XL_H, T1_L
veor XH_L, XH_L, T1_H
vshr.u64 T1, XL, #1
veor XH, XH, XL
veor XL, XL, T1
vshr.u64 T1, T1, #6
vshr.u64 XL, XL, #1
.endm
.macro ghash_update, pn, enc, aggregate=1, head=1
.macro ghash_update, enc, aggregate=1, head=1
vld1.64 {XL}, [r1]
.if \head
@ -206,8 +88,7 @@
b 3f
.endif
0: .ifc \pn, p64
.if \aggregate
0: .if \aggregate
tst r0, #3 // skip until #blocks is a
bne 2f // round multiple of 4
@ -288,7 +169,6 @@
b 1b
.endif
.endif
2: vld1.8 {T1}, [r2]!
@ -308,15 +188,15 @@
veor T1_L, T1_L, XL_H
veor XL, XL, IN1
__pmull_\pn XH, XL_H, SHASH_H, s1h, s2h, s3h, s4h @ a1 * b1
vmull.p64 XH, XL_H, SHASH_H @ a1 * b1
veor T1, T1, XL
__pmull_\pn XL, XL_L, SHASH_L, s1l, s2l, s3l, s4l @ a0 * b0
__pmull_\pn XM, T1_L, SHASH2_\pn @ (a1+a0)(b1+b0)
vmull.p64 XL, XL_L, SHASH_L @ a0 * b0
vmull.p64 XM, T1_L, SHASH2_p64 @ (a1+a0)(b1+b0)
4: veor T1, XL, XH
veor XM, XM, T1
__pmull_reduce_\pn
__pmull_reduce_p64
veor T1, T1, XH
veor XL, XL, T1
@ -325,8 +205,8 @@
.endm
/*
* void pmull_ghash_update(int blocks, u64 dg[], const char *src,
* struct ghash_key const *k, const char *head)
* void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
* u64 const h[4][2], const char *head)
*/
ENTRY(pmull_ghash_update_p64)
vld1.64 {SHASH}, [r3]!
@ -341,35 +221,12 @@ ENTRY(pmull_ghash_update_p64)
vmov.i8 MASK, #0xe1
vshl.u64 MASK, MASK, #57
ghash_update p64
ghash_update
vst1.64 {XL}, [r1]
bx lr
ENDPROC(pmull_ghash_update_p64)
ENTRY(pmull_ghash_update_p8)
vld1.64 {SHASH}, [r3]
veor SHASH2_p8, SHASH_L, SHASH_H
vext.8 s1l, SHASH_L, SHASH_L, #1
vext.8 s2l, SHASH_L, SHASH_L, #2
vext.8 s3l, SHASH_L, SHASH_L, #3
vext.8 s4l, SHASH_L, SHASH_L, #4
vext.8 s1h, SHASH_H, SHASH_H, #1
vext.8 s2h, SHASH_H, SHASH_H, #2
vext.8 s3h, SHASH_H, SHASH_H, #3
vext.8 s4h, SHASH_H, SHASH_H, #4
vmov.i64 k16, #0xffff
vmov.i64 k32, #0xffffffff
vmov.i64 k48, #0xffffffffffff
ghash_update p8
vst1.64 {XL}, [r1]
bx lr
ENDPROC(pmull_ghash_update_p8)
e0 .req q9
e1 .req q10
e2 .req q11
@ -536,7 +393,7 @@ ENTRY(pmull_gcm_encrypt)
vld1.64 {SHASH}, [r3]
ghash_update p64, enc, head=0
ghash_update enc, head=0
vst1.64 {XL}, [r1]
pop {r4-r8, pc}
@ -554,7 +411,7 @@ ENTRY(pmull_gcm_decrypt)
vld1.64 {SHASH}, [r3]
ghash_update p64, dec, head=0
ghash_update dec, head=0
vst1.64 {XL}, [r1]
pop {r4-r8, pc}
@ -603,7 +460,7 @@ ENTRY(pmull_gcm_enc_final)
vshl.u64 MASK, MASK, #57
mov r0, #1
bne 3f // process head block first
ghash_update p64, aggregate=0, head=0
ghash_update aggregate=0, head=0
vrev64.8 XL, XL
vext.8 XL, XL, XL, #8
@ -660,7 +517,7 @@ ENTRY(pmull_gcm_dec_final)
vshl.u64 MASK, MASK, #57
mov r0, #1
bne 3f // process head block first
ghash_update p64, aggregate=0, head=0
ghash_update aggregate=0, head=0
vrev64.8 XL, XL
vext.8 XL, XL, XL, #8

166
arch/arm/crypto/ghash-ce-glue.c
View File

@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Accelerated GHASH implementation with ARMv8 vmull.p64 instructions.
* AES-GCM using ARMv8 Crypto Extensions
*
* Copyright (C) 2015 - 2018 Linaro Ltd.
* Copyright (C) 2023 Google LLC.
@ -14,7 +14,6 @@
#include <crypto/gf128mul.h>
#include <crypto/ghash.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
#include <crypto/scatterwalk.h>
#include <linux/cpufeature.h>
@ -25,20 +24,14 @@
#include <linux/string.h>
#include <linux/unaligned.h>
MODULE_DESCRIPTION("GHASH hash function using ARMv8 Crypto Extensions");
MODULE_DESCRIPTION("AES-GCM using ARMv8 Crypto Extensions");
MODULE_AUTHOR("Ard Biesheuvel <ardb@kernel.org>");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("ghash");
MODULE_ALIAS_CRYPTO("gcm(aes)");
MODULE_ALIAS_CRYPTO("rfc4106(gcm(aes))");
#define RFC4106_NONCE_SIZE 4
struct ghash_key {
be128 k;
u64 h[][2];
};
struct gcm_key {
u64 h[4][2];
u32 rk[AES_MAX_KEYLENGTH_U32];
@ -46,84 +39,8 @@ struct gcm_key {
u8 nonce[]; // for RFC4106 nonce
};
struct arm_ghash_desc_ctx {
u64 digest[GHASH_DIGEST_SIZE/sizeof(u64)];
};
asmlinkage void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
u64 const h[][2], const char *head);
asmlinkage void pmull_ghash_update_p8(int blocks, u64 dg[], const char *src,
u64 const h[][2], const char *head);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_p64);
static int ghash_init(struct shash_desc *desc)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
*ctx = (struct arm_ghash_desc_ctx){};
return 0;
}
static void ghash_do_update(int blocks, u64 dg[], const char *src,
struct ghash_key *key, const char *head)
{
kernel_neon_begin();
if (static_branch_likely(&use_p64))
pmull_ghash_update_p64(blocks, dg, src, key->h, head);
else
pmull_ghash_update_p8(blocks, dg, src, key->h, head);
kernel_neon_end();
}
static int ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
int blocks;
blocks = len / GHASH_BLOCK_SIZE;
ghash_do_update(blocks, ctx->digest, src, key, NULL);
return len - blocks * GHASH_BLOCK_SIZE;
}
static int ghash_export(struct shash_desc *desc, void *out)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
u8 *dst = out;
put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8);
return 0;
}
static int ghash_import(struct shash_desc *desc, const void *in)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
const u8 *src = in;
ctx->digest[1] = get_unaligned_be64(src);
ctx->digest[0] = get_unaligned_be64(src + 8);
return 0;
}
static int ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *dst)
{
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
if (len) {
u8 buf[GHASH_BLOCK_SIZE] = {};
memcpy(buf, src, len);
ghash_do_update(1, ctx->digest, buf, key, NULL);
memzero_explicit(buf, sizeof(buf));
}
return ghash_export(desc, dst);
}
u64 const h[4][2], const char *head);
static void ghash_reflect(u64 h[], const be128 *k)
{
@ -136,53 +53,6 @@ static void ghash_reflect(u64 h[], const be128 *k)
h[1] ^= 0xc200000000000000UL;
}
static int ghash_setkey(struct crypto_shash *tfm,
const u8 *inkey, unsigned int keylen)
{
struct ghash_key *key = crypto_shash_ctx(tfm);
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
/* needed for the fallback */
memcpy(&key->k, inkey, GHASH_BLOCK_SIZE);
ghash_reflect(key->h[0], &key->k);
if (static_branch_likely(&use_p64)) {
be128 h = key->k;
gf128mul_lle(&h, &key->k);
ghash_reflect(key->h[1], &h);
gf128mul_lle(&h, &key->k);
ghash_reflect(key->h[2], &h);
gf128mul_lle(&h, &key->k);
ghash_reflect(key->h[3], &h);
}
return 0;
}
static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.finup = ghash_finup,
.setkey = ghash_setkey,
.export = ghash_export,
.import = ghash_import,
.descsize = sizeof(struct arm_ghash_desc_ctx),
.statesize = sizeof(struct ghash_desc_ctx),
.base.cra_name = "ghash",
.base.cra_driver_name = "ghash-ce",
.base.cra_priority = 300,
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.base.cra_blocksize = GHASH_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct ghash_key) + sizeof(u64[2]),
.base.cra_module = THIS_MODULE,
};
void pmull_gcm_encrypt(int blocks, u64 dg[], const char *src,
struct gcm_key const *k, char *dst,
const char *iv, int rounds, u32 counter);
@ -561,39 +431,15 @@ static struct aead_alg gcm_aes_algs[] = {{
static int __init ghash_ce_mod_init(void)
{
int err;
if (!(elf_hwcap & HWCAP_NEON))
if (!(elf_hwcap & HWCAP_NEON) || !(elf_hwcap2 & HWCAP2_PMULL))
return -ENODEV;
if (elf_hwcap2 & HWCAP2_PMULL) {
err = crypto_register_aeads(gcm_aes_algs,
ARRAY_SIZE(gcm_aes_algs));
if (err)
return err;
ghash_alg.base.cra_ctxsize += 3 * sizeof(u64[2]);
static_branch_enable(&use_p64);
}
err = crypto_register_shash(&ghash_alg);
if (err)
goto err_aead;
return 0;
err_aead:
if (elf_hwcap2 & HWCAP2_PMULL)
crypto_unregister_aeads(gcm_aes_algs,
ARRAY_SIZE(gcm_aes_algs));
return err;
return crypto_register_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs));
}
static void __exit ghash_ce_mod_exit(void)
{
crypto_unregister_shash(&ghash_alg);
if (elf_hwcap2 & HWCAP2_PMULL)
crypto_unregister_aeads(gcm_aes_algs,
ARRAY_SIZE(gcm_aes_algs));
crypto_unregister_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs));
}
module_init(ghash_ce_mod_init);

2
arch/arm64/configs/defconfig
View File

@ -1916,9 +1916,9 @@ CONFIG_CRYPTO_BENCHMARK=m
CONFIG_CRYPTO_ECHAINIV=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_GHASH_ARM64_CE=y
CONFIG_CRYPTO_SM3_ARM64_CE=m
CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
CONFIG_CRYPTO_AES_ARM64_BS=m
CONFIG_CRYPTO_AES_ARM64_CE_CCM=y

29
arch/arm64/crypto/Kconfig
View File

@ -3,40 +3,17 @@
menu "Accelerated Cryptographic Algorithms for CPU (arm64)"
config CRYPTO_GHASH_ARM64_CE
tristate "Hash functions: GHASH (ARMv8 Crypto Extensions)"
tristate "AEAD cipher: AES in GCM mode (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_LIB_AES
select CRYPTO_LIB_GF128MUL
select CRYPTO_AEAD
help
GCM GHASH function (NIST SP800-38D)
AEAD cipher: AES-GCM
Architecture: arm64 using:
- ARMv8 Crypto Extensions
config CRYPTO_SM3_NEON
tristate "Hash functions: SM3 (NEON)"
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_LIB_SM3
help
SM3 (ShangMi 3) secure hash function (OSCCA GM/T 0004-2012)
Architecture: arm64 using:
- NEON (Advanced SIMD) extensions
config CRYPTO_SM3_ARM64_CE
tristate "Hash functions: SM3 (ARMv8.2 Crypto Extensions)"
depends on KERNEL_MODE_NEON
select CRYPTO_HASH
select CRYPTO_LIB_SM3
help
SM3 (ShangMi 3) secure hash function (OSCCA GM/T 0004-2012)
Architecture: arm64 using:
- ARMv8.2 Crypto Extensions
config CRYPTO_AES_ARM64_CE_BLK
tristate "Ciphers: AES, modes: ECB/CBC/CTR/XTS (ARMv8 Crypto Extensions)"
depends on KERNEL_MODE_NEON
@ -144,7 +121,7 @@ config CRYPTO_AES_ARM64_CE_CCM
select CRYPTO_ALGAPI
select CRYPTO_AES_ARM64_CE_BLK
select CRYPTO_AEAD
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC_MACS
help
AEAD cipher: AES cipher algorithms (FIPS-197) with
CCM (Counter with Cipher Block Chaining-Message Authentication Code)

10
arch/arm64/crypto/Makefile
View File

@ -5,12 +5,6 @@
# Copyright (C) 2014 Linaro Ltd <ard.biesheuvel@linaro.org>
#
obj-$(CONFIG_CRYPTO_SM3_NEON) += sm3-neon.o
sm3-neon-y := sm3-neon-glue.o sm3-neon-core.o
obj-$(CONFIG_CRYPTO_SM3_ARM64_CE) += sm3-ce.o
sm3-ce-y := sm3-ce-glue.o sm3-ce-core.o
obj-$(CONFIG_CRYPTO_SM4_ARM64_CE) += sm4-ce-cipher.o
sm4-ce-cipher-y := sm4-ce-cipher-glue.o sm4-ce-cipher-core.o
@ -33,10 +27,10 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_CE_CCM) += aes-ce-ccm.o
aes-ce-ccm-y := aes-ce-ccm-glue.o aes-ce-ccm-core.o
obj-$(CONFIG_CRYPTO_AES_ARM64_CE_BLK) += aes-ce-blk.o
aes-ce-blk-y := aes-glue-ce.o aes-ce.o
aes-ce-blk-y := aes-glue-ce.o
obj-$(CONFIG_CRYPTO_AES_ARM64_NEON_BLK) += aes-neon-blk.o
aes-neon-blk-y := aes-glue-neon.o aes-neon.o
aes-neon-blk-y := aes-glue-neon.o
obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o
aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o

17
arch/arm64/crypto/aes-ce-ccm-glue.c
View File

@ -31,10 +31,6 @@ static int num_rounds(struct crypto_aes_ctx *ctx)
return 6 + ctx->key_length / 4;
}
asmlinkage u32 ce_aes_mac_update(u8 const in[], u32 const rk[], int rounds,
int blocks, u8 dg[], int enc_before,
int enc_after);
asmlinkage void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
u32 const rk[], u32 rounds, u8 mac[],
u8 ctr[], u8 const final_iv[]);
@ -105,16 +101,11 @@ static u32 ce_aes_ccm_auth_data(u8 mac[], u8 const in[], u32 abytes,
u32 blocks = abytes / AES_BLOCK_SIZE;
if (macp == AES_BLOCK_SIZE || (!macp && blocks > 0)) {
u32 rem = ce_aes_mac_update(in, rk, rounds, blocks, mac,
macp, enc_after);
u32 adv = (blocks - rem) * AES_BLOCK_SIZE;
ce_aes_mac_update(in, rk, rounds, blocks, mac, macp,
enc_after);
macp = enc_after ? 0 : AES_BLOCK_SIZE;
in += adv;
abytes -= adv;
if (unlikely(rem))
macp = 0;
in += blocks * AES_BLOCK_SIZE;
abytes -= blocks * AES_BLOCK_SIZE;
} else {
u32 l = min(AES_BLOCK_SIZE - macp, abytes);

261
arch/arm64/crypto/aes-glue.c
View File

@ -7,7 +7,6 @@
#include <crypto/aes.h>
#include <crypto/ctr.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
#include <crypto/scatterwalk.h>
#include <crypto/sha2.h>
@ -37,7 +36,6 @@
#define aes_xctr_encrypt ce_aes_xctr_encrypt
#define aes_xts_encrypt ce_aes_xts_encrypt
#define aes_xts_decrypt ce_aes_xts_decrypt
#define aes_mac_update ce_aes_mac_update
MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 Crypto Extensions");
#else
#define MODE "neon"
@ -54,7 +52,6 @@ MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 Crypto Extensions");
#define aes_xctr_encrypt neon_aes_xctr_encrypt
#define aes_xts_encrypt neon_aes_xts_encrypt
#define aes_xts_decrypt neon_aes_xts_decrypt
#define aes_mac_update neon_aes_mac_update
MODULE_DESCRIPTION("AES-ECB/CBC/CTR/XTS/XCTR using ARMv8 NEON");
#endif
#if defined(USE_V8_CRYPTO_EXTENSIONS) || !IS_ENABLED(CONFIG_CRYPTO_AES_ARM64_BS)
@ -66,53 +63,11 @@ MODULE_ALIAS_CRYPTO("xctr(aes)");
#endif
MODULE_ALIAS_CRYPTO("cts(cbc(aes))");
MODULE_ALIAS_CRYPTO("essiv(cbc(aes),sha256)");
MODULE_ALIAS_CRYPTO("cmac(aes)");
MODULE_ALIAS_CRYPTO("xcbc(aes)");
MODULE_ALIAS_CRYPTO("cbcmac(aes)");
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_IMPORT_NS("CRYPTO_INTERNAL");
MODULE_LICENSE("GPL v2");
/* defined in aes-modes.S */
asmlinkage void aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void aes_ecb_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void aes_cbc_cts_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 const iv[]);
asmlinkage void aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 const iv[]);
asmlinkage void aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[]);
asmlinkage void aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[], int byte_ctr);
asmlinkage void aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[], u8 iv[],
int first);
asmlinkage void aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[], u8 iv[],
int first);
asmlinkage void aes_essiv_cbc_encrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int blocks, u8 iv[],
u32 const rk2[]);
asmlinkage void aes_essiv_cbc_decrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int blocks, u8 iv[],
u32 const rk2[]);
asmlinkage int aes_mac_update(u8 const in[], u32 const rk[], int rounds,
int blocks, u8 dg[], int enc_before,
int enc_after);
struct crypto_aes_xts_ctx {
struct crypto_aes_ctx key1;
struct crypto_aes_ctx __aligned(8) key2;
@ -123,15 +78,6 @@ struct crypto_aes_essiv_cbc_ctx {
struct crypto_aes_ctx __aligned(8) key2;
};
struct mac_tfm_ctx {
struct crypto_aes_ctx key;
u8 __aligned(8) consts[];
};
struct mac_desc_ctx {
u8 dg[AES_BLOCK_SIZE];
};
static int skcipher_aes_setkey(struct crypto_skcipher *tfm, const u8 *in_key,
unsigned int key_len)
{
@ -762,222 +708,19 @@ static struct skcipher_alg aes_algs[] = { {
.decrypt = essiv_cbc_decrypt,
} };
static int cbcmac_setkey(struct crypto_shash *tfm, const u8 *in_key,
unsigned int key_len)
{
struct mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
return aes_expandkey(&ctx->key, in_key, key_len);
}
static void cmac_gf128_mul_by_x(be128 *y, const be128 *x)
{
u64 a = be64_to_cpu(x->a);
u64 b = be64_to_cpu(x->b);
y->a = cpu_to_be64((a << 1) | (b >> 63));
y->b = cpu_to_be64((b << 1) ^ ((a >> 63) ? 0x87 : 0));
}
static int cmac_setkey(struct crypto_shash *tfm, const u8 *in_key,
unsigned int key_len)
{
struct mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
be128 *consts = (be128 *)ctx->consts;
int rounds = 6 + key_len / 4;
int err;
err = cbcmac_setkey(tfm, in_key, key_len);
if (err)
return err;
/* encrypt the zero vector */
scoped_ksimd()
aes_ecb_encrypt(ctx->consts, (u8[AES_BLOCK_SIZE]){},
ctx->key.key_enc, rounds, 1);
cmac_gf128_mul_by_x(consts, consts);
cmac_gf128_mul_by_x(consts + 1, consts);
return 0;
}
static int xcbc_setkey(struct crypto_shash *tfm, const u8 *in_key,
unsigned int key_len)
{
static u8 const ks[3][AES_BLOCK_SIZE] = {
{ [0 ... AES_BLOCK_SIZE - 1] = 0x1 },
{ [0 ... AES_BLOCK_SIZE - 1] = 0x2 },
{ [0 ... AES_BLOCK_SIZE - 1] = 0x3 },
};
struct mac_tfm_ctx *ctx = crypto_shash_ctx(tfm);
int rounds = 6 + key_len / 4;
u8 key[AES_BLOCK_SIZE];
int err;
err = cbcmac_setkey(tfm, in_key, key_len);
if (err)
return err;
scoped_ksimd() {
aes_ecb_encrypt(key, ks[0], ctx->key.key_enc, rounds, 1);
aes_ecb_encrypt(ctx->consts, ks[1], ctx->key.key_enc, rounds, 2);
}
return cbcmac_setkey(tfm, key, sizeof(key));
}
static int mac_init(struct shash_desc *desc)
{
struct mac_desc_ctx *ctx = shash_desc_ctx(desc);
memset(ctx->dg, 0, AES_BLOCK_SIZE);
return 0;
}
static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks,
u8 dg[], int enc_before)
{
int rounds = 6 + ctx->key_length / 4;
int rem;
do {
scoped_ksimd()
rem = aes_mac_update(in, ctx->key_enc, rounds, blocks,
dg, enc_before, !enc_before);
in += (blocks - rem) * AES_BLOCK_SIZE;
blocks = rem;
} while (blocks);
}
static int mac_update(struct shash_desc *desc, const u8 *p, unsigned int len)
{
struct mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct mac_desc_ctx *ctx = shash_desc_ctx(desc);
int blocks = len / AES_BLOCK_SIZE;
len %= AES_BLOCK_SIZE;
mac_do_update(&tctx->key, p, blocks, ctx->dg, 0);
return len;
}
static int cbcmac_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *out)
{
struct mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct mac_desc_ctx *ctx = shash_desc_ctx(desc);
if (len) {
crypto_xor(ctx->dg, src, len);
mac_do_update(&tctx->key, NULL, 0, ctx->dg, 1);
}
memcpy(out, ctx->dg, AES_BLOCK_SIZE);
return 0;
}
static int cmac_finup(struct shash_desc *desc, const u8 *src, unsigned int len,
u8 *out)
{
struct mac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct mac_desc_ctx *ctx = shash_desc_ctx(desc);
u8 *consts = tctx->consts;
crypto_xor(ctx->dg, src, len);
if (len != AES_BLOCK_SIZE) {
ctx->dg[len] ^= 0x80;
consts += AES_BLOCK_SIZE;
}
mac_do_update(&tctx->key, consts, 1, ctx->dg, 0);
memcpy(out, ctx->dg, AES_BLOCK_SIZE);
return 0;
}
static struct shash_alg mac_algs[] = { {
.base.cra_name = "cmac(aes)",
.base.cra_driver_name = "cmac-aes-" MODE,
.base.cra_priority = PRIO,
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINAL_NONZERO,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct mac_tfm_ctx) +
2 * AES_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.init = mac_init,
.update = mac_update,
.finup = cmac_finup,
.setkey = cmac_setkey,
.descsize = sizeof(struct mac_desc_ctx),
}, {
.base.cra_name = "xcbc(aes)",
.base.cra_driver_name = "xcbc-aes-" MODE,
.base.cra_priority = PRIO,
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINAL_NONZERO,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct mac_tfm_ctx) +
2 * AES_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.init = mac_init,
.update = mac_update,
.finup = cmac_finup,
.setkey = xcbc_setkey,
.descsize = sizeof(struct mac_desc_ctx),
}, {
.base.cra_name = "cbcmac(aes)",
.base.cra_driver_name = "cbcmac-aes-" MODE,
.base.cra_priority = PRIO,
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct mac_tfm_ctx),
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.init = mac_init,
.update = mac_update,
.finup = cbcmac_finup,
.setkey = cbcmac_setkey,
.descsize = sizeof(struct mac_desc_ctx),
} };
static void aes_exit(void)
{
crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs));
crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs));
}
static int __init aes_init(void)
{
int err;
err = crypto_register_skciphers(aes_algs, ARRAY_SIZE(aes_algs));
if (err)
return err;
err = crypto_register_shashes(mac_algs, ARRAY_SIZE(mac_algs));
if (err)
goto unregister_ciphers;
return 0;
unregister_ciphers:
crypto_unregister_skciphers(aes_algs, ARRAY_SIZE(aes_algs));
return err;
return crypto_register_skciphers(aes_algs, ARRAY_SIZE(aes_algs));
}
#ifdef USE_V8_CRYPTO_EXTENSIONS
module_cpu_feature_match(AES, aes_init);
EXPORT_SYMBOL_NS(ce_aes_mac_update, "CRYPTO_INTERNAL");
#else
module_init(aes_init);
EXPORT_SYMBOL(neon_aes_ecb_encrypt);
EXPORT_SYMBOL(neon_aes_cbc_encrypt);
EXPORT_SYMBOL(neon_aes_ctr_encrypt);
EXPORT_SYMBOL(neon_aes_xts_encrypt);
EXPORT_SYMBOL(neon_aes_xts_decrypt);
#endif
module_exit(aes_exit);

15
arch/arm64/crypto/aes-neonbs-glue.c
View File

@ -17,6 +17,7 @@
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_DESCRIPTION("Bit sliced AES using NEON instructions");
MODULE_IMPORT_NS("CRYPTO_INTERNAL");
MODULE_LICENSE("GPL v2");
MODULE_ALIAS_CRYPTO("ecb(aes)");
@ -42,20 +43,6 @@ asmlinkage void aesbs_xts_encrypt(u8 out[], u8 const in[], u8 const rk[],
asmlinkage void aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[],
int rounds, int blocks, u8 iv[]);
/* borrowed from aes-neon-blk.ko */
asmlinkage void neon_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void neon_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void neon_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[]);
asmlinkage void neon_aes_xts_encrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds, int bytes,
u32 const rk2[], u8 iv[], int first);
asmlinkage void neon_aes_xts_decrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds, int bytes,
u32 const rk2[], u8 iv[], int first);
struct aesbs_ctx {
u8 rk[13 * (8 * AES_BLOCK_SIZE) + 32];
int rounds;

221
arch/arm64/crypto/ghash-ce-core.S
View File

@ -1,12 +1,11 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Accelerated GHASH implementation with ARMv8 PMULL instructions.
* Accelerated AES-GCM implementation with ARMv8 Crypto Extensions.
*
* Copyright (C) 2014 - 2018 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
#include <linux/linkage.h>
#include <linux/cfi_types.h>
#include <asm/assembler.h>
SHASH .req v0
@ -19,31 +18,6 @@
XH .req v7
IN1 .req v7
k00_16 .req v8
k32_48 .req v9
t3 .req v10
t4 .req v11
t5 .req v12
t6 .req v13
t7 .req v14
t8 .req v15
t9 .req v16
perm1 .req v17
perm2 .req v18
perm3 .req v19
sh1 .req v20
sh2 .req v21
sh3 .req v22
sh4 .req v23
ss1 .req v24
ss2 .req v25
ss3 .req v26
ss4 .req v27
XL2 .req v8
XM2 .req v9
XH2 .req v10
@ -60,90 +34,6 @@
.text
.arch armv8-a+crypto
.macro __pmull_p64, rd, rn, rm
pmull \rd\().1q, \rn\().1d, \rm\().1d
.endm
.macro __pmull2_p64, rd, rn, rm
pmull2 \rd\().1q, \rn\().2d, \rm\().2d
.endm
.macro __pmull_p8, rq, ad, bd
ext t3.8b, \ad\().8b, \ad\().8b, #1 // A1
ext t5.8b, \ad\().8b, \ad\().8b, #2 // A2
ext t7.8b, \ad\().8b, \ad\().8b, #3 // A3
__pmull_p8_\bd \rq, \ad
.endm
.macro __pmull2_p8, rq, ad, bd
tbl t3.16b, {\ad\().16b}, perm1.16b // A1
tbl t5.16b, {\ad\().16b}, perm2.16b // A2
tbl t7.16b, {\ad\().16b}, perm3.16b // A3
__pmull2_p8_\bd \rq, \ad
.endm
.macro __pmull_p8_SHASH, rq, ad
__pmull_p8_tail \rq, \ad\().8b, SHASH.8b, 8b,, sh1, sh2, sh3, sh4
.endm
.macro __pmull_p8_SHASH2, rq, ad
__pmull_p8_tail \rq, \ad\().8b, SHASH2.8b, 8b,, ss1, ss2, ss3, ss4
.endm
.macro __pmull2_p8_SHASH, rq, ad
__pmull_p8_tail \rq, \ad\().16b, SHASH.16b, 16b, 2, sh1, sh2, sh3, sh4
.endm
.macro __pmull_p8_tail, rq, ad, bd, nb, t, b1, b2, b3, b4
pmull\t t3.8h, t3.\nb, \bd // F = A1*B
pmull\t t4.8h, \ad, \b1\().\nb // E = A*B1
pmull\t t5.8h, t5.\nb, \bd // H = A2*B
pmull\t t6.8h, \ad, \b2\().\nb // G = A*B2
pmull\t t7.8h, t7.\nb, \bd // J = A3*B
pmull\t t8.8h, \ad, \b3\().\nb // I = A*B3
pmull\t t9.8h, \ad, \b4\().\nb // K = A*B4
pmull\t \rq\().8h, \ad, \bd // D = A*B
eor t3.16b, t3.16b, t4.16b // L = E + F
eor t5.16b, t5.16b, t6.16b // M = G + H
eor t7.16b, t7.16b, t8.16b // N = I + J
uzp1 t4.2d, t3.2d, t5.2d
uzp2 t3.2d, t3.2d, t5.2d
uzp1 t6.2d, t7.2d, t9.2d
uzp2 t7.2d, t7.2d, t9.2d
// t3 = (L) (P0 + P1) << 8
// t5 = (M) (P2 + P3) << 16
eor t4.16b, t4.16b, t3.16b
and t3.16b, t3.16b, k32_48.16b
// t7 = (N) (P4 + P5) << 24
// t9 = (K) (P6 + P7) << 32
eor t6.16b, t6.16b, t7.16b
and t7.16b, t7.16b, k00_16.16b
eor t4.16b, t4.16b, t3.16b
eor t6.16b, t6.16b, t7.16b
zip2 t5.2d, t4.2d, t3.2d
zip1 t3.2d, t4.2d, t3.2d
zip2 t9.2d, t6.2d, t7.2d
zip1 t7.2d, t6.2d, t7.2d
ext t3.16b, t3.16b, t3.16b, #15
ext t5.16b, t5.16b, t5.16b, #14
ext t7.16b, t7.16b, t7.16b, #13
ext t9.16b, t9.16b, t9.16b, #12
eor t3.16b, t3.16b, t5.16b
eor t7.16b, t7.16b, t9.16b
eor \rq\().16b, \rq\().16b, t3.16b
eor \rq\().16b, \rq\().16b, t7.16b
.endm
.macro __pmull_pre_p64
add x8, x3, #16
ld1 {HH.2d-HH4.2d}, [x8]
@ -160,43 +50,6 @@
shl MASK.2d, MASK.2d, #57
.endm
.macro __pmull_pre_p8
ext SHASH2.16b, SHASH.16b, SHASH.16b, #8
eor SHASH2.16b, SHASH2.16b, SHASH.16b
// k00_16 := 0x0000000000000000_000000000000ffff
// k32_48 := 0x00000000ffffffff_0000ffffffffffff
movi k32_48.2d, #0xffffffff
mov k32_48.h[2], k32_48.h[0]
ushr k00_16.2d, k32_48.2d, #32
// prepare the permutation vectors
mov_q x5, 0x080f0e0d0c0b0a09
movi T1.8b, #8
dup perm1.2d, x5
eor perm1.16b, perm1.16b, T1.16b
ushr perm2.2d, perm1.2d, #8
ushr perm3.2d, perm1.2d, #16
ushr T1.2d, perm1.2d, #24
sli perm2.2d, perm1.2d, #56
sli perm3.2d, perm1.2d, #48
sli T1.2d, perm1.2d, #40
// precompute loop invariants
tbl sh1.16b, {SHASH.16b}, perm1.16b
tbl sh2.16b, {SHASH.16b}, perm2.16b
tbl sh3.16b, {SHASH.16b}, perm3.16b
tbl sh4.16b, {SHASH.16b}, T1.16b
ext ss1.8b, SHASH2.8b, SHASH2.8b, #1
ext ss2.8b, SHASH2.8b, SHASH2.8b, #2
ext ss3.8b, SHASH2.8b, SHASH2.8b, #3
ext ss4.8b, SHASH2.8b, SHASH2.8b, #4
.endm
//
// PMULL (64x64->128) based reduction for CPUs that can do
// it in a single instruction.
//
.macro __pmull_reduce_p64
pmull T2.1q, XL.1d, MASK.1d
eor XM.16b, XM.16b, T1.16b
@ -209,39 +62,15 @@
pmull XL.1q, XL.1d, MASK.1d
.endm
//
// Alternative reduction for CPUs that lack support for the
// 64x64->128 PMULL instruction
//
.macro __pmull_reduce_p8
eor XM.16b, XM.16b, T1.16b
mov XL.d[1], XM.d[0]
mov XH.d[0], XM.d[1]
shl T1.2d, XL.2d, #57
shl T2.2d, XL.2d, #62
eor T2.16b, T2.16b, T1.16b
shl T1.2d, XL.2d, #63
eor T2.16b, T2.16b, T1.16b
ext T1.16b, XL.16b, XH.16b, #8
eor T2.16b, T2.16b, T1.16b
mov XL.d[1], T2.d[0]
mov XH.d[0], T2.d[1]
ushr T2.2d, XL.2d, #1
eor XH.16b, XH.16b, XL.16b
eor XL.16b, XL.16b, T2.16b
ushr T2.2d, T2.2d, #6
ushr XL.2d, XL.2d, #1
.endm
.macro __pmull_ghash, pn
/*
* void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
* u64 const h[4][2], const char *head)
*/
SYM_FUNC_START(pmull_ghash_update_p64)
ld1 {SHASH.2d}, [x3]
ld1 {XL.2d}, [x1]
__pmull_pre_\pn
__pmull_pre_p64
/* do the head block first, if supplied */
cbz x4, 0f
@ -249,7 +78,7 @@
mov x4, xzr
b 3f
0: .ifc \pn, p64
0:
tbnz w0, #0, 2f // skip until #blocks is a
tbnz w0, #1, 2f // round multiple of 4
@ -314,7 +143,6 @@
cbz w0, 5f
b 1b
.endif
2: ld1 {T1.2d}, [x2], #16
sub w0, w0, #1
@ -327,16 +155,16 @@ CPU_LE( rev64 T1.16b, T1.16b )
eor T1.16b, T1.16b, T2.16b
eor XL.16b, XL.16b, IN1.16b
__pmull2_\pn XH, XL, SHASH // a1 * b1
pmull2 XH.1q, XL.2d, SHASH.2d // a1 * b1
eor T1.16b, T1.16b, XL.16b
__pmull_\pn XL, XL, SHASH // a0 * b0
__pmull_\pn XM, T1, SHASH2 // (a1 + a0)(b1 + b0)
pmull XL.1q, XL.1d, SHASH.1d // a0 * b0
pmull XM.1q, T1.1d, SHASH2.1d // (a1 + a0)(b1 + b0)
4: eor T2.16b, XL.16b, XH.16b
ext T1.16b, XL.16b, XH.16b, #8
eor XM.16b, XM.16b, T2.16b
__pmull_reduce_\pn
__pmull_reduce_p64
eor T2.16b, T2.16b, XH.16b
eor XL.16b, XL.16b, T2.16b
@ -345,20 +173,8 @@ CPU_LE( rev64 T1.16b, T1.16b )
5: st1 {XL.2d}, [x1]
ret
.endm
/*
* void pmull_ghash_update(int blocks, u64 dg[], const char *src,
* struct ghash_key const *k, const char *head)
*/
SYM_TYPED_FUNC_START(pmull_ghash_update_p64)
__pmull_ghash p64
SYM_FUNC_END(pmull_ghash_update_p64)
SYM_TYPED_FUNC_START(pmull_ghash_update_p8)
__pmull_ghash p8
SYM_FUNC_END(pmull_ghash_update_p8)
KS0 .req v8
KS1 .req v9
KS2 .req v10
@ -597,18 +413,19 @@ CPU_LE( rev w8, w8 )
.endm
/*
* void pmull_gcm_encrypt(int blocks, u8 dst[], const u8 src[],
* struct ghash_key const *k, u64 dg[], u8 ctr[],
* int rounds, u8 tag)
* void pmull_gcm_encrypt(int bytes, u8 dst[], const u8 src[],
* u64 const h[4][2], u64 dg[], u8 ctr[],
* u32 const rk[], int rounds, u8 tag[])
*/
SYM_FUNC_START(pmull_gcm_encrypt)
pmull_gcm_do_crypt 1
SYM_FUNC_END(pmull_gcm_encrypt)
/*
* void pmull_gcm_decrypt(int blocks, u8 dst[], const u8 src[],
* struct ghash_key const *k, u64 dg[], u8 ctr[],
* int rounds, u8 tag)
* int pmull_gcm_decrypt(int bytes, u8 dst[], const u8 src[],
* u64 const h[4][2], u64 dg[], u8 ctr[],
* u32 const rk[], int rounds, const u8 l[],
* const u8 tag[], u64 authsize)
*/
SYM_FUNC_START(pmull_gcm_decrypt)
pmull_gcm_do_crypt 0

168
arch/arm64/crypto/ghash-ce-glue.c
View File

@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Accelerated GHASH implementation with ARMv8 PMULL instructions.
* AES-GCM using ARMv8 Crypto Extensions
*
* Copyright (C) 2014 - 2018 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
@ -11,7 +11,6 @@
#include <crypto/ghash.h>
#include <crypto/gf128mul.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
#include <crypto/scatterwalk.h>
#include <linux/cpufeature.h>
@ -23,121 +22,41 @@
#include <asm/simd.h>
MODULE_DESCRIPTION("GHASH and AES-GCM using ARMv8 Crypto Extensions");
MODULE_DESCRIPTION("AES-GCM using ARMv8 Crypto Extensions");
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_LICENSE("GPL v2");
MODULE_ALIAS_CRYPTO("ghash");
MODULE_ALIAS_CRYPTO("gcm(aes)");
MODULE_ALIAS_CRYPTO("rfc4106(gcm(aes))");
#define RFC4106_NONCE_SIZE 4
struct ghash_key {
struct arm_ghash_key {
be128 k;
u64 h[][2];
};
struct arm_ghash_desc_ctx {
u64 digest[GHASH_DIGEST_SIZE/sizeof(u64)];
u64 h[4][2];
};
struct gcm_aes_ctx {
struct aes_enckey aes_key;
u8 nonce[RFC4106_NONCE_SIZE];
struct ghash_key ghash_key;
struct arm_ghash_key ghash_key;
};
asmlinkage void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
u64 const h[][2], const char *head);
asmlinkage void pmull_ghash_update_p8(int blocks, u64 dg[], const char *src,
u64 const h[][2], const char *head);
u64 const h[4][2], const char *head);
asmlinkage void pmull_gcm_encrypt(int bytes, u8 dst[], const u8 src[],
u64 const h[][2], u64 dg[], u8 ctr[],
u64 const h[4][2], u64 dg[], u8 ctr[],
u32 const rk[], int rounds, u8 tag[]);
asmlinkage int pmull_gcm_decrypt(int bytes, u8 dst[], const u8 src[],
u64 const h[][2], u64 dg[], u8 ctr[],
u64 const h[4][2], u64 dg[], u8 ctr[],
u32 const rk[], int rounds, const u8 l[],
const u8 tag[], u64 authsize);
static int ghash_init(struct shash_desc *desc)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
*ctx = (struct arm_ghash_desc_ctx){};
return 0;
}
static __always_inline
void ghash_do_simd_update(int blocks, u64 dg[], const char *src,
struct ghash_key *key, const char *head,
void (*simd_update)(int blocks, u64 dg[],
const char *src,
u64 const h[][2],
const char *head))
static void ghash_do_simd_update(int blocks, u64 dg[], const char *src,
struct arm_ghash_key *key, const char *head)
{
scoped_ksimd()
simd_update(blocks, dg, src, key->h, head);
}
/* avoid hogging the CPU for too long */
#define MAX_BLOCKS (SZ_64K / GHASH_BLOCK_SIZE)
static int ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
int blocks;
blocks = len / GHASH_BLOCK_SIZE;
len -= blocks * GHASH_BLOCK_SIZE;
do {
int chunk = min(blocks, MAX_BLOCKS);
ghash_do_simd_update(chunk, ctx->digest, src, key, NULL,
pmull_ghash_update_p8);
blocks -= chunk;
src += chunk * GHASH_BLOCK_SIZE;
} while (unlikely(blocks > 0));
return len;
}
static int ghash_export(struct shash_desc *desc, void *out)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
u8 *dst = out;
put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8);
return 0;
}
static int ghash_import(struct shash_desc *desc, const void *in)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
const u8 *src = in;
ctx->digest[1] = get_unaligned_be64(src);
ctx->digest[0] = get_unaligned_be64(src + 8);
return 0;
}
static int ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *dst)
{
struct arm_ghash_desc_ctx *ctx = shash_desc_ctx(desc);
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
if (len) {
u8 buf[GHASH_BLOCK_SIZE] = {};
memcpy(buf, src, len);
ghash_do_simd_update(1, ctx->digest, buf, key, NULL,
pmull_ghash_update_p8);
memzero_explicit(buf, sizeof(buf));
}
return ghash_export(desc, dst);
pmull_ghash_update_p64(blocks, dg, src, key->h, head);
}
static void ghash_reflect(u64 h[], const be128 *k)
@ -151,41 +70,6 @@ static void ghash_reflect(u64 h[], const be128 *k)
h[1] ^= 0xc200000000000000UL;
}
static int ghash_setkey(struct crypto_shash *tfm,
const u8 *inkey, unsigned int keylen)
{
struct ghash_key *key = crypto_shash_ctx(tfm);
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
/* needed for the fallback */
memcpy(&key->k, inkey, GHASH_BLOCK_SIZE);
ghash_reflect(key->h[0], &key->k);
return 0;
}
static struct shash_alg ghash_alg = {
.base.cra_name = "ghash",
.base.cra_driver_name = "ghash-neon",
.base.cra_priority = 150,
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.base.cra_blocksize = GHASH_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct ghash_key) + sizeof(u64[2]),
.base.cra_module = THIS_MODULE,
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.finup = ghash_finup,
.setkey = ghash_setkey,
.export = ghash_export,
.import = ghash_import,
.descsize = sizeof(struct arm_ghash_desc_ctx),
.statesize = sizeof(struct ghash_desc_ctx),
};
static int gcm_aes_setkey(struct crypto_aead *tfm, const u8 *inkey,
unsigned int keylen)
{
@ -240,9 +124,7 @@ static void gcm_update_mac(u64 dg[], const u8 *src, int count, u8 buf[],
int blocks = count / GHASH_BLOCK_SIZE;
ghash_do_simd_update(blocks, dg, src, &ctx->ghash_key,
*buf_count ? buf : NULL,
pmull_ghash_update_p64);
*buf_count ? buf : NULL);
src += blocks * GHASH_BLOCK_SIZE;
count %= GHASH_BLOCK_SIZE;
*buf_count = 0;
@ -275,8 +157,7 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u64 dg[], u32 len)
if (buf_count) {
memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count);
ghash_do_simd_update(1, dg, buf, &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
ghash_do_simd_update(1, dg, buf, &ctx->ghash_key, NULL);
}
}
@ -482,8 +363,7 @@ static struct aead_alg gcm_aes_algs[] = {{
.base.cra_driver_name = "gcm-aes-ce",
.base.cra_priority = 300,
.base.cra_blocksize = 1,
.base.cra_ctxsize = sizeof(struct gcm_aes_ctx) +
4 * sizeof(u64[2]),
.base.cra_ctxsize = sizeof(struct gcm_aes_ctx),
.base.cra_module = THIS_MODULE,
}, {
.ivsize = GCM_RFC4106_IV_SIZE,
@ -498,29 +378,21 @@ static struct aead_alg gcm_aes_algs[] = {{
.base.cra_driver_name = "rfc4106-gcm-aes-ce",
.base.cra_priority = 300,
.base.cra_blocksize = 1,
.base.cra_ctxsize = sizeof(struct gcm_aes_ctx) +
4 * sizeof(u64[2]),
.base.cra_ctxsize = sizeof(struct gcm_aes_ctx),
.base.cra_module = THIS_MODULE,
}};
static int __init ghash_ce_mod_init(void)
{
if (!cpu_have_named_feature(ASIMD))
if (!cpu_have_named_feature(ASIMD) || !cpu_have_named_feature(PMULL))
return -ENODEV;
if (cpu_have_named_feature(PMULL))
return crypto_register_aeads(gcm_aes_algs,
ARRAY_SIZE(gcm_aes_algs));
return crypto_register_shash(&ghash_alg);
return crypto_register_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs));
}
static void __exit ghash_ce_mod_exit(void)
{
if (cpu_have_named_feature(PMULL))
crypto_unregister_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs));
else
crypto_unregister_shash(&ghash_alg);
crypto_unregister_aeads(gcm_aes_algs, ARRAY_SIZE(gcm_aes_algs));
}
static const struct cpu_feature __maybe_unused ghash_cpu_feature[] = {

70
arch/arm64/crypto/sm3-ce-glue.c
View File

@ -1,70 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* sm3-ce-glue.c - SM3 secure hash using ARMv8.2 Crypto Extensions
*
* Copyright (C) 2018 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
#include <crypto/internal/hash.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/cpufeature.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <asm/simd.h>
MODULE_DESCRIPTION("SM3 secure hash using ARMv8 Crypto Extensions");
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
MODULE_LICENSE("GPL v2");
asmlinkage void sm3_ce_transform(struct sm3_state *sst, u8 const *src,
int blocks);
static int sm3_ce_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
int remain;
scoped_ksimd() {
remain = sm3_base_do_update_blocks(desc, data, len, sm3_ce_transform);
}
return remain;
}
static int sm3_ce_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
scoped_ksimd() {
sm3_base_do_finup(desc, data, len, sm3_ce_transform);
}
return sm3_base_finish(desc, out);
}
static struct shash_alg sm3_alg = {
.digestsize = SM3_DIGEST_SIZE,
.init = sm3_base_init,
.update = sm3_ce_update,
.finup = sm3_ce_finup,
.descsize = SM3_STATE_SIZE,
.base.cra_name = "sm3",
.base.cra_driver_name = "sm3-ce",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINUP_MAX,
.base.cra_blocksize = SM3_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.base.cra_priority = 400,
};
static int __init sm3_ce_mod_init(void)
{
return crypto_register_shash(&sm3_alg);
}
static void __exit sm3_ce_mod_fini(void)
{
crypto_unregister_shash(&sm3_alg);
}
module_cpu_feature_match(SM3, sm3_ce_mod_init);
module_exit(sm3_ce_mod_fini);

67
arch/arm64/crypto/sm3-neon-glue.c
View File

@ -1,67 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* sm3-neon-glue.c - SM3 secure hash using NEON instructions
*
* Copyright (C) 2022 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <asm/simd.h>
#include <crypto/internal/hash.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/cpufeature.h>
#include <linux/kernel.h>
#include <linux/module.h>
asmlinkage void sm3_neon_transform(struct sm3_state *sst, u8 const *src,
int blocks);
static int sm3_neon_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
scoped_ksimd()
return sm3_base_do_update_blocks(desc, data, len,
sm3_neon_transform);
}
static int sm3_neon_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
scoped_ksimd()
sm3_base_do_finup(desc, data, len, sm3_neon_transform);
return sm3_base_finish(desc, out);
}
static struct shash_alg sm3_alg = {
.digestsize = SM3_DIGEST_SIZE,
.init = sm3_base_init,
.update = sm3_neon_update,
.finup = sm3_neon_finup,
.descsize = SM3_STATE_SIZE,
.base.cra_name = "sm3",
.base.cra_driver_name = "sm3-neon",
.base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINUP_MAX,
.base.cra_blocksize = SM3_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.base.cra_priority = 200,
};
static int __init sm3_neon_init(void)
{
return crypto_register_shash(&sm3_alg);
}
static void __exit sm3_neon_fini(void)
{
crypto_unregister_shash(&sm3_alg);
}
module_init(sm3_neon_init);
module_exit(sm3_neon_fini);
MODULE_DESCRIPTION("SM3 secure hash using NEON instructions");
MODULE_AUTHOR("Jussi Kivilinna <jussi.kivilinna@iki.fi>");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_LICENSE("GPL v2");

22
arch/arm64/include/asm/assembler.h
View File

@ -748,28 +748,6 @@ alternative_else_nop_endif
set_sctlr sctlr_el2, \reg
.endm
/*
* Check whether asm code should yield as soon as it is able. This is
* the case if we are currently running in task context, and the
* TIF_NEED_RESCHED flag is set. (Note that the TIF_NEED_RESCHED flag
* is stored negated in the top word of the thread_info::preempt_count
* field)
*/
.macro cond_yield, lbl:req, tmp:req, tmp2
#ifdef CONFIG_PREEMPT_VOLUNTARY
get_current_task \tmp
ldr \tmp, [\tmp, #TSK_TI_PREEMPT]
/*
* If we are serving a softirq, there is no point in yielding: the
* softirq will not be preempted no matter what we do, so we should
* run to completion as quickly as we can. The preempt_count field will
* have BIT(SOFTIRQ_SHIFT) set in this case, so the zero check will
* catch this case too.
*/
cbz \tmp, \lbl
#endif
.endm
/*
* Branch Target Identifier (BTI)
*/

2
arch/loongarch/configs/loongson32_defconfig
View File

@ -1080,7 +1080,7 @@ CONFIG_CRYPTO_SM4_GENERIC=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m

2
arch/loongarch/configs/loongson64_defconfig
View File

@ -1113,7 +1113,7 @@ CONFIG_CRYPTO_SM4_GENERIC=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_CHACHA20POLY1305=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/amiga_defconfig
View File

@ -581,7 +581,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/apollo_defconfig
View File

@ -538,7 +538,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/atari_defconfig
View File

@ -558,7 +558,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/bvme6000_defconfig
View File

@ -530,7 +530,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/hp300_defconfig
View File

@ -540,7 +540,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/mac_defconfig
View File

@ -557,7 +557,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/multi_defconfig
View File

@ -644,7 +644,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/mvme147_defconfig
View File

@ -530,7 +530,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/mvme16x_defconfig
View File

@ -531,7 +531,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/q40_defconfig
View File

@ -547,7 +547,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/sun3_defconfig
View File

@ -528,7 +528,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

2
arch/m68k/configs/sun3x_defconfig
View File

@ -528,7 +528,7 @@ CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_LZO=m

5
arch/powerpc/crypto/Kconfig
View File

@ -54,12 +54,11 @@ config CRYPTO_DEV_VMX_ENCRYPT
select CRYPTO_AES
select CRYPTO_CBC
select CRYPTO_CTR
select CRYPTO_GHASH
select CRYPTO_XTS
default m
help
Support for VMX cryptographic acceleration instructions on Power8 CPU.
This module supports acceleration for AES and GHASH in hardware. If you
choose 'M' here, this module will be called vmx-crypto.
This module supports acceleration for AES in hardware. If you choose
'M' here, this module will be called vmx-crypto.
endmenu

8
arch/powerpc/crypto/Makefile
View File

@ -11,7 +11,7 @@ obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
aes-ppc-spe-y := aes-spe-glue.o
aes-gcm-p10-crypto-y := aes-gcm-p10-glue.o aes-gcm-p10.o ghashp10-ppc.o aesp10-ppc.o
vmx-crypto-objs := vmx.o ghashp8-ppc.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
vmx-crypto-objs := vmx.o aes_cbc.o aes_ctr.o aes_xts.o
ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
override flavour := linux-ppc64le
@ -26,14 +26,10 @@ endif
quiet_cmd_perl = PERL $@
cmd_perl = $(PERL) $< $(flavour) > $@
targets += aesp10-ppc.S ghashp10-ppc.S ghashp8-ppc.S
targets += aesp10-ppc.S ghashp10-ppc.S
$(obj)/aesp10-ppc.S $(obj)/ghashp10-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
$(obj)/ghashp8-ppc.S: $(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perl)
OBJECT_FILES_NON_STANDARD_aesp10-ppc.o := y
OBJECT_FILES_NON_STANDARD_ghashp10-ppc.o := y
OBJECT_FILES_NON_STANDARD_ghashp8-ppc.o := y

1
arch/powerpc/crypto/aesp8-ppc.h
View File

@ -2,7 +2,6 @@
#include <linux/types.h>
#include <crypto/aes.h>
extern struct shash_alg p8_ghash_alg;
extern struct skcipher_alg p8_aes_cbc_alg;
extern struct skcipher_alg p8_aes_ctr_alg;
extern struct skcipher_alg p8_aes_xts_alg;

160
arch/powerpc/crypto/ghash.c
View File

@ -1,160 +0,0 @@
// SPDX-License-Identifier: GPL-2.0
/*
* GHASH routines supporting VMX instructions on the Power 8
*
* Copyright (C) 2015, 2019 International Business Machines Inc.
*
* Author: Marcelo Henrique Cerri <mhcerri@br.ibm.com>
*
* Extended by Daniel Axtens <dja@axtens.net> to replace the fallback
* mechanism. The new approach is based on arm64 code, which is:
* Copyright (C) 2014 - 2018 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
#include "aesp8-ppc.h"
#include <asm/switch_to.h>
#include <crypto/aes.h>
#include <crypto/gf128mul.h>
#include <crypto/ghash.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/simd.h>
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
#include <linux/uaccess.h>
void gcm_init_p8(u128 htable[16], const u64 Xi[2]);
void gcm_gmult_p8(u64 Xi[2], const u128 htable[16]);
void gcm_ghash_p8(u64 Xi[2], const u128 htable[16],
const u8 *in, size_t len);
struct p8_ghash_ctx {
/* key used by vector asm */
u128 htable[16];
/* key used by software fallback */
be128 key;
};
struct p8_ghash_desc_ctx {
u64 shash[2];
};
static int p8_ghash_init(struct shash_desc *desc)
{
struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memset(dctx->shash, 0, GHASH_DIGEST_SIZE);
return 0;
}
static int p8_ghash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen)
{
struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(tfm));
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
preempt_disable();
pagefault_disable();
enable_kernel_vsx();
gcm_init_p8(ctx->htable, (const u64 *) key);
disable_kernel_vsx();
pagefault_enable();
preempt_enable();
memcpy(&ctx->key, key, GHASH_BLOCK_SIZE);
return 0;
}
static inline void __ghash_block(struct p8_ghash_ctx *ctx,
struct p8_ghash_desc_ctx *dctx,
const u8 *src)
{
if (crypto_simd_usable()) {
preempt_disable();
pagefault_disable();
enable_kernel_vsx();
gcm_ghash_p8(dctx->shash, ctx->htable, src, GHASH_BLOCK_SIZE);
disable_kernel_vsx();
pagefault_enable();
preempt_enable();
} else {
crypto_xor((u8 *)dctx->shash, src, GHASH_BLOCK_SIZE);
gf128mul_lle((be128 *)dctx->shash, &ctx->key);
}
}
static inline int __ghash_blocks(struct p8_ghash_ctx *ctx,
struct p8_ghash_desc_ctx *dctx,
const u8 *src, unsigned int srclen)
{
int remain = srclen - round_down(srclen, GHASH_BLOCK_SIZE);
srclen -= remain;
if (crypto_simd_usable()) {
preempt_disable();
pagefault_disable();
enable_kernel_vsx();
gcm_ghash_p8(dctx->shash, ctx->htable,
src, srclen);
disable_kernel_vsx();
pagefault_enable();
preempt_enable();
} else {
do {
crypto_xor((u8 *)dctx->shash, src, GHASH_BLOCK_SIZE);
gf128mul_lle((be128 *)dctx->shash, &ctx->key);
srclen -= GHASH_BLOCK_SIZE;
src += GHASH_BLOCK_SIZE;
} while (srclen);
}
return remain;
}
static int p8_ghash_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
return __ghash_blocks(ctx, dctx, src, srclen);
}
static int p8_ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *out)
{
struct p8_ghash_ctx *ctx = crypto_tfm_ctx(crypto_shash_tfm(desc->tfm));
struct p8_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
if (len) {
u8 buf[GHASH_BLOCK_SIZE] = {};
memcpy(buf, src, len);
__ghash_block(ctx, dctx, buf);
memzero_explicit(buf, sizeof(buf));
}
memcpy(out, dctx->shash, GHASH_DIGEST_SIZE);
return 0;
}
struct shash_alg p8_ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = p8_ghash_init,
.update = p8_ghash_update,
.finup = p8_ghash_finup,
.setkey = p8_ghash_setkey,
.descsize = sizeof(struct p8_ghash_desc_ctx),
.base = {
.cra_name = "ghash",
.cra_driver_name = "p8_ghash",
.cra_priority = 1000,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.cra_blocksize = GHASH_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct p8_ghash_ctx),
.cra_module = THIS_MODULE,
},
};

10
arch/powerpc/crypto/vmx.c
View File

@ -14,7 +14,6 @@
#include <linux/cpufeature.h>
#include <linux/crypto.h>
#include <asm/cputable.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
#include "aesp8-ppc.h"
@ -23,13 +22,9 @@ static int __init p8_init(void)
{
int ret;
ret = crypto_register_shash(&p8_ghash_alg);
if (ret)
goto err;
ret = crypto_register_skcipher(&p8_aes_cbc_alg);
if (ret)
goto err_unregister_ghash;
goto err;
ret = crypto_register_skcipher(&p8_aes_ctr_alg);
if (ret)
@ -45,8 +40,6 @@ err_unregister_aes_ctr:
crypto_unregister_skcipher(&p8_aes_ctr_alg);
err_unregister_aes_cbc:
crypto_unregister_skcipher(&p8_aes_cbc_alg);
err_unregister_ghash:
crypto_unregister_shash(&p8_ghash_alg);
err:
return ret;
}
@ -56,7 +49,6 @@ static void __exit p8_exit(void)
crypto_unregister_skcipher(&p8_aes_xts_alg);
crypto_unregister_skcipher(&p8_aes_ctr_alg);
crypto_unregister_skcipher(&p8_aes_cbc_alg);
crypto_unregister_shash(&p8_ghash_alg);
}
module_cpu_feature_match(PPC_MODULE_FEATURE_VEC_CRYPTO, p8_init);

24
arch/riscv/crypto/Kconfig
View File

@ -17,30 +17,6 @@ config CRYPTO_AES_RISCV64
- Zvkb vector crypto extension (CTR)
- Zvkg vector crypto extension (XTS)
config CRYPTO_GHASH_RISCV64
tristate "Hash functions: GHASH"
depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
select CRYPTO_GCM
help
GCM GHASH function (NIST SP 800-38D)
Architecture: riscv64 using:
- Zvkg vector crypto extension
config CRYPTO_SM3_RISCV64
tristate "Hash functions: SM3 (ShangMi 3)"
depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
select CRYPTO_HASH
select CRYPTO_LIB_SM3
help
SM3 (ShangMi 3) secure hash function (OSCCA GM/T 0004-2012)
Architecture: riscv64 using:
- Zvksh vector crypto extension
- Zvkb vector crypto extension
config CRYPTO_SM4_RISCV64
tristate "Ciphers: SM4 (ShangMi 4)"
depends on 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \

6
arch/riscv/crypto/Makefile
View File

@ -4,11 +4,5 @@ obj-$(CONFIG_CRYPTO_AES_RISCV64) += aes-riscv64.o
aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \
aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o
obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o
ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o
obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o
sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.o
obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o
sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed-zvkb.o

146
arch/riscv/crypto/ghash-riscv64-glue.c
View File

@ -1,146 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* GHASH using the RISC-V vector crypto extensions
*
* Copyright (C) 2023 VRULL GmbH
* Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
*
* Copyright (C) 2023 SiFive, Inc.
* Author: Jerry Shih <jerry.shih@sifive.com>
*/
#include <asm/simd.h>
#include <asm/vector.h>
#include <crypto/b128ops.h>
#include <crypto/gf128mul.h>
#include <crypto/ghash.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/simd.h>
#include <crypto/utils.h>
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
asmlinkage void ghash_zvkg(be128 *accumulator, const be128 *key, const u8 *data,
size_t len);
struct riscv64_ghash_tfm_ctx {
be128 key;
};
struct riscv64_ghash_desc_ctx {
be128 accumulator;
};
static int riscv64_ghash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen)
{
struct riscv64_ghash_tfm_ctx *tctx = crypto_shash_ctx(tfm);
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
memcpy(&tctx->key, key, GHASH_BLOCK_SIZE);
return 0;
}
static int riscv64_ghash_init(struct shash_desc *desc)
{
struct riscv64_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
*dctx = (struct riscv64_ghash_desc_ctx){};
return 0;
}
static inline void
riscv64_ghash_blocks(const struct riscv64_ghash_tfm_ctx *tctx,
struct riscv64_ghash_desc_ctx *dctx,
const u8 *src, size_t srclen)
{
/* The srclen is nonzero and a multiple of 16. */
if (crypto_simd_usable()) {
kernel_vector_begin();
ghash_zvkg(&dctx->accumulator, &tctx->key, src, srclen);
kernel_vector_end();
} else {
do {
crypto_xor((u8 *)&dctx->accumulator, src,
GHASH_BLOCK_SIZE);
gf128mul_lle(&dctx->accumulator, &tctx->key);
src += GHASH_BLOCK_SIZE;
srclen -= GHASH_BLOCK_SIZE;
} while (srclen);
}
}
static int riscv64_ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int srclen)
{
const struct riscv64_ghash_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct riscv64_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
riscv64_ghash_blocks(tctx, dctx, src,
round_down(srclen, GHASH_BLOCK_SIZE));
return srclen - round_down(srclen, GHASH_BLOCK_SIZE);
}
static int riscv64_ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *out)
{
const struct riscv64_ghash_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
struct riscv64_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
if (len) {
u8 buf[GHASH_BLOCK_SIZE] = {};
memcpy(buf, src, len);
riscv64_ghash_blocks(tctx, dctx, buf, GHASH_BLOCK_SIZE);
memzero_explicit(buf, sizeof(buf));
}
memcpy(out, &dctx->accumulator, GHASH_DIGEST_SIZE);
return 0;
}
static struct shash_alg riscv64_ghash_alg = {
.init = riscv64_ghash_init,
.update = riscv64_ghash_update,
.finup = riscv64_ghash_finup,
.setkey = riscv64_ghash_setkey,
.descsize = sizeof(struct riscv64_ghash_desc_ctx),
.digestsize = GHASH_DIGEST_SIZE,
.base = {
.cra_blocksize = GHASH_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct riscv64_ghash_tfm_ctx),
.cra_priority = 300,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.cra_name = "ghash",
.cra_driver_name = "ghash-riscv64-zvkg",
.cra_module = THIS_MODULE,
},
};
static int __init riscv64_ghash_mod_init(void)
{
if (riscv_isa_extension_available(NULL, ZVKG) &&
riscv_vector_vlen() >= 128)
return crypto_register_shash(&riscv64_ghash_alg);
return -ENODEV;
}
static void __exit riscv64_ghash_mod_exit(void)
{
crypto_unregister_shash(&riscv64_ghash_alg);
}
module_init(riscv64_ghash_mod_init);
module_exit(riscv64_ghash_mod_exit);
MODULE_DESCRIPTION("GHASH (RISC-V accelerated)");
MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("ghash");

97
arch/riscv/crypto/sm3-riscv64-glue.c
View File

@ -1,97 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
* SM3 using the RISC-V vector crypto extensions
*
* Copyright (C) 2023 VRULL GmbH
* Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
*
* Copyright (C) 2023 SiFive, Inc.
* Author: Jerry Shih <jerry.shih@sifive.com>
*/
#include <asm/simd.h>
#include <asm/vector.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/simd.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/kernel.h>
#include <linux/module.h>
/*
* Note: the asm function only uses the 'state' field of struct sm3_state.
* It is assumed to be the first field.
*/
asmlinkage void sm3_transform_zvksh_zvkb(
struct sm3_state *state, const u8 *data, int num_blocks);
static void sm3_block(struct sm3_state *state, const u8 *data,
int num_blocks)
{
/*
* Ensure struct sm3_state begins directly with the SM3
* 256-bit internal state, as this is what the asm function expects.
*/
BUILD_BUG_ON(offsetof(struct sm3_state, state) != 0);
if (crypto_simd_usable()) {
kernel_vector_begin();
sm3_transform_zvksh_zvkb(state, data, num_blocks);
kernel_vector_end();
} else {
sm3_block_generic(state, data, num_blocks);
}
}
static int riscv64_sm3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
return sm3_base_do_update_blocks(desc, data, len, sm3_block);
}
static int riscv64_sm3_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
sm3_base_do_finup(desc, data, len, sm3_block);
return sm3_base_finish(desc, out);
}
static struct shash_alg riscv64_sm3_alg = {
.init = sm3_base_init,
.update = riscv64_sm3_update,
.finup = riscv64_sm3_finup,
.descsize = SM3_STATE_SIZE,
.digestsize = SM3_DIGEST_SIZE,
.base = {
.cra_blocksize = SM3_BLOCK_SIZE,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINUP_MAX,
.cra_priority = 300,
.cra_name = "sm3",
.cra_driver_name = "sm3-riscv64-zvksh-zvkb",
.cra_module = THIS_MODULE,
},
};
static int __init riscv64_sm3_mod_init(void)
{
if (riscv_isa_extension_available(NULL, ZVKSH) &&
riscv_isa_extension_available(NULL, ZVKB) &&
riscv_vector_vlen() >= 128)
return crypto_register_shash(&riscv64_sm3_alg);
return -ENODEV;
}
static void __exit riscv64_sm3_mod_exit(void)
{
crypto_unregister_shash(&riscv64_sm3_alg);
}
module_init(riscv64_sm3_mod_init);
module_exit(riscv64_sm3_mod_exit);
MODULE_DESCRIPTION("SM3 (RISC-V accelerated)");
MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("sm3");

3
arch/s390/configs/debug_defconfig
View File

@ -797,7 +797,7 @@ CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_CRC32=m
@ -809,7 +809,6 @@ CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_CRYPTO_DES_S390=m
CONFIG_CRYPTO_HMAC_S390=m

3
arch/s390/configs/defconfig
View File

@ -781,7 +781,7 @@ CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_SHA3=m
CONFIG_CRYPTO_SM3_GENERIC=m
CONFIG_CRYPTO_SM3=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_CRC32=m
@ -794,7 +794,6 @@ CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_CRYPTO_DES_S390=m
CONFIG_CRYPTO_HMAC_S390=m

10
arch/s390/crypto/Kconfig
View File

@ -2,16 +2,6 @@
menu "Accelerated Cryptographic Algorithms for CPU (s390)"
config CRYPTO_GHASH_S390
tristate "Hash functions: GHASH"
select CRYPTO_HASH
help
GCM GHASH hash function (NIST SP800-38D)
Architecture: s390
It is available as of z196.
config CRYPTO_AES_S390
tristate "Ciphers: AES, modes: ECB, CBC, CTR, XTS, GCM"
select CRYPTO_SKCIPHER

1
arch/s390/crypto/Makefile
View File

@ -7,7 +7,6 @@ obj-$(CONFIG_CRYPTO_DES_S390) += des_s390.o
obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o
obj-$(CONFIG_CRYPTO_PAES_S390) += paes_s390.o
obj-$(CONFIG_S390_PRNG) += prng.o
obj-$(CONFIG_CRYPTO_GHASH_S390) += ghash_s390.o
obj-$(CONFIG_CRYPTO_HMAC_S390) += hmac_s390.o
obj-$(CONFIG_CRYPTO_PHMAC_S390) += phmac_s390.o
obj-y += arch_random.o

144
arch/s390/crypto/ghash_s390.c
View File

@ -1,144 +0,0 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Cryptographic API.
*
* s390 implementation of the GHASH algorithm for GCM (Galois/Counter Mode).
*
* Copyright IBM Corp. 2011
* Author(s): Gerald Schaefer <gerald.schaefer@de.ibm.com>
*/
#include <asm/cpacf.h>
#include <crypto/ghash.h>
#include <crypto/internal/hash.h>
#include <linux/cpufeature.h>
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
struct s390_ghash_ctx {
u8 key[GHASH_BLOCK_SIZE];
};
struct s390_ghash_desc_ctx {
u8 icv[GHASH_BLOCK_SIZE];
u8 key[GHASH_BLOCK_SIZE];
};
static int ghash_init(struct shash_desc *desc)
{
struct s390_ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
struct s390_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memset(dctx, 0, sizeof(*dctx));
memcpy(dctx->key, ctx->key, GHASH_BLOCK_SIZE);
return 0;
}
static int ghash_setkey(struct crypto_shash *tfm,
const u8 *key, unsigned int keylen)
{
struct s390_ghash_ctx *ctx = crypto_shash_ctx(tfm);
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
memcpy(ctx->key, key, GHASH_BLOCK_SIZE);
return 0;
}
static int ghash_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
struct s390_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
unsigned int n;
n = srclen & ~(GHASH_BLOCK_SIZE - 1);
cpacf_kimd(CPACF_KIMD_GHASH, dctx, src, n);
return srclen - n;
}
static void ghash_flush(struct s390_ghash_desc_ctx *dctx, const u8 *src,
unsigned int len)
{
if (len) {
u8 buf[GHASH_BLOCK_SIZE] = {};
memcpy(buf, src, len);
cpacf_kimd(CPACF_KIMD_GHASH, dctx, buf, GHASH_BLOCK_SIZE);
memzero_explicit(buf, sizeof(buf));
}
}
static int ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *dst)
{
struct s390_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
ghash_flush(dctx, src, len);
memcpy(dst, dctx->icv, GHASH_BLOCK_SIZE);
return 0;
}
static int ghash_export(struct shash_desc *desc, void *out)
{
struct s390_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memcpy(out, dctx->icv, GHASH_DIGEST_SIZE);
return 0;
}
static int ghash_import(struct shash_desc *desc, const void *in)
{
struct s390_ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
struct s390_ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memcpy(dctx->icv, in, GHASH_DIGEST_SIZE);
memcpy(dctx->key, ctx->key, GHASH_BLOCK_SIZE);
return 0;
}
static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.finup = ghash_finup,
.setkey = ghash_setkey,
.export = ghash_export,
.import = ghash_import,
.statesize = sizeof(struct ghash_desc_ctx),
.descsize = sizeof(struct s390_ghash_desc_ctx),
.base = {
.cra_name = "ghash",
.cra_driver_name = "ghash-s390",
.cra_priority = 300,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.cra_blocksize = GHASH_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct s390_ghash_ctx),
.cra_module = THIS_MODULE,
},
};
static int __init ghash_mod_init(void)
{
if (!cpacf_query_func(CPACF_KIMD, CPACF_KIMD_GHASH))
return -ENODEV;
return crypto_register_shash(&ghash_alg);
}
static void __exit ghash_mod_exit(void)
{
crypto_unregister_shash(&ghash_alg);
}
module_cpu_feature_match(S390_CPU_FEATURE_MSA, ghash_mod_init);
module_exit(ghash_mod_exit);
MODULE_ALIAS_CRYPTO("ghash");
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("GHASH hash function, s390 implementation");

23
arch/x86/crypto/Kconfig
View File

@ -331,27 +331,4 @@ config CRYPTO_AEGIS128_AESNI_SSE2
- AES-NI (AES New Instructions)
- SSE4.1 (Streaming SIMD Extensions 4.1)
config CRYPTO_SM3_AVX_X86_64
tristate "Hash functions: SM3 (AVX)"
depends on 64BIT
select CRYPTO_HASH
select CRYPTO_LIB_SM3
help
SM3 secure hash function as defined by OSCCA GM/T 0004-2012 SM3
Architecture: x86_64 using:
- AVX (Advanced Vector Extensions)
If unsure, say N.
config CRYPTO_GHASH_CLMUL_NI_INTEL
tristate "Hash functions: GHASH (CLMUL-NI)"
depends on 64BIT
select CRYPTO_CRYPTD
help
GCM GHASH hash function (NIST SP800-38D)
Architecture: x86_64 using:
- CLMUL-NI (carry-less multiplication new instructions)
endmenu

6
arch/x86/crypto/Makefile
View File

@ -50,12 +50,6 @@ aesni-intel-$(CONFIG_64BIT) += aes-ctr-avx-x86_64.o \
aes-gcm-vaes-avx512.o \
aes-xts-avx-x86_64.o
obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
obj-$(CONFIG_CRYPTO_SM3_AVX_X86_64) += sm3-avx-x86_64.o
sm3-avx-x86_64-y := sm3-avx-asm_64.o sm3_avx_glue.o
obj-$(CONFIG_CRYPTO_SM4_AESNI_AVX_X86_64) += sm4-aesni-avx-x86_64.o
sm4-aesni-avx-x86_64-y := sm4-aesni-avx-asm_64.o sm4_aesni_avx_glue.o

1
arch/x86/crypto/aesni-intel_glue.c
View File

@ -25,6 +25,7 @@
#include <crypto/aes.h>
#include <crypto/b128ops.h>
#include <crypto/gcm.h>
#include <crypto/gf128mul.h>
#include <crypto/xts.h>
#include <asm/cpu_device_id.h>
#include <asm/simd.h>

163
arch/x86/crypto/ghash-clmulni-intel_glue.c
View File

@ -1,163 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Accelerated GHASH implementation with Intel PCLMULQDQ-NI
* instructions. This file contains glue code.
*
* Copyright (c) 2009 Intel Corp.
* Author: Huang Ying <ying.huang@intel.com>
*/
#include <asm/cpu_device_id.h>
#include <asm/simd.h>
#include <crypto/b128ops.h>
#include <crypto/ghash.h>
#include <crypto/internal/hash.h>
#include <crypto/utils.h>
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
#include <linux/unaligned.h>
asmlinkage void clmul_ghash_mul(char *dst, const le128 *shash);
asmlinkage int clmul_ghash_update(char *dst, const char *src,
unsigned int srclen, const le128 *shash);
struct x86_ghash_ctx {
le128 shash;
};
static int ghash_init(struct shash_desc *desc)
{
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memset(dctx, 0, sizeof(*dctx));
return 0;
}
static int ghash_setkey(struct crypto_shash *tfm,
const u8 *key, unsigned int keylen)
{
struct x86_ghash_ctx *ctx = crypto_shash_ctx(tfm);
u64 a, b;
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
/*
* GHASH maps bits to polynomial coefficients backwards, which makes it
* hard to implement. But it can be shown that the GHASH multiplication
*
* D * K (mod x^128 + x^7 + x^2 + x + 1)
*
* (where D is a data block and K is the key) is equivalent to:
*
* bitreflect(D) * bitreflect(K) * x^(-127)
* (mod x^128 + x^127 + x^126 + x^121 + 1)
*
* So, the code below precomputes:
*
* bitreflect(K) * x^(-127) (mod x^128 + x^127 + x^126 + x^121 + 1)
*
* ... but in Montgomery form (so that Montgomery multiplication can be
* used), i.e. with an extra x^128 factor, which means actually:
*
* bitreflect(K) * x (mod x^128 + x^127 + x^126 + x^121 + 1)
*
* The within-a-byte part of bitreflect() cancels out GHASH's built-in
* reflection, and thus bitreflect() is actually a byteswap.
*/
a = get_unaligned_be64(key);
b = get_unaligned_be64(key + 8);
ctx->shash.a = cpu_to_le64((a << 1) | (b >> 63));
ctx->shash.b = cpu_to_le64((b << 1) | (a >> 63));
if (a >> 63)
ctx->shash.a ^= cpu_to_le64((u64)0xc2 << 56);
return 0;
}
static int ghash_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
struct x86_ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
u8 *dst = dctx->buffer;
int remain;
kernel_fpu_begin();
remain = clmul_ghash_update(dst, src, srclen, &ctx->shash);
kernel_fpu_end();
return remain;
}
static void ghash_flush(struct x86_ghash_ctx *ctx, struct ghash_desc_ctx *dctx,
const u8 *src, unsigned int len)
{
u8 *dst = dctx->buffer;
kernel_fpu_begin();
if (len) {
crypto_xor(dst, src, len);
clmul_ghash_mul(dst, &ctx->shash);
}
kernel_fpu_end();
}
static int ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *dst)
{
struct x86_ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
u8 *buf = dctx->buffer;
ghash_flush(ctx, dctx, src, len);
memcpy(dst, buf, GHASH_BLOCK_SIZE);
return 0;
}
static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.finup = ghash_finup,
.setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx),
.base = {
.cra_name = "ghash",
.cra_driver_name = "ghash-pclmulqdqni",
.cra_priority = 400,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.cra_blocksize = GHASH_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct x86_ghash_ctx),
.cra_module = THIS_MODULE,
},
};
static const struct x86_cpu_id pcmul_cpu_id[] = {
X86_MATCH_FEATURE(X86_FEATURE_PCLMULQDQ, NULL), /* Pickle-Mickle-Duck */
{}
};
MODULE_DEVICE_TABLE(x86cpu, pcmul_cpu_id);
static int __init ghash_pclmulqdqni_mod_init(void)
{
if (!x86_match_cpu(pcmul_cpu_id))
return -ENODEV;
return crypto_register_shash(&ghash_alg);
}
static void __exit ghash_pclmulqdqni_mod_exit(void)
{
crypto_unregister_shash(&ghash_alg);
}
module_init(ghash_pclmulqdqni_mod_init);
module_exit(ghash_pclmulqdqni_mod_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("GHASH hash function, accelerated by PCLMULQDQ-NI");
MODULE_ALIAS_CRYPTO("ghash");

100
arch/x86/crypto/sm3_avx_glue.c
View File

@ -1,100 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* SM3 Secure Hash Algorithm, AVX assembler accelerated.
* specified in: https://datatracker.ietf.org/doc/html/draft-sca-cfrg-sm3-02
*
* Copyright (C) 2021 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <crypto/internal/hash.h>
#include <crypto/internal/simd.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/cpufeature.h>
#include <linux/kernel.h>
#include <linux/module.h>
asmlinkage void sm3_transform_avx(struct sm3_state *state,
const u8 *data, int nblocks);
static int sm3_avx_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
int remain;
/*
* Make sure struct sm3_state begins directly with the SM3
* 256-bit internal state, as this is what the asm functions expect.
*/
BUILD_BUG_ON(offsetof(struct sm3_state, state) != 0);
kernel_fpu_begin();
remain = sm3_base_do_update_blocks(desc, data, len, sm3_transform_avx);
kernel_fpu_end();
return remain;
}
static int sm3_avx_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
kernel_fpu_begin();
sm3_base_do_finup(desc, data, len, sm3_transform_avx);
kernel_fpu_end();
return sm3_base_finish(desc, out);
}
static struct shash_alg sm3_avx_alg = {
.digestsize = SM3_DIGEST_SIZE,
.init = sm3_base_init,
.update = sm3_avx_update,
.finup = sm3_avx_finup,
.descsize = SM3_STATE_SIZE,
.base = {
.cra_name = "sm3",
.cra_driver_name = "sm3-avx",
.cra_priority = 300,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINUP_MAX,
.cra_blocksize = SM3_BLOCK_SIZE,
.cra_module = THIS_MODULE,
}
};
static int __init sm3_avx_mod_init(void)
{
const char *feature_name;
if (!boot_cpu_has(X86_FEATURE_AVX)) {
pr_info("AVX instruction are not detected.\n");
return -ENODEV;
}
if (!boot_cpu_has(X86_FEATURE_BMI2)) {
pr_info("BMI2 instruction are not detected.\n");
return -ENODEV;
}
if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
&feature_name)) {
pr_info("CPU feature '%s' is not supported.\n", feature_name);
return -ENODEV;
}
return crypto_register_shash(&sm3_avx_alg);
}
static void __exit sm3_avx_mod_exit(void)
{
crypto_unregister_shash(&sm3_avx_alg);
}
module_init(sm3_avx_mod_init);
module_exit(sm3_avx_mod_exit);
MODULE_LICENSE("GPL v2");
MODULE_AUTHOR("Tianjia Zhang <tianjia.zhang@linux.alibaba.com>");
MODULE_DESCRIPTION("SM3 Secure Hash Algorithm, AVX assembler accelerated");
MODULE_ALIAS_CRYPTO("sm3");
MODULE_ALIAS_CRYPTO("sm3-avx");

17
crypto/Kconfig
View File

@ -355,6 +355,8 @@ config CRYPTO_AES
tristate "AES (Advanced Encryption Standard)"
select CRYPTO_ALGAPI
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
help
AES cipher algorithms (Rijndael)(FIPS-197, ISO/IEC 18033-3)
@ -678,7 +680,7 @@ config CRYPTO_ECB
config CRYPTO_HCTR2
tristate "HCTR2"
select CRYPTO_XCTR
select CRYPTO_LIB_POLYVAL
select CRYPTO_LIB_GF128HASH
select CRYPTO_MANAGER
help
HCTR2 length-preserving encryption mode
@ -786,7 +788,7 @@ config CRYPTO_GCM
tristate "GCM (Galois/Counter Mode) and GMAC (GCM MAC)"
select CRYPTO_CTR
select CRYPTO_AEAD
select CRYPTO_GHASH
select CRYPTO_LIB_GF128HASH
select CRYPTO_MANAGER
help
GCM (Galois/Counter Mode) authenticated encryption mode and GMAC
@ -880,13 +882,6 @@ config CRYPTO_CMAC
CMAC (Cipher-based Message Authentication Code) authentication
mode (NIST SP800-38B and IETF RFC4493)
config CRYPTO_GHASH
tristate "GHASH"
select CRYPTO_HASH
select CRYPTO_LIB_GF128MUL
help
GCM GHASH function (NIST SP800-38D)
config CRYPTO_HMAC
tristate "HMAC (Keyed-Hash MAC)"
select CRYPTO_HASH
@ -973,7 +968,7 @@ config CRYPTO_SHA3
help
SHA-3 secure hash algorithms (FIPS 202, ISO/IEC 10118-3)
config CRYPTO_SM3_GENERIC
config CRYPTO_SM3
tristate "SM3 (ShangMi 3)"
select CRYPTO_HASH
select CRYPTO_LIB_SM3
@ -1169,8 +1164,8 @@ endif # if CRYPTO_DRBG_MENU
config CRYPTO_JITTERENTROPY
tristate "CPU Jitter Non-Deterministic RNG (Random Number Generator)"
select CRYPTO_LIB_SHA3
select CRYPTO_RNG
select CRYPTO_SHA3
help
CPU Jitter RNG (Random Number Generator) from the Jitterentropy library

3
crypto/Makefile
View File

@ -82,7 +82,7 @@ obj-$(CONFIG_CRYPTO_SHA1) += sha1.o
obj-$(CONFIG_CRYPTO_SHA256) += sha256.o
obj-$(CONFIG_CRYPTO_SHA512) += sha512.o
obj-$(CONFIG_CRYPTO_SHA3) += sha3.o
obj-$(CONFIG_CRYPTO_SM3_GENERIC) += sm3_generic.o
obj-$(CONFIG_CRYPTO_SM3) += sm3.o
obj-$(CONFIG_CRYPTO_STREEBOG) += streebog_generic.o
obj-$(CONFIG_CRYPTO_WP512) += wp512.o
CFLAGS_wp512.o := $(call cc-option,-fno-schedule-insns) # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149
@ -170,7 +170,6 @@ UBSAN_SANITIZE_jitterentropy.o = n
jitterentropy_rng-y := jitterentropy.o jitterentropy-kcapi.o
obj-$(CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE) += jitterentropy-testing.o
obj-$(CONFIG_CRYPTO_BENCHMARK) += tcrypt.o
obj-$(CONFIG_CRYPTO_GHASH) += ghash-generic.o
obj-$(CONFIG_CRYPTO_USER_API) += af_alg.o
obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o
obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o

183
crypto/aes.c
View File

@ -5,8 +5,10 @@
* Copyright 2026 Google LLC
*/
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/internal/hash.h>
#include <linux/module.h>
static_assert(__alignof__(struct aes_key) <= CRYPTO_MINALIGN);
@ -33,6 +35,98 @@ static void crypto_aes_decrypt(struct crypto_tfm *tfm, u8 *out, const u8 *in)
aes_decrypt(key, out, in);
}
static_assert(__alignof__(struct aes_cmac_key) <= CRYPTO_MINALIGN);
#define AES_CMAC_KEY(tfm) ((struct aes_cmac_key *)crypto_shash_ctx(tfm))
#define AES_CMAC_CTX(desc) ((struct aes_cmac_ctx *)shash_desc_ctx(desc))
static int __maybe_unused crypto_aes_cmac_setkey(struct crypto_shash *tfm,
const u8 *in_key,
unsigned int key_len)
{
return aes_cmac_preparekey(AES_CMAC_KEY(tfm), in_key, key_len);
}
static int __maybe_unused crypto_aes_xcbc_setkey(struct crypto_shash *tfm,
const u8 *in_key,
unsigned int key_len)
{
if (key_len != AES_KEYSIZE_128)
return -EINVAL;
aes_xcbcmac_preparekey(AES_CMAC_KEY(tfm), in_key);
return 0;
}
static int __maybe_unused crypto_aes_cmac_init(struct shash_desc *desc)
{
aes_cmac_init(AES_CMAC_CTX(desc), AES_CMAC_KEY(desc->tfm));
return 0;
}
static int __maybe_unused crypto_aes_cmac_update(struct shash_desc *desc,
const u8 *data,
unsigned int len)
{
aes_cmac_update(AES_CMAC_CTX(desc), data, len);
return 0;
}
static int __maybe_unused crypto_aes_cmac_final(struct shash_desc *desc,
u8 *out)
{
aes_cmac_final(AES_CMAC_CTX(desc), out);
return 0;
}
static int __maybe_unused crypto_aes_cmac_digest(struct shash_desc *desc,
const u8 *data,
unsigned int len, u8 *out)
{
aes_cmac(AES_CMAC_KEY(desc->tfm), data, len, out);
return 0;
}
static_assert(__alignof__(struct aes_enckey) <= CRYPTO_MINALIGN);
#define AES_CBCMAC_KEY(tfm) ((struct aes_enckey *)crypto_shash_ctx(tfm))
#define AES_CBCMAC_CTX(desc) ((struct aes_cbcmac_ctx *)shash_desc_ctx(desc))
static int __maybe_unused crypto_aes_cbcmac_setkey(struct crypto_shash *tfm,
const u8 *in_key,
unsigned int key_len)
{
return aes_prepareenckey(AES_CBCMAC_KEY(tfm), in_key, key_len);
}
static int __maybe_unused crypto_aes_cbcmac_init(struct shash_desc *desc)
{
aes_cbcmac_init(AES_CBCMAC_CTX(desc), AES_CBCMAC_KEY(desc->tfm));
return 0;
}
static int __maybe_unused crypto_aes_cbcmac_update(struct shash_desc *desc,
const u8 *data,
unsigned int len)
{
aes_cbcmac_update(AES_CBCMAC_CTX(desc), data, len);
return 0;
}
static int __maybe_unused crypto_aes_cbcmac_final(struct shash_desc *desc,
u8 *out)
{
aes_cbcmac_final(AES_CBCMAC_CTX(desc), out);
return 0;
}
static int __maybe_unused crypto_aes_cbcmac_digest(struct shash_desc *desc,
const u8 *data,
unsigned int len, u8 *out)
{
aes_cbcmac_init(AES_CBCMAC_CTX(desc), AES_CBCMAC_KEY(desc->tfm));
aes_cbcmac_update(AES_CBCMAC_CTX(desc), data, len);
aes_cbcmac_final(AES_CBCMAC_CTX(desc), out);
return 0;
}
static struct crypto_alg alg = {
.cra_name = "aes",
.cra_driver_name = "aes-lib",
@ -48,19 +142,106 @@ static struct crypto_alg alg = {
.cia_decrypt = crypto_aes_decrypt } }
};
static struct shash_alg mac_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_CMAC)
{
.base.cra_name = "cmac(aes)",
.base.cra_driver_name = "cmac-aes-lib",
.base.cra_priority = 300,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct aes_cmac_key),
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.setkey = crypto_aes_cmac_setkey,
.init = crypto_aes_cmac_init,
.update = crypto_aes_cmac_update,
.final = crypto_aes_cmac_final,
.digest = crypto_aes_cmac_digest,
.descsize = sizeof(struct aes_cmac_ctx),
},
#endif
#if IS_ENABLED(CONFIG_CRYPTO_XCBC)
{
/*
* Note that the only difference between xcbc(aes) and cmac(aes)
* is the preparekey function.
*/
.base.cra_name = "xcbc(aes)",
.base.cra_driver_name = "xcbc-aes-lib",
.base.cra_priority = 300,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct aes_cmac_key),
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.setkey = crypto_aes_xcbc_setkey,
.init = crypto_aes_cmac_init,
.update = crypto_aes_cmac_update,
.final = crypto_aes_cmac_final,
.digest = crypto_aes_cmac_digest,
.descsize = sizeof(struct aes_cmac_ctx),
},
#endif
#if IS_ENABLED(CONFIG_CRYPTO_CCM)
{
.base.cra_name = "cbcmac(aes)",
.base.cra_driver_name = "cbcmac-aes-lib",
.base.cra_priority = 300,
.base.cra_blocksize = AES_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct aes_enckey),
.base.cra_module = THIS_MODULE,
.digestsize = AES_BLOCK_SIZE,
.setkey = crypto_aes_cbcmac_setkey,
.init = crypto_aes_cbcmac_init,
.update = crypto_aes_cbcmac_update,
.final = crypto_aes_cbcmac_final,
.digest = crypto_aes_cbcmac_digest,
.descsize = sizeof(struct aes_cbcmac_ctx),
},
#endif
};
static int __init crypto_aes_mod_init(void)
{
return crypto_register_alg(&alg);
int err = crypto_register_alg(&alg);
if (err)
return err;
if (ARRAY_SIZE(mac_algs) > 0) {
err = crypto_register_shashes(mac_algs, ARRAY_SIZE(mac_algs));
if (err)
goto err_unregister_alg;
} /* Else, CONFIG_CRYPTO_HASH might not be enabled. */
return 0;
err_unregister_alg:
crypto_unregister_alg(&alg);
return err;
}
module_init(crypto_aes_mod_init);
static void __exit crypto_aes_mod_exit(void)
{
if (ARRAY_SIZE(mac_algs) > 0)
crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs));
crypto_unregister_alg(&alg);
}
module_exit(crypto_aes_mod_exit);
MODULE_DESCRIPTION("Crypto API support for AES block cipher");
MODULE_IMPORT_NS("CRYPTO_INTERNAL");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("aes");
MODULE_ALIAS_CRYPTO("aes-lib");
#if IS_ENABLED(CONFIG_CRYPTO_CMAC)
MODULE_ALIAS_CRYPTO("cmac(aes)");
MODULE_ALIAS_CRYPTO("cmac-aes-lib");
#endif
#if IS_ENABLED(CONFIG_CRYPTO_XCBC)
MODULE_ALIAS_CRYPTO("xcbc(aes)");
MODULE_ALIAS_CRYPTO("xcbc-aes-lib");
#endif
#if IS_ENABLED(CONFIG_CRYPTO_CCM)
MODULE_ALIAS_CRYPTO("cbcmac(aes)");
MODULE_ALIAS_CRYPTO("cbcmac-aes-lib");
#endif

427
crypto/gcm.c
View File

@ -5,13 +5,11 @@
* Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen <mh1@iki.fi>
*/
#include <crypto/gf128mul.h>
#include <crypto/internal/aead.h>
#include <crypto/internal/skcipher.h>
#include <crypto/internal/hash.h>
#include <crypto/scatterwalk.h>
#include <crypto/gcm.h>
#include <crypto/hash.h>
#include <crypto/gf128hash.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/kernel.h>
@ -20,12 +18,11 @@
struct gcm_instance_ctx {
struct crypto_skcipher_spawn ctr;
struct crypto_ahash_spawn ghash;
};
struct crypto_gcm_ctx {
struct crypto_skcipher *ctr;
struct crypto_ahash *ghash;
struct ghash_key ghash;
};
struct crypto_rfc4106_ctx {
@ -52,31 +49,15 @@ struct crypto_rfc4543_req_ctx {
struct aead_request subreq;
};
struct crypto_gcm_ghash_ctx {
unsigned int cryptlen;
struct scatterlist *src;
int (*complete)(struct aead_request *req, u32 flags);
};
struct crypto_gcm_req_priv_ctx {
u8 iv[16];
u8 auth_tag[16];
u8 iauth_tag[16];
struct scatterlist src[3];
struct scatterlist dst[3];
struct scatterlist sg;
struct crypto_gcm_ghash_ctx ghash_ctx;
union {
struct ahash_request ahreq;
struct skcipher_request skreq;
} u;
struct skcipher_request skreq; /* Must be last */
};
static struct {
u8 buf[16];
struct scatterlist sg;
} *gcm_zeroes;
static inline struct crypto_gcm_req_priv_ctx *crypto_gcm_reqctx(
struct aead_request *req)
{
@ -89,10 +70,9 @@ static int crypto_gcm_setkey(struct crypto_aead *aead, const u8 *key,
unsigned int keylen)
{
struct crypto_gcm_ctx *ctx = crypto_aead_ctx(aead);
struct crypto_ahash *ghash = ctx->ghash;
struct crypto_skcipher *ctr = ctx->ctr;
struct {
be128 hash;
u8 h[GHASH_BLOCK_SIZE];
u8 iv[16];
struct crypto_wait wait;
@ -115,14 +95,14 @@ static int crypto_gcm_setkey(struct crypto_aead *aead, const u8 *key,
return -ENOMEM;
crypto_init_wait(&data->wait);
sg_init_one(data->sg, &data->hash, sizeof(data->hash));
sg_init_one(data->sg, data->h, sizeof(data->h));
skcipher_request_set_tfm(&data->req, ctr);
skcipher_request_set_callback(&data->req, CRYPTO_TFM_REQ_MAY_SLEEP |
CRYPTO_TFM_REQ_MAY_BACKLOG,
crypto_req_done,
&data->wait);
skcipher_request_set_crypt(&data->req, data->sg, data->sg,
sizeof(data->hash), data->iv);
sizeof(data->h), data->iv);
err = crypto_wait_req(crypto_skcipher_encrypt(&data->req),
&data->wait);
@ -130,10 +110,7 @@ static int crypto_gcm_setkey(struct crypto_aead *aead, const u8 *key,
if (err)
goto out;
crypto_ahash_clear_flags(ghash, CRYPTO_TFM_REQ_MASK);
crypto_ahash_set_flags(ghash, crypto_aead_get_flags(aead) &
CRYPTO_TFM_REQ_MASK);
err = crypto_ahash_setkey(ghash, (u8 *)&data->hash, sizeof(be128));
ghash_preparekey(&ctx->ghash, data->h);
out:
kfree_sensitive(data);
return err;
@ -176,7 +153,7 @@ static void crypto_gcm_init_crypt(struct aead_request *req,
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct crypto_gcm_ctx *ctx = crypto_aead_ctx(aead);
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct skcipher_request *skreq = &pctx->u.skreq;
struct skcipher_request *skreq = &pctx->skreq;
struct scatterlist *dst;
dst = req->src == req->dst ? pctx->src : pctx->dst;
@ -187,244 +164,65 @@ static void crypto_gcm_init_crypt(struct aead_request *req,
pctx->iv);
}
static inline unsigned int gcm_remain(unsigned int len)
static void ghash_update_sg_and_pad(struct ghash_ctx *ghash,
struct scatterlist *sg, unsigned int len)
{
len &= 0xfU;
return len ? 16 - len : 0;
static const u8 zeroes[GHASH_BLOCK_SIZE];
if (len) {
unsigned int pad_len = -len % GHASH_BLOCK_SIZE;
struct scatter_walk walk;
scatterwalk_start(&walk, sg);
do {
unsigned int n = scatterwalk_next(&walk, len);
ghash_update(ghash, walk.addr, n);
scatterwalk_done_src(&walk, n);
len -= n;
} while (len);
if (pad_len)
ghash_update(ghash, zeroes, pad_len);
}
}
static void gcm_hash_len_done(void *data, int err);
static int gcm_hash_update(struct aead_request *req,
crypto_completion_t compl,
struct scatterlist *src,
unsigned int len, u32 flags)
static void gcm_hash(struct aead_request *req, struct scatterlist *ctext,
unsigned int datalen, u8 out[GHASH_BLOCK_SIZE])
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct ahash_request *ahreq = &pctx->u.ahreq;
const struct crypto_gcm_ctx *ctx =
crypto_aead_ctx(crypto_aead_reqtfm(req));
__be64 lengths[2] = {
cpu_to_be64(8 * (u64)req->assoclen),
cpu_to_be64(8 * (u64)datalen),
};
struct ghash_ctx ghash;
ahash_request_set_callback(ahreq, flags, compl, req);
ahash_request_set_crypt(ahreq, src, NULL, len);
ghash_init(&ghash, &ctx->ghash);
return crypto_ahash_update(ahreq);
/* Associated data, then zero-padding to the next 16-byte boundary */
ghash_update_sg_and_pad(&ghash, req->src, req->assoclen);
/* Ciphertext, then zero-padding to the next 16-byte boundary */
ghash_update_sg_and_pad(&ghash, ctext, datalen);
/* Lengths block */
ghash_update(&ghash, (const u8 *)lengths, sizeof(lengths));
ghash_final(&ghash, out);
}
static int gcm_hash_remain(struct aead_request *req,
unsigned int remain,
crypto_completion_t compl, u32 flags)
static int gcm_add_auth_tag(struct aead_request *req)
{
return gcm_hash_update(req, compl, &gcm_zeroes->sg, remain, flags);
}
static int gcm_hash_len(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct ahash_request *ahreq = &pctx->u.ahreq;
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
be128 lengths;
lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(gctx->cryptlen * 8);
memcpy(pctx->iauth_tag, &lengths, 16);
sg_init_one(&pctx->sg, pctx->iauth_tag, 16);
ahash_request_set_callback(ahreq, flags, gcm_hash_len_done, req);
ahash_request_set_crypt(ahreq, &pctx->sg,
pctx->iauth_tag, sizeof(lengths));
return crypto_ahash_finup(ahreq);
}
static int gcm_hash_len_continue(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
return gctx->complete(req, flags);
}
static void gcm_hash_len_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_len_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash_crypt_remain_continue(struct aead_request *req, u32 flags)
{
return gcm_hash_len(req, flags) ?:
gcm_hash_len_continue(req, flags);
}
static void gcm_hash_crypt_remain_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_crypt_remain_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash_crypt_continue(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
unsigned int remain;
remain = gcm_remain(gctx->cryptlen);
if (remain)
return gcm_hash_remain(req, remain,
gcm_hash_crypt_remain_done, flags) ?:
gcm_hash_crypt_remain_continue(req, flags);
return gcm_hash_crypt_remain_continue(req, flags);
}
static void gcm_hash_crypt_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_crypt_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash_assoc_remain_continue(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
if (gctx->cryptlen)
return gcm_hash_update(req, gcm_hash_crypt_done,
gctx->src, gctx->cryptlen, flags) ?:
gcm_hash_crypt_continue(req, flags);
return gcm_hash_crypt_remain_continue(req, flags);
}
static void gcm_hash_assoc_remain_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_assoc_remain_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash_assoc_continue(struct aead_request *req, u32 flags)
{
unsigned int remain;
remain = gcm_remain(req->assoclen);
if (remain)
return gcm_hash_remain(req, remain,
gcm_hash_assoc_remain_done, flags) ?:
gcm_hash_assoc_remain_continue(req, flags);
return gcm_hash_assoc_remain_continue(req, flags);
}
static void gcm_hash_assoc_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_assoc_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash_init_continue(struct aead_request *req, u32 flags)
{
if (req->assoclen)
return gcm_hash_update(req, gcm_hash_assoc_done,
req->src, req->assoclen, flags) ?:
gcm_hash_assoc_continue(req, flags);
return gcm_hash_assoc_remain_continue(req, flags);
}
static void gcm_hash_init_done(void *data, int err)
{
struct aead_request *req = data;
if (err)
goto out;
err = gcm_hash_init_continue(req, 0);
if (err == -EINPROGRESS)
return;
out:
aead_request_complete(req, err);
}
static int gcm_hash(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct ahash_request *ahreq = &pctx->u.ahreq;
struct crypto_gcm_ctx *ctx = crypto_aead_ctx(crypto_aead_reqtfm(req));
ahash_request_set_tfm(ahreq, ctx->ghash);
ahash_request_set_callback(ahreq, flags, gcm_hash_init_done, req);
return crypto_ahash_init(ahreq) ?:
gcm_hash_init_continue(req, flags);
}
static int gcm_enc_copy_hash(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_aead *aead = crypto_aead_reqtfm(req);
u8 *auth_tag = pctx->auth_tag;
crypto_xor(auth_tag, pctx->iauth_tag, 16);
scatterwalk_map_and_copy(auth_tag, req->dst,
req->assoclen + req->cryptlen,
crypto_aead_authsize(aead), 1);
return 0;
}
static int gcm_encrypt_continue(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
gctx->src = sg_next(req->src == req->dst ? pctx->src : pctx->dst);
gctx->cryptlen = req->cryptlen;
gctx->complete = gcm_enc_copy_hash;
return gcm_hash(req, flags);
gcm_hash(req, sg_next(req->src == req->dst ? pctx->src : pctx->dst),
req->cryptlen, pctx->iauth_tag);
crypto_xor(pctx->auth_tag, pctx->iauth_tag, 16);
memcpy_to_sglist(req->dst, req->assoclen + req->cryptlen,
pctx->auth_tag, crypto_aead_authsize(aead));
return 0;
}
static void gcm_encrypt_done(void *data, int err)
@ -434,9 +232,7 @@ static void gcm_encrypt_done(void *data, int err)
if (err)
goto out;
err = gcm_encrypt_continue(req, 0);
if (err == -EINPROGRESS)
return;
err = gcm_add_auth_tag(req);
out:
aead_request_complete(req, err);
@ -445,15 +241,14 @@ out:
static int crypto_gcm_encrypt(struct aead_request *req)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct skcipher_request *skreq = &pctx->u.skreq;
struct skcipher_request *skreq = &pctx->skreq;
u32 flags = aead_request_flags(req);
crypto_gcm_init_common(req);
crypto_gcm_init_crypt(req, req->cryptlen);
skcipher_request_set_callback(skreq, flags, gcm_encrypt_done, req);
return crypto_skcipher_encrypt(skreq) ?:
gcm_encrypt_continue(req, flags);
return crypto_skcipher_encrypt(skreq) ?: gcm_add_auth_tag(req);
}
static int crypto_gcm_verify(struct aead_request *req)
@ -481,35 +276,21 @@ static void gcm_decrypt_done(void *data, int err)
aead_request_complete(req, err);
}
static int gcm_dec_hash_continue(struct aead_request *req, u32 flags)
{
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct skcipher_request *skreq = &pctx->u.skreq;
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
crypto_gcm_init_crypt(req, gctx->cryptlen);
skcipher_request_set_callback(skreq, flags, gcm_decrypt_done, req);
return crypto_skcipher_decrypt(skreq) ?: crypto_gcm_verify(req);
}
static int crypto_gcm_decrypt(struct aead_request *req)
{
struct crypto_aead *aead = crypto_aead_reqtfm(req);
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
unsigned int authsize = crypto_aead_authsize(aead);
unsigned int cryptlen = req->cryptlen;
u32 flags = aead_request_flags(req);
cryptlen -= authsize;
struct skcipher_request *skreq = &pctx->skreq;
unsigned int datalen = req->cryptlen - crypto_aead_authsize(aead);
crypto_gcm_init_common(req);
gctx->src = sg_next(pctx->src);
gctx->cryptlen = cryptlen;
gctx->complete = gcm_dec_hash_continue;
gcm_hash(req, sg_next(pctx->src), datalen, pctx->iauth_tag);
return gcm_hash(req, flags);
crypto_gcm_init_crypt(req, datalen);
skcipher_request_set_callback(skreq, aead_request_flags(req),
gcm_decrypt_done, req);
return crypto_skcipher_decrypt(skreq) ?: crypto_gcm_verify(req);
}
static int crypto_gcm_init_tfm(struct crypto_aead *tfm)
@ -518,43 +299,26 @@ static int crypto_gcm_init_tfm(struct crypto_aead *tfm)
struct gcm_instance_ctx *ictx = aead_instance_ctx(inst);
struct crypto_gcm_ctx *ctx = crypto_aead_ctx(tfm);
struct crypto_skcipher *ctr;
struct crypto_ahash *ghash;
unsigned long align;
int err;
ghash = crypto_spawn_ahash(&ictx->ghash);
if (IS_ERR(ghash))
return PTR_ERR(ghash);
ctr = crypto_spawn_skcipher(&ictx->ctr);
err = PTR_ERR(ctr);
if (IS_ERR(ctr))
goto err_free_hash;
return PTR_ERR(ctr);
ctx->ctr = ctr;
ctx->ghash = ghash;
align = crypto_aead_alignmask(tfm);
align &= ~(crypto_tfm_ctx_alignment() - 1);
crypto_aead_set_reqsize(tfm,
align + offsetof(struct crypto_gcm_req_priv_ctx, u) +
max(sizeof(struct skcipher_request) +
crypto_skcipher_reqsize(ctr),
sizeof(struct ahash_request) +
crypto_ahash_reqsize(ghash)));
align + sizeof(struct crypto_gcm_req_priv_ctx) +
crypto_skcipher_reqsize(ctr));
return 0;
err_free_hash:
crypto_free_ahash(ghash);
return err;
}
static void crypto_gcm_exit_tfm(struct crypto_aead *tfm)
{
struct crypto_gcm_ctx *ctx = crypto_aead_ctx(tfm);
crypto_free_ahash(ctx->ghash);
crypto_free_skcipher(ctx->ctr);
}
@ -563,20 +327,16 @@ static void crypto_gcm_free(struct aead_instance *inst)
struct gcm_instance_ctx *ctx = aead_instance_ctx(inst);
crypto_drop_skcipher(&ctx->ctr);
crypto_drop_ahash(&ctx->ghash);
kfree(inst);
}
static int crypto_gcm_create_common(struct crypto_template *tmpl,
struct rtattr **tb,
const char *ctr_name,
const char *ghash_name)
struct rtattr **tb, const char *ctr_name)
{
struct skcipher_alg_common *ctr;
u32 mask;
struct aead_instance *inst;
struct gcm_instance_ctx *ctx;
struct hash_alg_common *ghash;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_AEAD, &mask);
@ -588,17 +348,6 @@ static int crypto_gcm_create_common(struct crypto_template *tmpl,
return -ENOMEM;
ctx = aead_instance_ctx(inst);
err = crypto_grab_ahash(&ctx->ghash, aead_crypto_instance(inst),
ghash_name, 0, mask);
if (err)
goto err_free_inst;
ghash = crypto_spawn_ahash_alg(&ctx->ghash);
err = -EINVAL;
if (strcmp(ghash->base.cra_name, "ghash") != 0 ||
ghash->digestsize != 16)
goto err_free_inst;
err = crypto_grab_skcipher(&ctx->ctr, aead_crypto_instance(inst),
ctr_name, 0, mask);
if (err)
@ -617,13 +366,11 @@ static int crypto_gcm_create_common(struct crypto_template *tmpl,
goto err_free_inst;
if (snprintf(inst->alg.base.cra_driver_name, CRYPTO_MAX_ALG_NAME,
"gcm_base(%s,%s)", ctr->base.cra_driver_name,
ghash->base.cra_driver_name) >=
CRYPTO_MAX_ALG_NAME)
"gcm_base(%s,ghash-lib)",
ctr->base.cra_driver_name) >= CRYPTO_MAX_ALG_NAME)
goto err_free_inst;
inst->alg.base.cra_priority = (ghash->base.cra_priority +
ctr->base.cra_priority) / 2;
inst->alg.base.cra_priority = ctr->base.cra_priority;
inst->alg.base.cra_blocksize = 1;
inst->alg.base.cra_alignmask = ctr->base.cra_alignmask;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_gcm_ctx);
@ -660,7 +407,7 @@ static int crypto_gcm_create(struct crypto_template *tmpl, struct rtattr **tb)
CRYPTO_MAX_ALG_NAME)
return -ENAMETOOLONG;
return crypto_gcm_create_common(tmpl, tb, ctr_name, "ghash");
return crypto_gcm_create_common(tmpl, tb, ctr_name);
}
static int crypto_gcm_base_create(struct crypto_template *tmpl,
@ -677,7 +424,16 @@ static int crypto_gcm_base_create(struct crypto_template *tmpl,
if (IS_ERR(ghash_name))
return PTR_ERR(ghash_name);
return crypto_gcm_create_common(tmpl, tb, ctr_name, ghash_name);
/*
* Originally this parameter allowed requesting a specific
* implementation of GHASH. This is no longer supported. Now the best
* implementation of GHASH is just always used.
*/
if (strcmp(ghash_name, "ghash") != 0 &&
strcmp(ghash_name, "ghash-lib") != 0)
return -EINVAL;
return crypto_gcm_create_common(tmpl, tb, ctr_name);
}
static int crypto_rfc4106_setkey(struct crypto_aead *parent, const u8 *key,
@ -1096,25 +852,12 @@ static struct crypto_template crypto_gcm_tmpls[] = {
static int __init crypto_gcm_module_init(void)
{
int err;
gcm_zeroes = kzalloc_obj(*gcm_zeroes);
if (!gcm_zeroes)
return -ENOMEM;
sg_init_one(&gcm_zeroes->sg, gcm_zeroes->buf, sizeof(gcm_zeroes->buf));
err = crypto_register_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
if (err)
kfree(gcm_zeroes);
return err;
return crypto_register_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
}
static void __exit crypto_gcm_module_exit(void)
{
kfree(gcm_zeroes);
crypto_unregister_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
}

162
crypto/ghash-generic.c
View File

@ -1,162 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* GHASH: hash function for GCM (Galois/Counter Mode).
*
* Copyright (c) 2007 Nokia Siemens Networks - Mikko Herranen <mh1@iki.fi>
* Copyright (c) 2009 Intel Corp.
* Author: Huang Ying <ying.huang@intel.com>
*/
/*
* GHASH is a keyed hash function used in GCM authentication tag generation.
*
* The original GCM paper [1] presents GHASH as a function GHASH(H, A, C) which
* takes a 16-byte hash key H, additional authenticated data A, and a ciphertext
* C. It formats A and C into a single byte string X, interprets X as a
* polynomial over GF(2^128), and evaluates this polynomial at the point H.
*
* However, the NIST standard for GCM [2] presents GHASH as GHASH(H, X) where X
* is the already-formatted byte string containing both A and C.
*
* "ghash" in the Linux crypto API uses the 'X' (pre-formatted) convention,
* since the API supports only a single data stream per hash. Thus, the
* formatting of 'A' and 'C' is done in the "gcm" template, not in "ghash".
*
* The reason "ghash" is separate from "gcm" is to allow "gcm" to use an
* accelerated "ghash" when a standalone accelerated "gcm(aes)" is unavailable.
* It is generally inappropriate to use "ghash" for other purposes, since it is
* an "ε-almost-XOR-universal hash function", not a cryptographic hash function.
* It can only be used securely in crypto modes specially designed to use it.
*
* [1] The Galois/Counter Mode of Operation (GCM)
* (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.694.695&rep=rep1&type=pdf)
* [2] Recommendation for Block Cipher Modes of Operation: Galois/Counter Mode (GCM) and GMAC
* (https://csrc.nist.gov/publications/detail/sp/800-38d/final)
*/
#include <crypto/gf128mul.h>
#include <crypto/ghash.h>
#include <crypto/internal/hash.h>
#include <crypto/utils.h>
#include <linux/err.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
static int ghash_init(struct shash_desc *desc)
{
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
memset(dctx, 0, sizeof(*dctx));
return 0;
}
static int ghash_setkey(struct crypto_shash *tfm,
const u8 *key, unsigned int keylen)
{
struct ghash_ctx *ctx = crypto_shash_ctx(tfm);
be128 k;
if (keylen != GHASH_BLOCK_SIZE)
return -EINVAL;
if (ctx->gf128)
gf128mul_free_4k(ctx->gf128);
BUILD_BUG_ON(sizeof(k) != GHASH_BLOCK_SIZE);
memcpy(&k, key, GHASH_BLOCK_SIZE); /* avoid violating alignment rules */
ctx->gf128 = gf128mul_init_4k_lle(&k);
memzero_explicit(&k, GHASH_BLOCK_SIZE);
if (!ctx->gf128)
return -ENOMEM;
return 0;
}
static int ghash_update(struct shash_desc *desc,
const u8 *src, unsigned int srclen)
{
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
struct ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
u8 *dst = dctx->buffer;
do {
crypto_xor(dst, src, GHASH_BLOCK_SIZE);
gf128mul_4k_lle((be128 *)dst, ctx->gf128);
src += GHASH_BLOCK_SIZE;
srclen -= GHASH_BLOCK_SIZE;
} while (srclen >= GHASH_BLOCK_SIZE);
return srclen;
}
static void ghash_flush(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
struct ghash_ctx *ctx = crypto_shash_ctx(desc->tfm);
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
u8 *dst = dctx->buffer;
if (len) {
crypto_xor(dst, src, len);
gf128mul_4k_lle((be128 *)dst, ctx->gf128);
}
}
static int ghash_finup(struct shash_desc *desc, const u8 *src,
unsigned int len, u8 *dst)
{
struct ghash_desc_ctx *dctx = shash_desc_ctx(desc);
u8 *buf = dctx->buffer;
ghash_flush(desc, src, len);
memcpy(dst, buf, GHASH_BLOCK_SIZE);
return 0;
}
static void ghash_exit_tfm(struct crypto_tfm *tfm)
{
struct ghash_ctx *ctx = crypto_tfm_ctx(tfm);
if (ctx->gf128)
gf128mul_free_4k(ctx->gf128);
}
static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update,
.finup = ghash_finup,
.setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx),
.base = {
.cra_name = "ghash",
.cra_driver_name = "ghash-generic",
.cra_priority = 100,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
.cra_blocksize = GHASH_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct ghash_ctx),
.cra_module = THIS_MODULE,
.cra_exit = ghash_exit_tfm,
},
};
static int __init ghash_mod_init(void)
{
return crypto_register_shash(&ghash_alg);
}
static void __exit ghash_mod_exit(void)
{
crypto_unregister_shash(&ghash_alg);
}
module_init(ghash_mod_init);
module_exit(ghash_mod_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("GHASH hash function");
MODULE_ALIAS_CRYPTO("ghash");
MODULE_ALIAS_CRYPTO("ghash-generic");

2
crypto/hctr2.c
View File

@ -16,9 +16,9 @@
* (https://eprint.iacr.org/2021/1441.pdf)
*/
#include <crypto/gf128hash.h>
#include <crypto/internal/cipher.h>
#include <crypto/internal/skcipher.h>
#include <crypto/polyval.h>
#include <crypto/scatterwalk.h>
#include <linux/module.h>

116
crypto/jitterentropy-kcapi.c
View File

@ -37,7 +37,6 @@
* DAMAGE.
*/
#include <crypto/hash.h>
#include <crypto/sha3.h>
#include <linux/fips.h>
#include <linux/kernel.h>
@ -48,8 +47,6 @@
#include "jitterentropy.h"
#define JENT_CONDITIONING_HASH "sha3-256"
/***************************************************************************
* Helper function
***************************************************************************/
@ -101,22 +98,14 @@ void jent_get_nstime(__u64 *out)
jent_raw_hires_entropy_store(tmp);
}
int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck)
void jent_hash_time(struct sha3_ctx *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck)
{
struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
SHASH_DESC_ON_STACK(desc, hash_state_desc->tfm);
struct sha3_ctx tmp_state; /* zeroized by sha3_final() */
u8 intermediary[SHA3_256_DIGEST_SIZE];
__u64 j = 0;
int ret;
desc->tfm = hash_state_desc->tfm;
if (sizeof(intermediary) != crypto_shash_digestsize(desc->tfm)) {
pr_warn_ratelimited("Unexpected digest size\n");
return -EINVAL;
}
kmsan_unpoison_memory(intermediary, sizeof(intermediary));
/*
@ -130,24 +119,20 @@ int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
*
* Note, it does not matter which or how much data you inject, we are
* interested in one Keccack1600 compression operation performed with
* the crypto_shash_final.
* the sha3_final.
*/
for (j = 0; j < hash_loop_cnt; j++) {
ret = crypto_shash_init(desc) ?:
crypto_shash_update(desc, intermediary,
sizeof(intermediary)) ?:
crypto_shash_finup(desc, addtl, addtl_len, intermediary);
if (ret)
goto err;
sha3_256_init(&tmp_state);
sha3_update(&tmp_state, intermediary, sizeof(intermediary));
sha3_update(&tmp_state, addtl, addtl_len);
sha3_final(&tmp_state, intermediary);
}
/*
* Inject the data from the previous loop into the pool. This data is
* not considered to contain any entropy, but it stirs the pool a bit.
*/
ret = crypto_shash_update(hash_state_desc, intermediary, sizeof(intermediary));
if (ret)
goto err;
sha3_update(hash_state, intermediary, sizeof(intermediary));
/*
* Insert the time stamp into the hash context representing the pool.
@ -162,30 +147,24 @@ int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
time = 0;
}
ret = crypto_shash_update(hash_state_desc, (u8 *)&time, sizeof(__u64));
err:
shash_desc_zero(desc);
sha3_update(hash_state, (u8 *)&time, sizeof(__u64));
memzero_explicit(intermediary, sizeof(intermediary));
return ret;
}
int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len)
void jent_read_random_block(struct sha3_ctx *hash_state, char *dst,
unsigned int dst_len)
{
struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
u8 jent_block[SHA3_256_DIGEST_SIZE];
/* Obtain data from entropy pool and re-initialize it */
int ret = crypto_shash_final(hash_state_desc, jent_block) ?:
crypto_shash_init(hash_state_desc) ?:
crypto_shash_update(hash_state_desc, jent_block,
sizeof(jent_block));
if (!ret && dst_len)
/* Obtain data from entropy pool and re-initialize it */
sha3_final(hash_state, jent_block);
sha3_256_init(hash_state);
sha3_update(hash_state, jent_block, sizeof(jent_block));
if (dst_len)
memcpy(dst, jent_block, dst_len);
memzero_explicit(jent_block, sizeof(jent_block));
return ret;
}
/***************************************************************************
@ -195,8 +174,7 @@ int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len)
struct jitterentropy {
spinlock_t jent_lock;
struct rand_data *entropy_collector;
struct crypto_shash *tfm;
struct shash_desc *sdesc;
struct sha3_ctx hash_state;
};
static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
@ -205,15 +183,7 @@ static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
spin_lock(&rng->jent_lock);
if (rng->sdesc) {
shash_desc_zero(rng->sdesc);
kfree(rng->sdesc);
}
rng->sdesc = NULL;
if (rng->tfm)
crypto_free_shash(rng->tfm);
rng->tfm = NULL;
memzero_explicit(&rng->hash_state, sizeof(rng->hash_state));
if (rng->entropy_collector)
jent_entropy_collector_free(rng->entropy_collector);
@ -224,34 +194,15 @@ static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
static int jent_kcapi_init(struct crypto_tfm *tfm)
{
struct jitterentropy *rng = crypto_tfm_ctx(tfm);
struct crypto_shash *hash;
struct shash_desc *sdesc;
int size, ret = 0;
int ret = 0;
spin_lock_init(&rng->jent_lock);
/* Use SHA3-256 as conditioner */
hash = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
if (IS_ERR(hash)) {
pr_err("Cannot allocate conditioning digest\n");
return PTR_ERR(hash);
}
rng->tfm = hash;
sha3_256_init(&rng->hash_state);
size = sizeof(struct shash_desc) + crypto_shash_descsize(hash);
sdesc = kmalloc(size, GFP_KERNEL);
if (!sdesc) {
ret = -ENOMEM;
goto err;
}
sdesc->tfm = hash;
crypto_shash_init(sdesc);
rng->sdesc = sdesc;
rng->entropy_collector =
jent_entropy_collector_alloc(CONFIG_CRYPTO_JITTERENTROPY_OSR, 0,
sdesc);
rng->entropy_collector = jent_entropy_collector_alloc(
CONFIG_CRYPTO_JITTERENTROPY_OSR, 0, &rng->hash_state);
if (!rng->entropy_collector) {
ret = -ENOMEM;
goto err;
@ -326,23 +277,16 @@ static struct rng_alg jent_alg = {
static int __init jent_mod_init(void)
{
SHASH_DESC_ON_STACK(desc, tfm);
struct crypto_shash *tfm;
struct sha3_ctx hash_state;
int ret = 0;
jent_testing_init();
tfm = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
if (IS_ERR(tfm)) {
jent_testing_exit();
return PTR_ERR(tfm);
}
sha3_256_init(&hash_state);
desc->tfm = tfm;
crypto_shash_init(desc);
ret = jent_entropy_init(CONFIG_CRYPTO_JITTERENTROPY_OSR, 0, desc, NULL);
shash_desc_zero(desc);
crypto_free_shash(tfm);
ret = jent_entropy_init(CONFIG_CRYPTO_JITTERENTROPY_OSR, 0, &hash_state,
NULL);
memzero_explicit(&hash_state, sizeof(hash_state));
if (ret) {
/* Handle permanent health test error */
if (fips_enabled)

25
crypto/jitterentropy.c
View File

@ -68,7 +68,7 @@ struct rand_data {
* of the RNG are marked as SENSITIVE. A user must not
* access that information while the RNG executes its loops to
* calculate the next random value. */
void *hash_state; /* SENSITIVE hash state entropy pool */
struct sha3_ctx *hash_state; /* SENSITIVE hash state entropy pool */
__u64 prev_time; /* SENSITIVE Previous time stamp */
__u64 last_delta; /* SENSITIVE stuck test */
__s64 last_delta2; /* SENSITIVE stuck test */
@ -417,10 +417,9 @@ static __u64 jent_loop_shuffle(unsigned int bits, unsigned int min)
* time [in] time stamp to be injected
* stuck [in] Is the time stamp identified as stuck?
*
* Output:
* updated hash context in the entropy collector or error code
* Output: updated hash context in the entropy collector
*/
static int jent_condition_data(struct rand_data *ec, __u64 time, int stuck)
static void jent_condition_data(struct rand_data *ec, __u64 time, int stuck)
{
#define SHA3_HASH_LOOP (1<<3)
struct {
@ -435,8 +434,8 @@ static int jent_condition_data(struct rand_data *ec, __u64 time, int stuck)
ec->apt_base
};
return jent_hash_time(ec->hash_state, time, (u8 *)&addtl, sizeof(addtl),
SHA3_HASH_LOOP, stuck);
jent_hash_time(ec->hash_state, time, (u8 *)&addtl, sizeof(addtl),
SHA3_HASH_LOOP, stuck);
}
/*
@ -538,8 +537,7 @@ static int jent_measure_jitter(struct rand_data *ec, __u64 *ret_current_delta)
stuck = jent_stuck(ec, current_delta);
/* Now call the next noise sources which also injects the data */
if (jent_condition_data(ec, current_delta, stuck))
stuck = 1;
jent_condition_data(ec, current_delta, stuck);
/* return the raw entropy value */
if (ret_current_delta)
@ -597,7 +595,7 @@ static void jent_gen_entropy(struct rand_data *ec)
* @return 0 when request is fulfilled or an error
*
* The following error codes can occur:
* -1 entropy_collector is NULL or the generation failed
* -1 entropy_collector is NULL
* -2 Intermittent health failure
* -3 Permanent health failure
*/
@ -640,8 +638,7 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
}
tocopy = min(DATA_SIZE_BITS / 8, len);
if (jent_read_random_block(ec->hash_state, p, tocopy))
return -1;
jent_read_random_block(ec->hash_state, p, tocopy);
len -= tocopy;
p += tocopy;
@ -656,7 +653,7 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
unsigned int flags,
void *hash_state)
struct sha3_ctx *hash_state)
{
struct rand_data *entropy_collector;
@ -704,8 +701,8 @@ void jent_entropy_collector_free(struct rand_data *entropy_collector)
jent_zfree(entropy_collector);
}
int jent_entropy_init(unsigned int osr, unsigned int flags, void *hash_state,
struct rand_data *p_ec)
int jent_entropy_init(unsigned int osr, unsigned int flags,
struct sha3_ctx *hash_state, struct rand_data *p_ec)
{
/*
* If caller provides an allocated ec, reuse it which implies that the

19
crypto/jitterentropy.h
View File

@ -1,24 +1,27 @@
// SPDX-License-Identifier: GPL-2.0-or-later
struct sha3_ctx;
extern void *jent_kvzalloc(unsigned int len);
extern void jent_kvzfree(void *ptr, unsigned int len);
extern void *jent_zalloc(unsigned int len);
extern void jent_zfree(void *ptr);
extern void jent_get_nstime(__u64 *out);
extern int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck);
int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len);
void jent_hash_time(struct sha3_ctx *hash_state, __u64 time, u8 *addtl,
unsigned int addtl_len, __u64 hash_loop_cnt,
unsigned int stuck);
void jent_read_random_block(struct sha3_ctx *hash_state, char *dst,
unsigned int dst_len);
struct rand_data;
extern int jent_entropy_init(unsigned int osr, unsigned int flags,
void *hash_state, struct rand_data *p_ec);
struct sha3_ctx *hash_state,
struct rand_data *p_ec);
extern int jent_read_entropy(struct rand_data *ec, unsigned char *data,
unsigned int len);
extern struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
unsigned int flags,
void *hash_state);
extern struct rand_data *
jent_entropy_collector_alloc(unsigned int osr, unsigned int flags,
struct sha3_ctx *hash_state);
extern void jent_entropy_collector_free(struct rand_data *entropy_collector);
#ifdef CONFIG_CRYPTO_JITTERENTROPY_TESTINTERFACE

89
crypto/sm3.c Normal file
View File

@ -0,0 +1,89 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* SM3 secure hash, as specified by OSCCA GM/T 0004-2012 SM3 and
* described at https://tools.ietf.org/html/draft-shen-sm3-hash-01
*
* Copyright (C) 2017 ARM Limited or its affiliates.
* Written by Gilad Ben-Yossef <gilad@benyossef.com>
* Copyright (C) 2021 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
* Copyright 2026 Google LLC
*/
#include <crypto/internal/hash.h>
#include <crypto/sm3.h>
#include <linux/kernel.h>
#include <linux/module.h>
#define SM3_CTX(desc) ((struct sm3_ctx *)shash_desc_ctx(desc))
static int crypto_sm3_init(struct shash_desc *desc)
{
sm3_init(SM3_CTX(desc));
return 0;
}
static int crypto_sm3_update(struct shash_desc *desc,
const u8 *data, unsigned int len)
{
sm3_update(SM3_CTX(desc), data, len);
return 0;
}
static int crypto_sm3_final(struct shash_desc *desc, u8 *out)
{
sm3_final(SM3_CTX(desc), out);
return 0;
}
static int crypto_sm3_digest(struct shash_desc *desc,
const u8 *data, unsigned int len, u8 *out)
{
sm3(data, len, out);
return 0;
}
static int crypto_sm3_export_core(struct shash_desc *desc, void *out)
{
memcpy(out, SM3_CTX(desc), sizeof(struct sm3_ctx));
return 0;
}
static int crypto_sm3_import_core(struct shash_desc *desc, const void *in)
{
memcpy(SM3_CTX(desc), in, sizeof(struct sm3_ctx));
return 0;
}
static struct shash_alg sm3_alg = {
.base.cra_name = "sm3",
.base.cra_driver_name = "sm3-lib",
.base.cra_priority = 300,
.base.cra_blocksize = SM3_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
.digestsize = SM3_DIGEST_SIZE,
.init = crypto_sm3_init,
.update = crypto_sm3_update,
.final = crypto_sm3_final,
.digest = crypto_sm3_digest,
.export_core = crypto_sm3_export_core,
.import_core = crypto_sm3_import_core,
.descsize = sizeof(struct sm3_ctx),
};
static int __init crypto_sm3_mod_init(void)
{
return crypto_register_shash(&sm3_alg);
}
module_init(crypto_sm3_mod_init);
static void __exit crypto_sm3_mod_exit(void)
{
crypto_unregister_shash(&sm3_alg);
}
module_exit(crypto_sm3_mod_exit);
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("Crypto API support for SM3");
MODULE_ALIAS_CRYPTO("sm3");
MODULE_ALIAS_CRYPTO("sm3-lib");

72
crypto/sm3_generic.c
View File

@ -1,72 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* SM3 secure hash, as specified by OSCCA GM/T 0004-2012 SM3 and
* described at https://tools.ietf.org/html/draft-shen-sm3-hash-01
*
* Copyright (C) 2017 ARM Limited or its affiliates.
* Written by Gilad Ben-Yossef <gilad@benyossef.com>
* Copyright (C) 2021 Tianjia Zhang <tianjia.zhang@linux.alibaba.com>
*/
#include <crypto/internal/hash.h>
#include <crypto/sm3.h>
#include <crypto/sm3_base.h>
#include <linux/kernel.h>
#include <linux/module.h>
const u8 sm3_zero_message_hash[SM3_DIGEST_SIZE] = {
0x1A, 0xB2, 0x1D, 0x83, 0x55, 0xCF, 0xA1, 0x7F,
0x8e, 0x61, 0x19, 0x48, 0x31, 0xE8, 0x1A, 0x8F,
0x22, 0xBE, 0xC8, 0xC7, 0x28, 0xFE, 0xFB, 0x74,
0x7E, 0xD0, 0x35, 0xEB, 0x50, 0x82, 0xAA, 0x2B
};
EXPORT_SYMBOL_GPL(sm3_zero_message_hash);
static int crypto_sm3_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
return sm3_base_do_update_blocks(desc, data, len, sm3_block_generic);
}
static int crypto_sm3_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *hash)
{
sm3_base_do_finup(desc, data, len, sm3_block_generic);
return sm3_base_finish(desc, hash);
}
static struct shash_alg sm3_alg = {
.digestsize = SM3_DIGEST_SIZE,
.init = sm3_base_init,
.update = crypto_sm3_update,
.finup = crypto_sm3_finup,
.descsize = SM3_STATE_SIZE,
.base = {
.cra_name = "sm3",
.cra_driver_name = "sm3-generic",
.cra_priority = 100,
.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
CRYPTO_AHASH_ALG_FINUP_MAX,
.cra_blocksize = SM3_BLOCK_SIZE,
.cra_module = THIS_MODULE,
}
};
static int __init sm3_generic_mod_init(void)
{
return crypto_register_shash(&sm3_alg);
}
static void __exit sm3_generic_mod_fini(void)
{
crypto_unregister_shash(&sm3_alg);
}
module_init(sm3_generic_mod_init);
module_exit(sm3_generic_mod_fini);
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("SM3 Secure Hash Algorithm");
MODULE_ALIAS_CRYPTO("sm3");
MODULE_ALIAS_CRYPTO("sm3-generic");

9
crypto/tcrypt.c
View File

@ -1650,10 +1650,6 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
ret = min(ret, tcrypt_test("rfc4309(ccm(aes))"));
break;
case 46:
ret = min(ret, tcrypt_test("ghash"));
break;
case 48:
ret = min(ret, tcrypt_test("sha3-224"));
break;
@ -2251,11 +2247,6 @@ static int do_test(const char *alg, u32 type, u32 mask, int m, u32 num_mb)
test_hash_speed("blake2b-512", sec, generic_hash_speed_template);
if (mode > 300 && mode < 400) break;
fallthrough;
case 318:
klen = 16;
test_hash_speed("ghash", sec, generic_hash_speed_template);
if (mode > 300 && mode < 400) break;
fallthrough;
case 319:
test_hash_speed("crc32c", sec, generic_hash_speed_template);
if (mode > 300 && mode < 400) break;

28
crypto/testmgr.c
View File

@ -4388,7 +4388,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}, {
#endif
.alg = "cbcmac(aes)",
.generic_driver = "cbcmac(aes-lib)",
.generic_driver = "cbcmac-aes-lib",
.test = alg_test_hash,
.suite = {
.hash = __VECS(aes_cbcmac_tv_template)
@ -4401,7 +4401,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "ccm(aes)",
.generic_driver = "ccm_base(ctr(aes-lib),cbcmac(aes-lib))",
.generic_driver = "ccm_base(ctr(aes-lib),cbcmac-aes-lib)",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
@ -4429,7 +4429,7 @@ static const struct alg_test_desc alg_test_descs[] = {
},
}, {
.alg = "cmac(aes)",
.generic_driver = "cmac(aes-lib)",
.generic_driver = "cmac-aes-lib",
.fips_allowed = 1,
.test = alg_test_hash,
.suite = {
@ -4965,7 +4965,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}, {
#endif /* CONFIG_CRYPTO_DH_RFC7919_GROUPS */
.alg = "gcm(aes)",
.generic_driver = "gcm_base(ctr(aes-lib),ghash-generic)",
.generic_driver = "gcm_base(ctr(aes-lib),ghash-lib)",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
@ -4973,24 +4973,18 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "gcm(aria)",
.generic_driver = "gcm_base(ctr(aria-generic),ghash-generic)",
.generic_driver = "gcm_base(ctr(aria-generic),ghash-lib)",
.test = alg_test_aead,
.suite = {
.aead = __VECS(aria_gcm_tv_template)
}
}, {
.alg = "gcm(sm4)",
.generic_driver = "gcm_base(ctr(sm4-generic),ghash-generic)",
.generic_driver = "gcm_base(ctr(sm4-generic),ghash-lib)",
.test = alg_test_aead,
.suite = {
.aead = __VECS(sm4_gcm_tv_template)
}
}, {
.alg = "ghash",
.test = alg_test_hash,
.suite = {
.hash = __VECS(ghash_tv_template)
}
}, {
.alg = "hctr2(aes)",
.generic_driver = "hctr2_base(xctr(aes-lib),polyval-lib)",
@ -5085,6 +5079,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "hmac(sm3)",
.generic_driver = "hmac(sm3-lib)",
.test = alg_test_hash,
.suite = {
.hash = __VECS(hmac_sm3_tv_template)
@ -5314,7 +5309,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "rfc4106(gcm(aes))",
.generic_driver = "rfc4106(gcm_base(ctr(aes-lib),ghash-generic))",
.generic_driver = "rfc4106(gcm_base(ctr(aes-lib),ghash-lib))",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
@ -5326,7 +5321,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "rfc4309(ccm(aes))",
.generic_driver = "rfc4309(ccm_base(ctr(aes-lib),cbcmac(aes-lib)))",
.generic_driver = "rfc4309(ccm_base(ctr(aes-lib),cbcmac-aes-lib))",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
@ -5338,7 +5333,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "rfc4543(gcm(aes))",
.generic_driver = "rfc4543(gcm_base(ctr(aes-lib),ghash-generic))",
.generic_driver = "rfc4543(gcm_base(ctr(aes-lib),ghash-lib))",
.test = alg_test_aead,
.suite = {
.aead = {
@ -5452,6 +5447,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "sm3",
.generic_driver = "sm3-lib",
.test = alg_test_hash,
.suite = {
.hash = __VECS(sm3_tv_template)
@ -5515,7 +5511,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "xcbc(aes)",
.generic_driver = "xcbc(aes-lib)",
.generic_driver = "xcbc-aes-lib",
.test = alg_test_hash,
.suite = {
.hash = __VECS(aes_xcbc128_tv_template)

109
crypto/testmgr.h
View File

@ -6183,115 +6183,6 @@ static const struct hash_testvec wp256_tv_template[] = {
},
};
static const struct hash_testvec ghash_tv_template[] =
{
{
.key = "\xdf\xa6\xbf\x4d\xed\x81\xdb\x03"
"\xff\xca\xff\x95\xf8\x30\xf0\x61",
.ksize = 16,
.plaintext = "\x95\x2b\x2a\x56\xa5\x60\x04a\xc0"
"\xb3\x2b\x66\x56\xa0\x5b\x40\xb6",
.psize = 16,
.digest = "\xda\x53\xeb\x0a\xd2\xc5\x5b\xb6"
"\x4f\xc4\x80\x2c\xc3\xfe\xda\x60",
}, {
.key = "\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b"
"\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b",
.ksize = 16,
.plaintext = "what do ya want for nothing?",
.psize = 28,
.digest = "\x3e\x1f\x5c\x4d\x65\xf0\xef\xce"
"\x0d\x61\x06\x27\x66\x51\xd5\xe2",
}, {
.key = "\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa"
"\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa",
.ksize = 16,
.plaintext = "\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd"
"\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd"
"\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd"
"\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd\xdd",
.psize = 50,
.digest = "\xfb\x49\x8a\x36\xe1\x96\xe1\x96"
"\xe1\x96\xe1\x96\xe1\x96\xe1\x96",
}, {
.key = "\xda\x53\xeb\x0a\xd2\xc5\x5b\xb6"
"\x4f\xc4\x80\x2c\xc3\xfe\xda\x60",
.ksize = 16,
.plaintext = "\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd"
"\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd"
"\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd"
"\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd\xcd",
.psize = 50,
.digest = "\x2b\x5c\x0c\x7f\x52\xd1\x60\xc2"
"\x49\xed\x6e\x32\x7a\xa9\xbe\x08",
}, {
.key = "\x95\x2b\x2a\x56\xa5\x60\x04a\xc0"
"\xb3\x2b\x66\x56\xa0\x5b\x40\xb6",
.ksize = 16,
.plaintext = "Test With Truncation",
.psize = 20,
.digest = "\xf8\x94\x87\x2a\x4b\x63\x99\x28"
"\x23\xf7\x93\xf7\x19\xf5\x96\xd9",
}, {
.key = "\x0a\x1b\x2c\x3d\x4e\x5f\x64\x71"
"\x82\x93\xa4\xb5\xc6\xd7\xe8\xf9",
.ksize = 16,
.plaintext = "\x56\x6f\x72\x20\x6c\x61\x75\x74"
"\x65\x72\x20\x4c\x61\x75\x73\x63"
"\x68\x65\x6e\x20\x75\x6e\x64\x20"
"\x53\x74\x61\x75\x6e\x65\x6e\x20"
"\x73\x65\x69\x20\x73\x74\x69\x6c"
"\x6c\x2c\x0a\x64\x75\x20\x6d\x65"
"\x69\x6e\x20\x74\x69\x65\x66\x74"
"\x69\x65\x66\x65\x73\x20\x4c\x65"
"\x62\x65\x6e\x3b\x0a\x64\x61\x73"
"\x73\x20\x64\x75\x20\x77\x65\x69"
"\xc3\x9f\x74\x20\x77\x61\x73\x20"
"\x64\x65\x72\x20\x57\x69\x6e\x64"
"\x20\x64\x69\x72\x20\x77\x69\x6c"
"\x6c\x2c\x0a\x65\x68\x20\x6e\x6f"
"\x63\x68\x20\x64\x69\x65\x20\x42"
"\x69\x72\x6b\x65\x6e\x20\x62\x65"
"\x62\x65\x6e\x2e\x0a\x0a\x55\x6e"
"\x64\x20\x77\x65\x6e\x6e\x20\x64"
"\x69\x72\x20\x65\x69\x6e\x6d\x61"
"\x6c\x20\x64\x61\x73\x20\x53\x63"
"\x68\x77\x65\x69\x67\x65\x6e\x20"
"\x73\x70\x72\x61\x63\x68\x2c\x0a"
"\x6c\x61\x73\x73\x20\x64\x65\x69"
"\x6e\x65\x20\x53\x69\x6e\x6e\x65"
"\x20\x62\x65\x73\x69\x65\x67\x65"
"\x6e\x2e\x0a\x4a\x65\x64\x65\x6d"
"\x20\x48\x61\x75\x63\x68\x65\x20"
"\x67\x69\x62\x74\x20\x64\x69\x63"
"\x68\x2c\x20\x67\x69\x62\x20\x6e"
"\x61\x63\x68\x2c\x0a\x65\x72\x20"
"\x77\x69\x72\x64\x20\x64\x69\x63"
"\x68\x20\x6c\x69\x65\x62\x65\x6e"
"\x20\x75\x6e\x64\x20\x77\x69\x65"
"\x67\x65\x6e\x2e\x0a\x0a\x55\x6e"
"\x64\x20\x64\x61\x6e\x6e\x20\x6d"
"\x65\x69\x6e\x65\x20\x53\x65\x65"
"\x6c\x65\x20\x73\x65\x69\x74\x20"
"\x77\x65\x69\x74\x2c\x20\x73\x65"
"\x69\x20\x77\x65\x69\x74\x2c\x0a"
"\x64\x61\x73\x73\x20\x64\x69\x72"
"\x20\x64\x61\x73\x20\x4c\x65\x62"
"\x65\x6e\x20\x67\x65\x6c\x69\x6e"
"\x67\x65\x2c\x0a\x62\x72\x65\x69"
"\x74\x65\x20\x64\x69\x63\x68\x20"
"\x77\x69\x65\x20\x65\x69\x6e\x20"
"\x46\x65\x69\x65\x72\x6b\x6c\x65"
"\x69\x64\x0a\xc3\xbc\x62\x65\x72"
"\x20\x64\x69\x65\x20\x73\x69\x6e"
"\x6e\x65\x6e\x64\x65\x6e\x20\x44"
"\x69\x6e\x67\x65\x2e\x2e\x2e\x0a",
.psize = 400,
.digest = "\xad\xb1\xc1\xe9\x56\x70\x31\x1d"
"\xbb\x5b\xdf\x5e\x70\x72\x1a\x57",
},
};
/*
* HMAC-MD5 test vectors from RFC2202
* (These need to be fixed to not use strlen).

2
drivers/crypto/Kconfig
View File

@ -843,7 +843,7 @@ config CRYPTO_DEV_CCREE
select CRYPTO_CTR
select CRYPTO_XTS
select CRYPTO_SM4_GENERIC
select CRYPTO_SM3_GENERIC
select CRYPTO_SM3
help
Say 'Y' to enable a driver for the REE interface of the Arm
TrustZone CryptoCell family of processors. Currently the

2
drivers/crypto/starfive/Kconfig
View File

@ -10,7 +10,7 @@ config CRYPTO_DEV_JH7110
select CRYPTO_HMAC
select CRYPTO_SHA256
select CRYPTO_SHA512
select CRYPTO_SM3_GENERIC
select CRYPTO_SM3
select CRYPTO_RSA
select CRYPTO_AES
select CRYPTO_CCM

4
drivers/crypto/starfive/jh7110-aes.c
View File

@ -1003,12 +1003,12 @@ static int starfive_aes_ctr_init_tfm(struct crypto_skcipher *tfm)
static int starfive_aes_ccm_init_tfm(struct crypto_aead *tfm)
{
return starfive_aes_aead_init_tfm(tfm, "ccm_base(ctr(aes-lib),cbcmac(aes-lib))");
return starfive_aes_aead_init_tfm(tfm, "ccm_base(ctr(aes-lib),cbcmac-aes-lib)");
}
static int starfive_aes_gcm_init_tfm(struct crypto_aead *tfm)
{
return starfive_aes_aead_init_tfm(tfm, "gcm_base(ctr(aes-lib),ghash-generic)");
return starfive_aes_aead_init_tfm(tfm, "gcm_base(ctr(aes-lib),ghash-lib)");
}
static struct skcipher_engine_alg skcipher_algs[] = {

8
drivers/crypto/starfive/jh7110-hash.c
View File

@ -520,7 +520,7 @@ static int starfive_sha512_init_tfm(struct crypto_ahash *hash)
static int starfive_sm3_init_tfm(struct crypto_ahash *hash)
{
return starfive_hash_init_tfm(hash, "sm3-generic",
return starfive_hash_init_tfm(hash, "sm3-lib",
STARFIVE_HASH_SM3, 0);
}
@ -550,7 +550,7 @@ static int starfive_hmac_sha512_init_tfm(struct crypto_ahash *hash)
static int starfive_hmac_sm3_init_tfm(struct crypto_ahash *hash)
{
return starfive_hash_init_tfm(hash, "hmac(sm3-generic)",
return starfive_hash_init_tfm(hash, "hmac(sm3-lib)",
STARFIVE_HASH_SM3, 1);
}
@ -795,7 +795,7 @@ static struct ahash_engine_alg algs_sha2_sm3[] = {
.base.exit_tfm = starfive_hash_exit_tfm,
.base.halg = {
.digestsize = SM3_DIGEST_SIZE,
.statesize = sizeof(struct sm3_state),
.statesize = sizeof(struct sm3_ctx),
.base = {
.cra_name = "sm3",
.cra_driver_name = "sm3-starfive",
@ -824,7 +824,7 @@ static struct ahash_engine_alg algs_sha2_sm3[] = {
.base.setkey = starfive_hash_setkey,
.base.halg = {
.digestsize = SM3_DIGEST_SIZE,
.statesize = sizeof(struct sm3_state),
.statesize = sizeof(struct sm3_ctx),
.base = {
.cra_name = "hmac(sm3)",
.cra_driver_name = "sm3-hmac-starfive",

1
drivers/virt/coco/guest/Kconfig
View File

@ -13,5 +13,4 @@ config TSM_REPORTS
config TSM_MEASUREMENTS
select TSM_GUEST
select CRYPTO_HASH_INFO
select CRYPTO
bool

154
include/crypto/aes-cbc-macs.h Normal file
View File

@ -0,0 +1,154 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Support for AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC
*
* Copyright 2026 Google LLC
*/
#ifndef _CRYPTO_AES_CBC_MACS_H
#define _CRYPTO_AES_CBC_MACS_H
#include <crypto/aes.h>
/**
* struct aes_cmac_key - Prepared key for AES-CMAC or AES-XCBC-MAC
* @aes: The AES key for cipher block chaining
* @k_final: Finalization subkeys for the final block.
* k_final[0] (CMAC K1, XCBC-MAC K2) is used if it's a full block.
* k_final[1] (CMAC K2, XCBC-MAC K3) is used if it's a partial block.
*/
struct aes_cmac_key {
struct aes_enckey aes;
union {
u8 b[AES_BLOCK_SIZE];
__be64 w[2];
} k_final[2];
};
/**
* struct aes_cmac_ctx - Context for computing an AES-CMAC or AES-XCBC-MAC value
* @key: Pointer to the key struct. A pointer is used rather than a copy of the
* struct, since the key struct size may be large. It is assumed that the
* key lives at least as long as the context.
* @partial_len: Number of bytes that have been XOR'ed into @h since the last
* AES encryption. This is 0 if no data has been processed yet,
* or between 1 and AES_BLOCK_SIZE inclusive otherwise.
* @h: The current chaining value
*/
struct aes_cmac_ctx {
const struct aes_cmac_key *key;
size_t partial_len;
u8 h[AES_BLOCK_SIZE];
};
/**
* aes_cmac_preparekey() - Prepare a key for AES-CMAC
* @key: (output) The key struct to initialize
* @in_key: The raw AES key
* @key_len: Length of the raw key in bytes. The supported values are
* AES_KEYSIZE_128, AES_KEYSIZE_192, and AES_KEYSIZE_256.
*
* Context: Any context.
* Return: 0 on success or -EINVAL if the given key length is invalid. No other
* errors are possible, so callers that always pass a valid key length
* don't need to check for errors.
*/
int aes_cmac_preparekey(struct aes_cmac_key *key, const u8 *in_key,
size_t key_len);
/**
* aes_xcbcmac_preparekey() - Prepare a key for AES-XCBC-MAC
* @key: (output) The key struct to initialize
* @in_key: The raw key. As per the AES-XCBC-MAC specification (RFC 3566), this
* is 128 bits, matching the internal use of AES-128.
*
* AES-XCBC-MAC and AES-CMAC are the same except for the key preparation. After
* that step, AES-XCBC-MAC is supported via the aes_cmac_* functions.
*
* New users should use AES-CMAC instead of AES-XCBC-MAC.
*
* Context: Any context.
*/
void aes_xcbcmac_preparekey(struct aes_cmac_key *key,
const u8 in_key[at_least AES_KEYSIZE_128]);
/**
* aes_cmac_init() - Start computing an AES-CMAC or AES-XCBC-MAC value
* @ctx: (output) The context to initialize
* @key: The key to use. Note that a pointer to the key is saved in the
* context, so the key must live at least as long as the context.
*
* This supports both AES-CMAC and AES-XCBC-MAC. Which one is done depends on
* whether aes_cmac_preparekey() or aes_xcbcmac_preparekey() was called.
*/
static inline void aes_cmac_init(struct aes_cmac_ctx *ctx,
const struct aes_cmac_key *key)
{
*ctx = (struct aes_cmac_ctx){ .key = key };
}
/**
* aes_cmac_update() - Update an AES-CMAC or AES-XCBC-MAC context with more data
* @ctx: The context to update; must have been initialized
* @data: The message data
* @data_len: The data length in bytes. Doesn't need to be block-aligned.
*
* This can be called any number of times.
*
* Context: Any context.
*/
void aes_cmac_update(struct aes_cmac_ctx *ctx, const u8 *data, size_t data_len);
/**
* aes_cmac_final() - Finish computing an AES-CMAC or AES-XCBC-MAC value
* @ctx: The context to finalize; must have been initialized
* @out: (output) The resulting MAC
*
* After finishing, this zeroizes @ctx. So the caller does not need to do it.
*
* Context: Any context.
*/
void aes_cmac_final(struct aes_cmac_ctx *ctx, u8 out[at_least AES_BLOCK_SIZE]);
/**
* aes_cmac() - Compute AES-CMAC or AES-XCBC-MAC in one shot
* @key: The key to use
* @data: The message data
* @data_len: The data length in bytes
* @out: (output) The resulting AES-CMAC or AES-XCBC-MAC value
*
* This supports both AES-CMAC and AES-XCBC-MAC. Which one is done depends on
* whether aes_cmac_preparekey() or aes_xcbcmac_preparekey() was called.
*
* Context: Any context.
*/
static inline void aes_cmac(const struct aes_cmac_key *key, const u8 *data,
size_t data_len, u8 out[at_least AES_BLOCK_SIZE])
{
struct aes_cmac_ctx ctx;
aes_cmac_init(&ctx, key);
aes_cmac_update(&ctx, data, data_len);
aes_cmac_final(&ctx, out);
}
/*
* AES-CBC-MAC support. This is provided only for use by the implementation of
* AES-CCM. It should have no other users. Warning: unlike AES-CMAC and
* AES-XCBC-MAC, AES-CBC-MAC isn't a secure MAC for variable-length messages.
*/
struct aes_cbcmac_ctx {
const struct aes_enckey *key;
size_t partial_len;
u8 h[AES_BLOCK_SIZE];
};
static inline void aes_cbcmac_init(struct aes_cbcmac_ctx *ctx,
const struct aes_enckey *key)
{
*ctx = (struct aes_cbcmac_ctx){ .key = key };
}
void aes_cbcmac_update(struct aes_cbcmac_ctx *ctx, const u8 *data,
size_t data_len);
void aes_cbcmac_final(struct aes_cbcmac_ctx *ctx,
u8 out[at_least AES_BLOCK_SIZE]);
#endif /* _CRYPTO_AES_CBC_MACS_H */

66
include/crypto/aes.h
View File

@ -167,6 +167,72 @@ int aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key,
#ifdef CONFIG_ARM64
int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key,
unsigned int key_len);
asmlinkage void neon_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void neon_aes_ecb_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void neon_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void neon_aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void neon_aes_cbc_cts_encrypt(u8 out[], u8 const in[],
u32 const rk[], int rounds, int bytes,
u8 const iv[]);
asmlinkage void neon_aes_cbc_cts_decrypt(u8 out[], u8 const in[],
u32 const rk[], int rounds, int bytes,
u8 const iv[]);
asmlinkage void neon_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[]);
asmlinkage void neon_aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[],
int byte_ctr);
asmlinkage void neon_aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[],
u8 iv[], int first);
asmlinkage void neon_aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[],
u8 iv[], int first);
asmlinkage void neon_aes_essiv_cbc_encrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds,
int blocks, u8 iv[],
u32 const rk2[]);
asmlinkage void neon_aes_essiv_cbc_decrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds,
int blocks, u8 iv[],
u32 const rk2[]);
asmlinkage void ce_aes_ecb_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void ce_aes_ecb_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks);
asmlinkage void ce_aes_cbc_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void ce_aes_cbc_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int blocks, u8 iv[]);
asmlinkage void ce_aes_cbc_cts_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 const iv[]);
asmlinkage void ce_aes_cbc_cts_decrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 const iv[]);
asmlinkage void ce_aes_ctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[]);
asmlinkage void ce_aes_xctr_encrypt(u8 out[], u8 const in[], u32 const rk[],
int rounds, int bytes, u8 ctr[],
int byte_ctr);
asmlinkage void ce_aes_xts_encrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[],
u8 iv[], int first);
asmlinkage void ce_aes_xts_decrypt(u8 out[], u8 const in[], u32 const rk1[],
int rounds, int bytes, u32 const rk2[],
u8 iv[], int first);
asmlinkage void ce_aes_essiv_cbc_encrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds,
int blocks, u8 iv[], u32 const rk2[]);
asmlinkage void ce_aes_essiv_cbc_decrypt(u8 out[], u8 const in[],
u32 const rk1[], int rounds,
int blocks, u8 iv[], u32 const rk2[]);
asmlinkage void ce_aes_mac_update(u8 const in[], u32 const rk[], int rounds,
size_t blocks, u8 dg[], int enc_before,
int enc_after);
#elif defined(CONFIG_PPC)
void ppc_expand_key_128(u32 *key_enc, const u8 *key);
void ppc_expand_key_192(u32 *key_enc, const u8 *key);

2
include/crypto/chacha20poly1305.h
View File

@ -46,6 +46,4 @@ bool chacha20poly1305_decrypt_sg_inplace(struct scatterlist *src, size_t src_len
const u64 nonce,
const u8 key[at_least CHACHA20POLY1305_KEY_SIZE]);
bool chacha20poly1305_selftest(void);
#endif /* __CHACHA20POLY1305_H */

4
include/crypto/gcm.h
View File

@ -4,7 +4,7 @@
#include <linux/errno.h>
#include <crypto/aes.h>
#include <crypto/gf128mul.h>
#include <crypto/gf128hash.h>
#define GCM_AES_IV_SIZE 12
#define GCM_RFC4106_IV_SIZE 8
@ -65,7 +65,7 @@ static inline int crypto_ipsec_check_assoclen(unsigned int assoclen)
}
struct aesgcm_ctx {
be128 ghash_key;
struct ghash_key ghash_key;
struct aes_enckey aes_key;
unsigned int authsize;
};

126
include/crypto/polyval.h → include/crypto/gf128hash.h
View File

@ -1,13 +1,14 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* POLYVAL library API
* GF(2^128) polynomial hashing: GHASH and POLYVAL
*
* Copyright 2025 Google LLC
*/
#ifndef _CRYPTO_POLYVAL_H
#define _CRYPTO_POLYVAL_H
#ifndef _CRYPTO_GF128HASH_H
#define _CRYPTO_GF128HASH_H
#include <crypto/ghash.h>
#include <linux/string.h>
#include <linux/types.h>
@ -33,6 +34,24 @@ struct polyval_elem {
};
};
/**
* struct ghash_key - Prepared key for GHASH
*
* Use ghash_preparekey() to initialize this.
*/
struct ghash_key {
#if defined(CONFIG_CRYPTO_LIB_GF128HASH_ARCH) && defined(CONFIG_PPC64)
/** @htable: GHASH key format used by the POWER8 assembly code */
u64 htable[4][2];
#elif defined(CONFIG_CRYPTO_LIB_GF128HASH_ARCH) && \
(defined(CONFIG_RISCV) || defined(CONFIG_S390))
/** @h_raw: The hash key H, in GHASH format */
u8 h_raw[GHASH_BLOCK_SIZE];
#endif
/** @h: The hash key H, in POLYVAL format */
struct polyval_elem h;
};
/**
* struct polyval_key - Prepared key for POLYVAL
*
@ -44,20 +63,28 @@ struct polyval_elem {
* exponentiation repeats the POLYVAL dot operation, with its "extra" x^-128.
*/
struct polyval_key {
#ifdef CONFIG_CRYPTO_LIB_POLYVAL_ARCH
#ifdef CONFIG_ARM64
/** @h_powers: Powers of the hash key H^8 through H^1 */
struct polyval_elem h_powers[8];
#elif defined(CONFIG_X86)
#if defined(CONFIG_CRYPTO_LIB_GF128HASH_ARCH) && \
(defined(CONFIG_ARM64) || defined(CONFIG_X86))
/** @h_powers: Powers of the hash key H^8 through H^1 */
struct polyval_elem h_powers[8];
#else
#error "Unhandled arch"
#endif
#else /* CONFIG_CRYPTO_LIB_POLYVAL_ARCH */
/** @h: The hash key H */
struct polyval_elem h;
#endif /* !CONFIG_CRYPTO_LIB_POLYVAL_ARCH */
#endif
};
/**
* struct ghash_ctx - Context for computing a GHASH value
* @key: Pointer to the prepared GHASH key. The user of the API is
* responsible for ensuring that the key lives as long as the context.
* @acc: The accumulator. It is stored in POLYVAL format rather than GHASH
* format, since most implementations want it in POLYVAL format.
* @partial: Number of data bytes processed so far modulo GHASH_BLOCK_SIZE
*/
struct ghash_ctx {
const struct ghash_key *key;
struct polyval_elem acc;
size_t partial;
};
/**
@ -73,6 +100,18 @@ struct polyval_ctx {
size_t partial;
};
/**
* ghash_preparekey() - Prepare a GHASH key
* @key: (output) The key structure to initialize
* @raw_key: The raw hash key
*
* Initialize a GHASH key structure from a raw key.
*
* Context: Any context.
*/
void ghash_preparekey(struct ghash_key *key,
const u8 raw_key[GHASH_BLOCK_SIZE]);
/**
* polyval_preparekey() - Prepare a POLYVAL key
* @key: (output) The key structure to initialize
@ -84,18 +123,20 @@ struct polyval_ctx {
*
* Context: Any context.
*/
#ifdef CONFIG_CRYPTO_LIB_POLYVAL_ARCH
void polyval_preparekey(struct polyval_key *key,
const u8 raw_key[POLYVAL_BLOCK_SIZE]);
#else
static inline void polyval_preparekey(struct polyval_key *key,
const u8 raw_key[POLYVAL_BLOCK_SIZE])
/**
* ghash_init() - Initialize a GHASH context for a new message
* @ctx: The context to initialize
* @key: The key to use. Note that a pointer to the key is saved in the
* context, so the key must live at least as long as the context.
*/
static inline void ghash_init(struct ghash_ctx *ctx,
const struct ghash_key *key)
{
/* Just a simple copy, so inline it. */
memcpy(key->h.bytes, raw_key, POLYVAL_BLOCK_SIZE);
*ctx = (struct ghash_ctx){ .key = key };
}
#endif
/**
* polyval_init() - Initialize a POLYVAL context for a new message
@ -141,6 +182,18 @@ static inline void polyval_export_blkaligned(const struct polyval_ctx *ctx,
*acc = ctx->acc;
}
/**
* ghash_update() - Update a GHASH context with message data
* @ctx: The context to update; must have been initialized
* @data: The message data
* @len: The data length in bytes. Doesn't need to be block-aligned.
*
* This can be called any number of times.
*
* Context: Any context.
*/
void ghash_update(struct ghash_ctx *ctx, const u8 *data, size_t len);
/**
* polyval_update() - Update a POLYVAL context with message data
* @ctx: The context to update; must have been initialized
@ -153,6 +206,20 @@ static inline void polyval_export_blkaligned(const struct polyval_ctx *ctx,
*/
void polyval_update(struct polyval_ctx *ctx, const u8 *data, size_t len);
/**
* ghash_final() - Finish computing a GHASH value
* @ctx: The context to finalize
* @out: The output value
*
* If the total data length isn't a multiple of GHASH_BLOCK_SIZE, then the
* final block is automatically zero-padded.
*
* After finishing, this zeroizes @ctx. So the caller does not need to do it.
*
* Context: Any context.
*/
void ghash_final(struct ghash_ctx *ctx, u8 out[GHASH_BLOCK_SIZE]);
/**
* polyval_final() - Finish computing a POLYVAL value
* @ctx: The context to finalize
@ -167,6 +234,25 @@ void polyval_update(struct polyval_ctx *ctx, const u8 *data, size_t len);
*/
void polyval_final(struct polyval_ctx *ctx, u8 out[POLYVAL_BLOCK_SIZE]);
/**
* ghash() - Compute a GHASH value
* @key: The prepared key
* @data: The message data
* @len: The data length in bytes. Doesn't need to be block-aligned.
* @out: The output value
*
* Context: Any context.
*/
static inline void ghash(const struct ghash_key *key, const u8 *data,
size_t len, u8 out[GHASH_BLOCK_SIZE])
{
struct ghash_ctx ctx;
ghash_init(&ctx, key);
ghash_update(&ctx, data, len);
ghash_final(&ctx, out);
}
/**
* polyval() - Compute a POLYVAL value
* @key: The prepared key
@ -187,4 +273,4 @@ static inline void polyval(const struct polyval_key *key,
polyval_final(&ctx, out);
}
#endif /* _CRYPTO_POLYVAL_H */
#endif /* _CRYPTO_GF128HASH_H */

17
include/crypto/gf128mul.h
View File

@ -215,25 +215,14 @@ static inline void gf128mul_x_ble(le128 *r, const le128 *x)
r->b = cpu_to_le64((b << 1) ^ _tt);
}
/* 4k table optimization */
struct gf128mul_4k {
be128 t[256];
};
struct gf128mul_4k *gf128mul_init_4k_lle(const be128 *g);
void gf128mul_4k_lle(be128 *a, const struct gf128mul_4k *t);
void gf128mul_x8_ble(le128 *r, const le128 *x);
static inline void gf128mul_free_4k(struct gf128mul_4k *t)
{
kfree_sensitive(t);
}
/* 64k table optimization, implemented for bbe */
struct gf128mul_64k {
struct gf128mul_4k *t[16];
struct {
be128 t[256];
} *t[16];
};
/* First initialize with the constant factor with which you

12
include/crypto/ghash.h
View File

@ -6,19 +6,7 @@
#ifndef __CRYPTO_GHASH_H__
#define __CRYPTO_GHASH_H__
#include <linux/types.h>
#define GHASH_BLOCK_SIZE 16
#define GHASH_DIGEST_SIZE 16
struct gf128mul_4k;
struct ghash_ctx {
struct gf128mul_4k *gf128;
};
struct ghash_desc_ctx {
u8 buffer[GHASH_BLOCK_SIZE];
};
#endif

52
include/crypto/internal/blockhash.h
View File

@ -1,52 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Handle partial blocks for block hash.
*
* Copyright (c) 2015 Linaro Ltd <ard.biesheuvel@linaro.org>
* Copyright (c) 2025 Herbert Xu <herbert@gondor.apana.org.au>
*/
#ifndef _CRYPTO_INTERNAL_BLOCKHASH_H
#define _CRYPTO_INTERNAL_BLOCKHASH_H
#include <linux/string.h>
#include <linux/types.h>
#define BLOCK_HASH_UPDATE_BASE(block_fn, state, src, nbytes, bs, dv, \
buf, buflen) \
({ \
typeof(block_fn) *_block_fn = &(block_fn); \
typeof(state + 0) _state = (state); \
unsigned int _buflen = (buflen); \
size_t _nbytes = (nbytes); \
unsigned int _bs = (bs); \
const u8 *_src = (src); \
u8 *_buf = (buf); \
while ((_buflen + _nbytes) >= _bs) { \
const u8 *data = _src; \
size_t len = _nbytes; \
size_t blocks; \
int remain; \
if (_buflen) { \
remain = _bs - _buflen; \
memcpy(_buf + _buflen, _src, remain); \
data = _buf; \
len = _bs; \
} \
remain = len % bs; \
blocks = (len - remain) / (dv); \
(*_block_fn)(_state, data, blocks); \
_src += len - remain - _buflen; \
_nbytes -= len - remain - _buflen; \
_buflen = 0; \
} \
memcpy(_buf + _buflen, _src, _nbytes); \
_buflen += _nbytes; \
})
#define BLOCK_HASH_UPDATE(block, state, src, nbytes, bs, buf, buflen) \
BLOCK_HASH_UPDATE_BASE(block, state, src, nbytes, bs, 1, buf, buflen)
#define BLOCK_HASH_UPDATE_BLOCKS(block, state, src, nbytes, bs, buf, buflen) \
BLOCK_HASH_UPDATE_BASE(block, state, src, nbytes, bs, bs, buf, buflen)
#endif /* _CRYPTO_INTERNAL_BLOCKHASH_H */

89
include/crypto/sm3.h
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Common values for SM3 algorithm
* SM3 hash algorithm
*
* Copyright (C) 2017 ARM Limited or its affiliates.
* Copyright (C) 2017 Gilad Ben-Yossef <gilad@benyossef.com>
@ -14,10 +14,6 @@
#define SM3_DIGEST_SIZE 32
#define SM3_BLOCK_SIZE 64
#define SM3_STATE_SIZE 40
#define SM3_T1 0x79CC4519
#define SM3_T2 0x7A879D8A
#define SM3_IVA 0x7380166f
#define SM3_IVB 0x4914b2b9
@ -28,37 +24,64 @@
#define SM3_IVG 0xe38dee4d
#define SM3_IVH 0xb0fb0e4e
extern const u8 sm3_zero_message_hash[SM3_DIGEST_SIZE];
struct sm3_state {
u32 state[SM3_DIGEST_SIZE / 4];
u64 count;
u8 buffer[SM3_BLOCK_SIZE];
/* State for the SM3 compression function */
struct sm3_block_state {
u32 h[SM3_DIGEST_SIZE / 4];
};
/*
* Stand-alone implementation of the SM3 algorithm. It is designed to
* have as little dependencies as possible so it can be used in the
* kexec_file purgatory. In other cases you should generally use the
* hash APIs from include/crypto/hash.h. Especially when hashing large
* amounts of data as those APIs may be hw-accelerated.
*
* For details see lib/crypto/sm3.c
/**
* struct sm3_ctx - Context for hashing a message with SM3
* @state: the compression function state
* @bytecount: number of bytes processed so far
* @buf: partial block buffer; bytecount % SM3_BLOCK_SIZE bytes are valid
*/
struct sm3_ctx {
struct sm3_block_state state;
u64 bytecount;
u8 buf[SM3_BLOCK_SIZE] __aligned(__alignof__(__be64));
};
static inline void sm3_init(struct sm3_state *sctx)
{
sctx->state[0] = SM3_IVA;
sctx->state[1] = SM3_IVB;
sctx->state[2] = SM3_IVC;
sctx->state[3] = SM3_IVD;
sctx->state[4] = SM3_IVE;
sctx->state[5] = SM3_IVF;
sctx->state[6] = SM3_IVG;
sctx->state[7] = SM3_IVH;
sctx->count = 0;
}
/**
* sm3_init() - Initialize an SM3 context for a new message
* @ctx: the context to initialize
*
* If you don't need incremental computation, consider sm3() instead.
*
* Context: Any context.
*/
void sm3_init(struct sm3_ctx *ctx);
void sm3_block_generic(struct sm3_state *sctx, u8 const *data, int blocks);
/**
* sm3_update() - Update an SM3 context with message data
* @ctx: the context to update; must have been initialized
* @data: the message data
* @len: the data length in bytes
*
* This can be called any number of times.
*
* Context: Any context.
*/
void sm3_update(struct sm3_ctx *ctx, const u8 *data, size_t len);
#endif
/**
* sm3_final() - Finish computing an SM3 message digest
* @ctx: the context to finalize; must have been initialized
* @out: (output) the resulting SM3 message digest
*
* After finishing, this zeroizes @ctx. So the caller does not need to do it.
*
* Context: Any context.
*/
void sm3_final(struct sm3_ctx *ctx, u8 out[at_least SM3_DIGEST_SIZE]);
/**
* sm3() - Compute SM3 message digest in one shot
* @data: the message data
* @len: the data length in bytes
* @out: (output) the resulting SM3 message digest
*
* Context: Any context.
*/
void sm3(const u8 *data, size_t len, u8 out[at_least SM3_DIGEST_SIZE]);
#endif /* _CRYPTO_SM3_H */

82
include/crypto/sm3_base.h
View File

@ -1,82 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* sm3_base.h - core logic for SM3 implementations
*
* Copyright (C) 2017 ARM Limited or its affiliates.
* Written by Gilad Ben-Yossef <gilad@benyossef.com>
*/
#ifndef _CRYPTO_SM3_BASE_H
#define _CRYPTO_SM3_BASE_H
#include <crypto/internal/hash.h>
#include <crypto/sm3.h>
#include <linux/math.h>
#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/unaligned.h>
typedef void (sm3_block_fn)(struct sm3_state *sst, u8 const *src, int blocks);
static inline int sm3_base_init(struct shash_desc *desc)
{
sm3_init(shash_desc_ctx(desc));
return 0;
}
static inline int sm3_base_do_update_blocks(struct shash_desc *desc,
const u8 *data, unsigned int len,
sm3_block_fn *block_fn)
{
unsigned int remain = len - round_down(len, SM3_BLOCK_SIZE);
struct sm3_state *sctx = shash_desc_ctx(desc);
sctx->count += len - remain;
block_fn(sctx, data, len / SM3_BLOCK_SIZE);
return remain;
}
static inline int sm3_base_do_finup(struct shash_desc *desc,
const u8 *src, unsigned int len,
sm3_block_fn *block_fn)
{
unsigned int bit_offset = SM3_BLOCK_SIZE / 8 - 1;
struct sm3_state *sctx = shash_desc_ctx(desc);
union {
__be64 b64[SM3_BLOCK_SIZE / 4];
u8 u8[SM3_BLOCK_SIZE * 2];
} block = {};
if (len >= SM3_BLOCK_SIZE) {
int remain;
remain = sm3_base_do_update_blocks(desc, src, len, block_fn);
src += len - remain;
len = remain;
}
if (len >= bit_offset * 8)
bit_offset += SM3_BLOCK_SIZE / 8;
memcpy(&block, src, len);
block.u8[len] = 0x80;
sctx->count += len;
block.b64[bit_offset] = cpu_to_be64(sctx->count << 3);
block_fn(sctx, block.u8, (bit_offset + 1) * 8 / SM3_BLOCK_SIZE);
memzero_explicit(&block, sizeof(block));
return 0;
}
static inline int sm3_base_finish(struct shash_desc *desc, u8 *out)
{
struct sm3_state *sctx = shash_desc_ctx(desc);
__be32 *digest = (__be32 *)out;
int i;
for (i = 0; i < SM3_DIGEST_SIZE / sizeof(__be32); i++)
put_unaligned_be32(sctx->state[i], digest++);
return 0;
}
#endif /* _CRYPTO_SM3_BASE_H */

2
lib/Kconfig.debug
View File

@ -3058,6 +3058,8 @@ config HW_BREAKPOINT_KUNIT_TEST
If unsure, say N.
source "lib/crypto/tests/Kconfig"
config SIPHASH_KUNIT_TEST
tristate "Perform selftest on siphash functions" if !KUNIT_ALL_TESTS
depends on KUNIT

24
lib/crypto/.kunitconfig
View File

@ -1,28 +1,13 @@
CONFIG_KUNIT=y
# These kconfig options select all the CONFIG_CRYPTO_LIB_* symbols that have a
# corresponding KUnit test. Those symbols cannot be directly enabled here,
# since they are hidden symbols.
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ADIANTUM=y
CONFIG_CRYPTO_BLAKE2B=y
CONFIG_CRYPTO_CHACHA20POLY1305=y
CONFIG_CRYPTO_HCTR2=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MLDSA=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=y
CONFIG_INET=y
CONFIG_IPV6=y
CONFIG_NET=y
CONFIG_NETDEVICES=y
CONFIG_WIREGUARD=y
CONFIG_CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT=y
CONFIG_CRYPTO_LIB_AES_CBC_MACS_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_BLAKE2B_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_CURVE25519_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_GHASH_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_MD5_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_MLDSA_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_NH_KUNIT_TEST=y
@ -32,3 +17,4 @@ CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA512_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA3_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SM3_KUNIT_TEST=y

68
lib/crypto/Kconfig
View File

@ -1,7 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
menu "Crypto library routines"
config CRYPTO_HASH_INFO
bool
@ -10,6 +8,8 @@ config CRYPTO_LIB_UTILS
config CRYPTO_LIB_AES
tristate
# Select dependencies of modes that are part of libaes.
select CRYPTO_LIB_UTILS if CRYPTO_LIB_AES_CBC_MACS
config CRYPTO_LIB_AES_ARCH
bool
@ -28,10 +28,18 @@ config CRYPTO_LIB_AESCFB
select CRYPTO_LIB_AES
select CRYPTO_LIB_UTILS
config CRYPTO_LIB_AES_CBC_MACS
tristate
select CRYPTO_LIB_AES
help
The AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC library functions. Select
this if your module uses any of the functions from
<crypto/aes-cbc-macs.h>.
config CRYPTO_LIB_AESGCM
tristate
select CRYPTO_LIB_AES
select CRYPTO_LIB_GF128MUL
select CRYPTO_LIB_GF128HASH
select CRYPTO_LIB_UTILS
config CRYPTO_LIB_ARC4
@ -70,7 +78,7 @@ config CRYPTO_LIB_CHACHA_ARCH
bool
depends on CRYPTO_LIB_CHACHA && !UML && !KMSAN
default y if ARM
default y if ARM64 && KERNEL_MODE_NEON
default y if ARM64
default y if MIPS && CPU_MIPS32_R2
default y if PPC64 && CPU_LITTLE_ENDIAN && VSX
default y if RISCV && 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
@ -100,6 +108,23 @@ config CRYPTO_LIB_CURVE25519_GENERIC
config CRYPTO_LIB_DES
tristate
config CRYPTO_LIB_GF128HASH
tristate
help
The GHASH and POLYVAL library functions. Select this if your module
uses any of the functions from <crypto/gf128hash.h>.
config CRYPTO_LIB_GF128HASH_ARCH
bool
depends on CRYPTO_LIB_GF128HASH && !UML
default y if ARM && KERNEL_MODE_NEON
default y if ARM64
default y if PPC64 && VSX
default y if RISCV && 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
default y if S390
default y if X86_64
config CRYPTO_LIB_MD5
tristate
help
@ -109,9 +134,7 @@ config CRYPTO_LIB_MD5
config CRYPTO_LIB_MD5_ARCH
bool
depends on CRYPTO_LIB_MD5 && !UML
default y if MIPS && CPU_CAVIUM_OCTEON
default y if PPC
default y if SPARC64
config CRYPTO_LIB_MLDSA
tristate
@ -130,7 +153,7 @@ config CRYPTO_LIB_NH_ARCH
bool
depends on CRYPTO_LIB_NH && !UML && !KMSAN
default y if ARM && KERNEL_MODE_NEON
default y if ARM64 && KERNEL_MODE_NEON
default y if ARM64
default y if X86_64
config CRYPTO_LIB_POLY1305
@ -143,7 +166,7 @@ config CRYPTO_LIB_POLY1305_ARCH
bool
depends on CRYPTO_LIB_POLY1305 && !UML && !KMSAN
default y if ARM
default y if ARM64 && KERNEL_MODE_NEON
default y if ARM64
default y if MIPS
# The PPC64 code needs to be fixed to work in softirq context.
default y if PPC64 && CPU_LITTLE_ENDIAN && VSX && BROKEN
@ -168,18 +191,6 @@ config CRYPTO_LIB_POLY1305_RSIZE
default 9 if ARM || ARM64
default 1
config CRYPTO_LIB_POLYVAL
tristate
help
The POLYVAL library functions. Select this if your module uses any of
the functions from <crypto/polyval.h>.
config CRYPTO_LIB_POLYVAL_ARCH
bool
depends on CRYPTO_LIB_POLYVAL && !UML
default y if ARM64 && KERNEL_MODE_NEON
default y if X86_64
config CRYPTO_LIB_CHACHA20POLY1305
tristate
select CRYPTO_LIB_CHACHA
@ -196,7 +207,7 @@ config CRYPTO_LIB_SHA1_ARCH
bool
depends on CRYPTO_LIB_SHA1 && !UML
default y if ARM
default y if ARM64 && KERNEL_MODE_NEON
default y if ARM64
default y if MIPS && CPU_CAVIUM_OCTEON
default y if PPC
default y if S390
@ -252,12 +263,19 @@ config CRYPTO_LIB_SHA3
config CRYPTO_LIB_SHA3_ARCH
bool
depends on CRYPTO_LIB_SHA3 && !UML
default y if ARM64 && KERNEL_MODE_NEON
default y if ARM64
default y if S390
config CRYPTO_LIB_SM3
tristate
help
The SM3 library functions. Select this if your module uses any of the
functions from <crypto/sm3.h>.
source "lib/crypto/tests/Kconfig"
endmenu
config CRYPTO_LIB_SM3_ARCH
bool
depends on CRYPTO_LIB_SM3 && !UML
default y if ARM64
default y if RISCV && 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
default y if X86_64

79
lib/crypto/Makefile
View File

@ -8,6 +8,10 @@ quiet_cmd_perlasm = PERLASM $@
quiet_cmd_perlasm_with_args = PERLASM $@
cmd_perlasm_with_args = $(PERL) $(<) void $(@)
ppc64-perlasm-flavour-y := linux-ppc64
ppc64-perlasm-flavour-$(CONFIG_PPC64_ELF_ABI_V2) := linux-ppc64-elfv2
ppc64-perlasm-flavour-$(CONFIG_CPU_LITTLE_ENDIAN) := linux-ppc64le
obj-$(CONFIG_KUNIT) += tests/
obj-$(CONFIG_CRYPTO_HASH_INFO) += hash_info.o
@ -23,11 +27,10 @@ ifeq ($(CONFIG_CRYPTO_LIB_AES_ARCH),y)
CFLAGS_aes.o += -I$(src)/$(SRCARCH)
libaes-$(CONFIG_ARM) += arm/aes-cipher-core.o
ifeq ($(CONFIG_ARM64),y)
libaes-y += arm64/aes-cipher-core.o
libaes-$(CONFIG_KERNEL_MODE_NEON) += arm64/aes-ce-core.o
endif
libaes-$(CONFIG_ARM64) += arm64/aes-cipher-core.o \
arm64/aes-ce-core.o \
arm64/aes-ce.o \
arm64/aes-neon.o
ifeq ($(CONFIG_PPC),y)
ifeq ($(CONFIG_SPE),y)
@ -37,11 +40,8 @@ libaes-y += powerpc/aes-spe-core.o \
powerpc/aes-tab-4k.o
else
libaes-y += powerpc/aesp8-ppc.o
aes-perlasm-flavour-y := linux-ppc64
aes-perlasm-flavour-$(CONFIG_PPC64_ELF_ABI_V2) := linux-ppc64-elfv2
aes-perlasm-flavour-$(CONFIG_CPU_LITTLE_ENDIAN) := linux-ppc64le
quiet_cmd_perlasm_aes = PERLASM $@
cmd_perlasm_aes = $(PERL) $< $(aes-perlasm-flavour-y) $@
cmd_perlasm_aes = $(PERL) $< $(ppc64-perlasm-flavour-y) $@
# Use if_changed instead of cmd, in case the flavour changed.
$(obj)/powerpc/aesp8-ppc.S: $(src)/powerpc/aesp8-ppc.pl FORCE
$(call if_changed,perlasm_aes)
@ -125,7 +125,6 @@ endif # CONFIG_CRYPTO_LIB_CHACHA_ARCH
obj-$(CONFIG_CRYPTO_LIB_CHACHA20POLY1305) += libchacha20poly1305.o
libchacha20poly1305-y += chacha20poly1305.o
libchacha20poly1305-$(CONFIG_CRYPTO_SELFTESTS) += chacha20poly1305-selftest.o
################################################################################
@ -158,12 +157,39 @@ libdes-y := des.o
################################################################################
obj-$(CONFIG_CRYPTO_LIB_GF128HASH) += libgf128hash.o
libgf128hash-y := gf128hash.o
ifeq ($(CONFIG_CRYPTO_LIB_GF128HASH_ARCH),y)
CFLAGS_gf128hash.o += -I$(src)/$(SRCARCH)
libgf128hash-$(CONFIG_ARM) += arm/ghash-neon-core.o
libgf128hash-$(CONFIG_ARM64) += arm64/ghash-neon-core.o \
arm64/polyval-ce-core.o
ifeq ($(CONFIG_PPC),y)
libgf128hash-y += powerpc/ghashp8-ppc.o
quiet_cmd_perlasm_ghash = PERLASM $@
cmd_perlasm_ghash = $(PERL) $< $(ppc64-perlasm-flavour-y) $@
$(obj)/powerpc/ghashp8-ppc.S: $(src)/powerpc/ghashp8-ppc.pl FORCE
$(call if_changed,perlasm_ghash)
targets += powerpc/ghashp8-ppc.S
OBJECT_FILES_NON_STANDARD_powerpc/ghashp8-ppc.o := y
endif
libgf128hash-$(CONFIG_RISCV) += riscv/ghash-riscv64-zvkg.o
libgf128hash-$(CONFIG_X86) += x86/ghash-pclmul.o \
x86/polyval-pclmul-avx.o
endif # CONFIG_CRYPTO_LIB_GF128HASH_ARCH
# clean-files must be defined unconditionally
clean-files += powerpc/ghashp8-ppc.S
################################################################################
obj-$(CONFIG_CRYPTO_LIB_MD5) += libmd5.o
libmd5-y := md5.o
ifeq ($(CONFIG_CRYPTO_LIB_MD5_ARCH),y)
CFLAGS_md5.o += -I$(src)/$(SRCARCH)
libmd5-$(CONFIG_PPC) += powerpc/md5-asm.o
libmd5-$(CONFIG_SPARC) += sparc/md5_asm.o
endif # CONFIG_CRYPTO_LIB_MD5_ARCH
################################################################################
@ -255,16 +281,6 @@ clean-files += arm/poly1305-core.S \
################################################################################
obj-$(CONFIG_CRYPTO_LIB_POLYVAL) += libpolyval.o
libpolyval-y := polyval.o
ifeq ($(CONFIG_CRYPTO_LIB_POLYVAL_ARCH),y)
CFLAGS_polyval.o += -I$(src)/$(SRCARCH)
libpolyval-$(CONFIG_ARM64) += arm64/polyval-ce-core.o
libpolyval-$(CONFIG_X86) += x86/polyval-pclmul-avx.o
endif
################################################################################
obj-$(CONFIG_CRYPTO_LIB_SHA1) += libsha1.o
libsha1-y := sha1.o
ifeq ($(CONFIG_CRYPTO_LIB_SHA1_ARCH),y)
@ -300,10 +316,9 @@ AFLAGS_arm/sha256-core.o += $(aflags-thumb2-y)
endif
ifeq ($(CONFIG_ARM64),y)
libsha256-y += arm64/sha256-core.o
libsha256-y += arm64/sha256-ce.o arm64/sha256-core.o
$(obj)/arm64/sha256-core.S: $(src)/arm64/sha2-armv8.pl
$(call cmd,perlasm_with_args)
libsha256-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha256-ce.o
endif
libsha256-$(CONFIG_PPC) += powerpc/sha256-spe-asm.o
@ -330,10 +345,9 @@ AFLAGS_arm/sha512-core.o += $(aflags-thumb2-y)
endif
ifeq ($(CONFIG_ARM64),y)
libsha512-y += arm64/sha512-core.o
libsha512-y += arm64/sha512-ce-core.o arm64/sha512-core.o
$(obj)/arm64/sha512-core.S: $(src)/arm64/sha2-armv8.pl
$(call cmd,perlasm_with_args)
libsha512-$(CONFIG_KERNEL_MODE_NEON) += arm64/sha512-ce-core.o
endif
libsha512-$(CONFIG_RISCV) += riscv/sha512-riscv64-zvknhb-zvkb.o
@ -355,13 +369,22 @@ endif # CONFIG_CRYPTO_LIB_SHA3_ARCH
################################################################################
obj-$(CONFIG_CRYPTO_LIB_SM3) += libsm3.o
libsm3-y := sm3.o
ifeq ($(CONFIG_CRYPTO_LIB_SM3_ARCH),y)
CFLAGS_sm3.o += -I$(src)/$(SRCARCH)
libsm3-$(CONFIG_ARM64) += arm64/sm3-ce-core.o \
arm64/sm3-neon-core.o
libsm3-$(CONFIG_RISCV) += riscv/sm3-riscv64-zvksh-zvkb.o
libsm3-$(CONFIG_X86) += x86/sm3-avx-asm_64.o
endif # CONFIG_CRYPTO_LIB_SM3_ARCH
################################################################################
obj-$(CONFIG_MPILIB) += mpi/
obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
obj-$(CONFIG_CRYPTO_LIB_SM3) += libsm3.o
libsm3-y := sm3.o
# clean-files must be defined unconditionally
clean-files += arm/sha256-core.S arm/sha512-core.S
clean-files += arm64/sha256-core.S arm64/sha512-core.S

231
lib/crypto/aes.c
View File

@ -4,12 +4,15 @@
* Copyright 2026 Google LLC
*/
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes.h>
#include <crypto/utils.h>
#include <linux/cache.h>
#include <linux/crypto.h>
#include <linux/export.h>
#include <linux/module.h>
#include <linux/unaligned.h>
#include "fips.h"
static const u8 ____cacheline_aligned aes_sbox[] = {
0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5,
@ -512,10 +515,235 @@ void aes_decrypt(const struct aes_key *key, u8 out[AES_BLOCK_SIZE],
}
EXPORT_SYMBOL(aes_decrypt);
#ifdef aes_mod_init_arch
#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC_MACS)
#ifndef aes_cbcmac_blocks_arch
static bool aes_cbcmac_blocks_arch(u8 h[AES_BLOCK_SIZE],
const struct aes_enckey *key, const u8 *data,
size_t nblocks, bool enc_before,
bool enc_after)
{
return false;
}
#endif
/* This assumes nblocks >= 1. */
static void aes_cbcmac_blocks(u8 h[AES_BLOCK_SIZE],
const struct aes_enckey *key, const u8 *data,
size_t nblocks, bool enc_before, bool enc_after)
{
if (aes_cbcmac_blocks_arch(h, key, data, nblocks, enc_before,
enc_after))
return;
if (enc_before)
aes_encrypt(key, h, h);
for (; nblocks > 1; nblocks--) {
crypto_xor(h, data, AES_BLOCK_SIZE);
data += AES_BLOCK_SIZE;
aes_encrypt(key, h, h);
}
crypto_xor(h, data, AES_BLOCK_SIZE);
if (enc_after)
aes_encrypt(key, h, h);
}
int aes_cmac_preparekey(struct aes_cmac_key *key, const u8 *in_key,
size_t key_len)
{
u64 hi, lo, mask;
int err;
/* Prepare the AES key. */
err = aes_prepareenckey(&key->aes, in_key, key_len);
if (err)
return err;
/*
* Prepare the subkeys K1 and K2 by encrypting the all-zeroes block,
* then multiplying by 'x' and 'x^2' (respectively) in GF(2^128).
* Reference: NIST SP 800-38B, Section 6.1 "Subkey Generation".
*/
memset(key->k_final[0].b, 0, AES_BLOCK_SIZE);
aes_encrypt(&key->aes, key->k_final[0].b, key->k_final[0].b);
hi = be64_to_cpu(key->k_final[0].w[0]);
lo = be64_to_cpu(key->k_final[0].w[1]);
for (int i = 0; i < 2; i++) {
mask = ((s64)hi >> 63) & 0x87;
hi = (hi << 1) ^ (lo >> 63);
lo = (lo << 1) ^ mask;
key->k_final[i].w[0] = cpu_to_be64(hi);
key->k_final[i].w[1] = cpu_to_be64(lo);
}
return 0;
}
EXPORT_SYMBOL_GPL(aes_cmac_preparekey);
void aes_xcbcmac_preparekey(struct aes_cmac_key *key,
const u8 in_key[AES_KEYSIZE_128])
{
static const u8 constants[3][AES_BLOCK_SIZE] = {
{ [0 ... AES_BLOCK_SIZE - 1] = 0x1 },
{ [0 ... AES_BLOCK_SIZE - 1] = 0x2 },
{ [0 ... AES_BLOCK_SIZE - 1] = 0x3 },
};
u8 new_aes_key[AES_BLOCK_SIZE];
static_assert(AES_BLOCK_SIZE == AES_KEYSIZE_128);
aes_prepareenckey(&key->aes, in_key, AES_BLOCK_SIZE);
aes_encrypt(&key->aes, new_aes_key, constants[0]);
aes_encrypt(&key->aes, key->k_final[0].b, constants[1]);
aes_encrypt(&key->aes, key->k_final[1].b, constants[2]);
aes_prepareenckey(&key->aes, new_aes_key, AES_BLOCK_SIZE);
memzero_explicit(new_aes_key, AES_BLOCK_SIZE);
}
EXPORT_SYMBOL_GPL(aes_xcbcmac_preparekey);
void aes_cmac_update(struct aes_cmac_ctx *ctx, const u8 *data, size_t data_len)
{
bool enc_before = false;
size_t nblocks;
if (ctx->partial_len) {
/* XOR data into a pending block. */
size_t l = min(data_len, AES_BLOCK_SIZE - ctx->partial_len);
crypto_xor(&ctx->h[ctx->partial_len], data, l);
data += l;
data_len -= l;
ctx->partial_len += l;
if (data_len == 0) {
/*
* Either the pending block hasn't been filled yet, or
* no more data was given so it's not yet known whether
* the block is the final block.
*/
return;
}
/* Pending block has been filled and isn't the final block. */
enc_before = true;
}
nblocks = data_len / AES_BLOCK_SIZE;
data_len %= AES_BLOCK_SIZE;
if (nblocks == 0) {
/* 0 additional full blocks, then optionally a partial block */
if (enc_before)
aes_encrypt(&ctx->key->aes, ctx->h, ctx->h);
crypto_xor(ctx->h, data, data_len);
ctx->partial_len = data_len;
} else if (data_len != 0) {
/* 1 or more additional full blocks, then a partial block */
aes_cbcmac_blocks(ctx->h, &ctx->key->aes, data, nblocks,
enc_before, /* enc_after= */ true);
data += nblocks * AES_BLOCK_SIZE;
crypto_xor(ctx->h, data, data_len);
ctx->partial_len = data_len;
} else {
/*
* 1 or more additional full blocks only. Encryption of the
* last block is delayed until it's known whether it's the final
* block in the message or not.
*/
aes_cbcmac_blocks(ctx->h, &ctx->key->aes, data, nblocks,
enc_before, /* enc_after= */ false);
ctx->partial_len = AES_BLOCK_SIZE;
}
}
EXPORT_SYMBOL_GPL(aes_cmac_update);
void aes_cmac_final(struct aes_cmac_ctx *ctx, u8 out[AES_BLOCK_SIZE])
{
if (ctx->partial_len == AES_BLOCK_SIZE) {
/* Final block is a full block. Use k_final[0]. */
crypto_xor(ctx->h, ctx->key->k_final[0].b, AES_BLOCK_SIZE);
} else {
/* Final block is a partial block. Pad, and use k_final[1]. */
ctx->h[ctx->partial_len] ^= 0x80;
crypto_xor(ctx->h, ctx->key->k_final[1].b, AES_BLOCK_SIZE);
}
aes_encrypt(&ctx->key->aes, out, ctx->h);
memzero_explicit(ctx, sizeof(*ctx));
}
EXPORT_SYMBOL_GPL(aes_cmac_final);
void aes_cbcmac_update(struct aes_cbcmac_ctx *ctx, const u8 *data,
size_t data_len)
{
bool enc_before = false;
size_t nblocks;
if (ctx->partial_len) {
size_t l = min(data_len, AES_BLOCK_SIZE - ctx->partial_len);
crypto_xor(&ctx->h[ctx->partial_len], data, l);
data += l;
data_len -= l;
ctx->partial_len += l;
if (ctx->partial_len < AES_BLOCK_SIZE)
return;
enc_before = true;
}
nblocks = data_len / AES_BLOCK_SIZE;
data_len %= AES_BLOCK_SIZE;
if (nblocks == 0) {
if (enc_before)
aes_encrypt(ctx->key, ctx->h, ctx->h);
} else {
aes_cbcmac_blocks(ctx->h, ctx->key, data, nblocks, enc_before,
/* enc_after= */ true);
data += nblocks * AES_BLOCK_SIZE;
}
crypto_xor(ctx->h, data, data_len);
ctx->partial_len = data_len;
}
EXPORT_SYMBOL_NS_GPL(aes_cbcmac_update, "CRYPTO_INTERNAL");
void aes_cbcmac_final(struct aes_cbcmac_ctx *ctx, u8 out[AES_BLOCK_SIZE])
{
if (ctx->partial_len)
aes_encrypt(ctx->key, out, ctx->h);
else
memcpy(out, ctx->h, AES_BLOCK_SIZE);
memzero_explicit(ctx, sizeof(*ctx));
}
EXPORT_SYMBOL_NS_GPL(aes_cbcmac_final, "CRYPTO_INTERNAL");
/*
* FIPS cryptographic algorithm self-test for AES-CMAC. As per the FIPS 140-3
* Implementation Guidance, a cryptographic algorithm self-test for at least one
* of AES-GCM, AES-CCM, AES-CMAC, or AES-GMAC is required if any of those modes
* is implemented. This fulfills that requirement via AES-CMAC.
*
* This is just for FIPS. The full tests are in the KUnit test suite.
*/
static void __init aes_cmac_fips_test(void)
{
struct aes_cmac_key key;
u8 mac[AES_BLOCK_SIZE];
if (aes_cmac_preparekey(&key, fips_test_key, sizeof(fips_test_key)) !=
0)
panic("aes: CMAC FIPS self-test failed (preparekey)\n");
aes_cmac(&key, fips_test_data, sizeof(fips_test_data), mac);
if (memcmp(fips_test_aes_cmac_value, mac, sizeof(mac)) != 0)
panic("aes: CMAC FIPS self-test failed (wrong MAC)\n");
memzero_explicit(&key, sizeof(key));
}
#else /* CONFIG_CRYPTO_LIB_AES_CBC_MACS */
static inline void aes_cmac_fips_test(void)
{
}
#endif /* !CONFIG_CRYPTO_LIB_AES_CBC_MACS */
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
aes_mod_init_arch();
#endif
if (fips_enabled)
aes_cmac_fips_test();
return 0;
}
subsys_initcall(aes_mod_init);
@ -524,7 +752,6 @@ static void __exit aes_mod_exit(void)
{
}
module_exit(aes_mod_exit);
#endif
MODULE_DESCRIPTION("AES block cipher");
MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");

27
lib/crypto/aescfb.c
View File

@ -6,28 +6,9 @@
*/
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/utils.h>
#include <linux/export.h>
#include <linux/module.h>
#include <asm/irqflags.h>
static void aescfb_encrypt_block(const struct aes_enckey *key, void *dst,
const void *src)
{
unsigned long flags;
/*
* In AES-CFB, the AES encryption operates on known 'plaintext' (the IV
* and ciphertext), making it susceptible to timing attacks on the
* encryption key. The AES library already mitigates this risk to some
* extent by pulling the entire S-box into the caches before doing any
* substitutions, but this strategy is more effective when running with
* interrupts disabled.
*/
local_irq_save(flags);
aes_encrypt(key, dst, src);
local_irq_restore(flags);
}
/**
* aescfb_encrypt - Perform AES-CFB encryption on a block of data
@ -45,7 +26,7 @@ void aescfb_encrypt(const struct aes_enckey *key, u8 *dst, const u8 *src,
const u8 *v = iv;
while (len > 0) {
aescfb_encrypt_block(key, ks, v);
aes_encrypt(key, ks, v);
crypto_xor_cpy(dst, src, ks, min(len, AES_BLOCK_SIZE));
v = dst;
@ -72,7 +53,7 @@ void aescfb_decrypt(const struct aes_enckey *key, u8 *dst, const u8 *src,
{
u8 ks[2][AES_BLOCK_SIZE];
aescfb_encrypt_block(key, ks[0], iv);
aes_encrypt(key, ks[0], iv);
for (int i = 0; len > 0; i ^= 1) {
if (len > AES_BLOCK_SIZE)
@ -81,7 +62,7 @@ void aescfb_decrypt(const struct aes_enckey *key, u8 *dst, const u8 *src,
* performing the XOR, as that may update in place and
* overwrite the ciphertext.
*/
aescfb_encrypt_block(key, ks[!i], src);
aes_encrypt(key, ks[!i], src);
crypto_xor_cpy(dst, src, ks[i], min(len, AES_BLOCK_SIZE));

76
lib/crypto/aesgcm.c
View File

@ -5,30 +5,10 @@
* Copyright 2022 Google LLC
*/
#include <crypto/algapi.h>
#include <crypto/gcm.h>
#include <crypto/ghash.h>
#include <crypto/utils.h>
#include <linux/export.h>
#include <linux/module.h>
#include <asm/irqflags.h>
static void aesgcm_encrypt_block(const struct aes_enckey *key, void *dst,
const void *src)
{
unsigned long flags;
/*
* In AES-GCM, both the GHASH key derivation and the CTR mode
* encryption operate on known plaintext, making them susceptible to
* timing attacks on the encryption key. The AES library already
* mitigates this risk to some extent by pulling the entire S-box into
* the caches before doing any substitutions, but this strategy is more
* effective when running with interrupts disabled.
*/
local_irq_save(flags);
aes_encrypt(key, dst, src);
local_irq_restore(flags);
}
/**
* aesgcm_expandkey - Expands the AES and GHASH keys for the AES-GCM key
@ -45,7 +25,7 @@ static void aesgcm_encrypt_block(const struct aes_enckey *key, void *dst,
int aesgcm_expandkey(struct aesgcm_ctx *ctx, const u8 *key,
unsigned int keysize, unsigned int authsize)
{
u8 kin[AES_BLOCK_SIZE] = {};
u8 h[AES_BLOCK_SIZE] = {};
int ret;
ret = crypto_gcm_check_authsize(authsize) ?:
@ -54,24 +34,13 @@ int aesgcm_expandkey(struct aesgcm_ctx *ctx, const u8 *key,
return ret;
ctx->authsize = authsize;
aesgcm_encrypt_block(&ctx->aes_key, &ctx->ghash_key, kin);
aes_encrypt(&ctx->aes_key, h, h);
ghash_preparekey(&ctx->ghash_key, h);
memzero_explicit(h, sizeof(h));
return 0;
}
EXPORT_SYMBOL(aesgcm_expandkey);
static void aesgcm_ghash(be128 *ghash, const be128 *key, const void *src,
int len)
{
while (len > 0) {
crypto_xor((u8 *)ghash, src, min(len, GHASH_BLOCK_SIZE));
gf128mul_lle(ghash, key);
src += GHASH_BLOCK_SIZE;
len -= GHASH_BLOCK_SIZE;
}
}
/**
* aesgcm_mac - Generates the authentication tag using AES-GCM algorithm.
* @ctx: The data structure that will hold the AES-GCM key schedule
@ -88,20 +57,33 @@ static void aesgcm_ghash(be128 *ghash, const be128 *key, const void *src,
static void aesgcm_mac(const struct aesgcm_ctx *ctx, const u8 *src, int src_len,
const u8 *assoc, int assoc_len, __be32 *ctr, u8 *authtag)
{
be128 tail = { cpu_to_be64(assoc_len * 8), cpu_to_be64(src_len * 8) };
u8 buf[AES_BLOCK_SIZE];
be128 ghash = {};
static const u8 zeroes[GHASH_BLOCK_SIZE];
__be64 tail[2] = {
cpu_to_be64((u64)assoc_len * 8),
cpu_to_be64((u64)src_len * 8),
};
struct ghash_ctx ghash;
u8 ghash_out[AES_BLOCK_SIZE];
u8 enc_ctr[AES_BLOCK_SIZE];
aesgcm_ghash(&ghash, &ctx->ghash_key, assoc, assoc_len);
aesgcm_ghash(&ghash, &ctx->ghash_key, src, src_len);
aesgcm_ghash(&ghash, &ctx->ghash_key, &tail, sizeof(tail));
ghash_init(&ghash, &ctx->ghash_key);
ghash_update(&ghash, assoc, assoc_len);
ghash_update(&ghash, zeroes, -assoc_len & (GHASH_BLOCK_SIZE - 1));
ghash_update(&ghash, src, src_len);
ghash_update(&ghash, zeroes, -src_len & (GHASH_BLOCK_SIZE - 1));
ghash_update(&ghash, (const u8 *)&tail, sizeof(tail));
ghash_final(&ghash, ghash_out);
ctr[3] = cpu_to_be32(1);
aesgcm_encrypt_block(&ctx->aes_key, buf, ctr);
crypto_xor_cpy(authtag, buf, (u8 *)&ghash, ctx->authsize);
aes_encrypt(&ctx->aes_key, enc_ctr, (const u8 *)ctr);
crypto_xor_cpy(authtag, ghash_out, enc_ctr, ctx->authsize);
memzero_explicit(&ghash, sizeof(ghash));
memzero_explicit(buf, sizeof(buf));
memzero_explicit(ghash_out, sizeof(ghash_out));
memzero_explicit(enc_ctr, sizeof(enc_ctr));
}
static void aesgcm_crypt(const struct aesgcm_ctx *ctx, u8 *dst, const u8 *src,
@ -119,7 +101,7 @@ static void aesgcm_crypt(const struct aesgcm_ctx *ctx, u8 *dst, const u8 *src,
* len', this cannot happen, so no explicit test is necessary.
*/
ctr[3] = cpu_to_be32(n++);
aesgcm_encrypt_block(&ctx->aes_key, buf, ctr);
aes_encrypt(&ctx->aes_key, buf, (const u8 *)ctr);
crypto_xor_cpy(dst, src, buf, min(len, AES_BLOCK_SIZE));
dst += AES_BLOCK_SIZE;

43
lib/crypto/arm/gf128hash.h Normal file
View File

@ -0,0 +1,43 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* GHASH, arm optimized
*
* Copyright 2026 Google LLC
*/
#include <asm/hwcap.h>
#include <asm/neon.h>
#include <asm/simd.h>
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
void pmull_ghash_update_p8(size_t blocks, struct polyval_elem *dg,
const u8 *src, const struct polyval_elem *h);
#define ghash_blocks_arch ghash_blocks_arch
static void ghash_blocks_arch(struct polyval_elem *acc,
const struct ghash_key *key,
const u8 *data, size_t nblocks)
{
if (static_branch_likely(&have_neon) && may_use_simd()) {
do {
/* Allow rescheduling every 4 KiB. */
size_t n =
min_t(size_t, nblocks, 4096 / GHASH_BLOCK_SIZE);
scoped_ksimd()
pmull_ghash_update_p8(n, acc, data, &key->h);
data += n * GHASH_BLOCK_SIZE;
nblocks -= n;
} while (nblocks);
} else {
ghash_blocks_generic(acc, &key->h, data, nblocks);
}
}
#define gf128hash_mod_init_arch gf128hash_mod_init_arch
static void gf128hash_mod_init_arch(void)
{
if (elf_hwcap & HWCAP_NEON)
static_branch_enable(&have_neon);
}

209
lib/crypto/arm/ghash-neon-core.S Normal file
View File

@ -0,0 +1,209 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Accelerated GHASH implementation with NEON vmull.p8 instructions.
*
* Copyright (C) 2015 - 2017 Linaro Ltd.
* Copyright (C) 2023 Google LLC. <ardb@google.com>
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
.fpu neon
SHASH .req q0
T1 .req q1
XL .req q2
XM .req q3
XH .req q4
IN1 .req q4
SHASH_L .req d0
SHASH_H .req d1
T1_L .req d2
T1_H .req d3
XL_L .req d4
XL_H .req d5
XM_L .req d6
XM_H .req d7
XH_L .req d8
t0l .req d10
t0h .req d11
t1l .req d12
t1h .req d13
t2l .req d14
t2h .req d15
t3l .req d16
t3h .req d17
t4l .req d18
t4h .req d19
t0q .req q5
t1q .req q6
t2q .req q7
t3q .req q8
t4q .req q9
s1l .req d20
s1h .req d21
s2l .req d22
s2h .req d23
s3l .req d24
s3h .req d25
s4l .req d26
s4h .req d27
SHASH2_p8 .req d28
k16 .req d29
k32 .req d30
k48 .req d31
T2 .req q7
.text
/*
* This implementation of 64x64 -> 128 bit polynomial multiplication
* using vmull.p8 instructions (8x8 -> 16) is taken from the paper
* "Fast Software Polynomial Multiplication on ARM Processors Using
* the NEON Engine" by Danilo Camara, Conrado Gouvea, Julio Lopez and
* Ricardo Dahab (https://hal.inria.fr/hal-01506572)
*
* It has been slightly tweaked for in-order performance, and to allow
* 'rq' to overlap with 'ad' or 'bd'.
*/
.macro __pmull_p8, rq, ad, bd, b1=t4l, b2=t3l, b3=t4l, b4=t3l
vext.8 t0l, \ad, \ad, #1 @ A1
.ifc \b1, t4l
vext.8 t4l, \bd, \bd, #1 @ B1
.endif
vmull.p8 t0q, t0l, \bd @ F = A1*B
vext.8 t1l, \ad, \ad, #2 @ A2
vmull.p8 t4q, \ad, \b1 @ E = A*B1
.ifc \b2, t3l
vext.8 t3l, \bd, \bd, #2 @ B2
.endif
vmull.p8 t1q, t1l, \bd @ H = A2*B
vext.8 t2l, \ad, \ad, #3 @ A3
vmull.p8 t3q, \ad, \b2 @ G = A*B2
veor t0q, t0q, t4q @ L = E + F
.ifc \b3, t4l
vext.8 t4l, \bd, \bd, #3 @ B3
.endif
vmull.p8 t2q, t2l, \bd @ J = A3*B
veor t0l, t0l, t0h @ t0 = (L) (P0 + P1) << 8
veor t1q, t1q, t3q @ M = G + H
.ifc \b4, t3l
vext.8 t3l, \bd, \bd, #4 @ B4
.endif
vmull.p8 t4q, \ad, \b3 @ I = A*B3
veor t1l, t1l, t1h @ t1 = (M) (P2 + P3) << 16
vmull.p8 t3q, \ad, \b4 @ K = A*B4
vand t0h, t0h, k48
vand t1h, t1h, k32
veor t2q, t2q, t4q @ N = I + J
veor t0l, t0l, t0h
veor t1l, t1l, t1h
veor t2l, t2l, t2h @ t2 = (N) (P4 + P5) << 24
vand t2h, t2h, k16
veor t3l, t3l, t3h @ t3 = (K) (P6 + P7) << 32
vmov.i64 t3h, #0
vext.8 t0q, t0q, t0q, #15
veor t2l, t2l, t2h
vext.8 t1q, t1q, t1q, #14
vmull.p8 \rq, \ad, \bd @ D = A*B
vext.8 t2q, t2q, t2q, #13
vext.8 t3q, t3q, t3q, #12
veor t0q, t0q, t1q
veor t2q, t2q, t3q
veor \rq, \rq, t0q
veor \rq, \rq, t2q
.endm
.macro __pmull_reduce_p8
veor XL_H, XL_H, XM_L
veor XH_L, XH_L, XM_H
vshl.i64 T1, XL, #57
vshl.i64 T2, XL, #62
veor T1, T1, T2
vshl.i64 T2, XL, #63
veor T1, T1, T2
veor XL_H, XL_H, T1_L
veor XH_L, XH_L, T1_H
vshr.u64 T1, XL, #1
veor XH, XH, XL
veor XL, XL, T1
vshr.u64 T1, T1, #6
vshr.u64 XL, XL, #1
.endm
.macro vrev64_if_be a
#ifdef CONFIG_CPU_BIG_ENDIAN
vrev64.8 \a, \a
#endif
.endm
.macro ghash_update
vld1.64 {XL}, [r1]
vrev64_if_be XL
0:
vld1.8 {T1}, [r2]!
subs r0, r0, #1
/* multiply XL by SHASH in GF(2^128) */
vrev64.8 T1, T1
vext.8 IN1, T1, T1, #8
veor T1_L, T1_L, XL_H
veor XL, XL, IN1
__pmull_p8 XH, XL_H, SHASH_H, s1h, s2h, s3h, s4h @ a1 * b1
veor T1, T1, XL
__pmull_p8 XL, XL_L, SHASH_L, s1l, s2l, s3l, s4l @ a0 * b0
__pmull_p8 XM, T1_L, SHASH2_p8 @ (a1+a0)(b1+b0)
veor T1, XL, XH
veor XM, XM, T1
__pmull_reduce_p8
veor T1, T1, XH
veor XL, XL, T1
bne 0b
.endm
/*
* void pmull_ghash_update_p8(size_t blocks, struct polyval_elem *dg,
* const u8 *src,
* const struct polyval_elem *h)
*/
ENTRY(pmull_ghash_update_p8)
vld1.64 {SHASH}, [r3]
vrev64_if_be SHASH
veor SHASH2_p8, SHASH_L, SHASH_H
vext.8 s1l, SHASH_L, SHASH_L, #1
vext.8 s2l, SHASH_L, SHASH_L, #2
vext.8 s3l, SHASH_L, SHASH_L, #3
vext.8 s4l, SHASH_L, SHASH_L, #4
vext.8 s1h, SHASH_H, SHASH_H, #1
vext.8 s2h, SHASH_H, SHASH_H, #2
vext.8 s3h, SHASH_H, SHASH_H, #3
vext.8 s4h, SHASH_H, SHASH_H, #4
vmov.i64 k16, #0xffff
vmov.i64 k32, #0xffffffff
vmov.i64 k48, #0xffffffffffff
ghash_update
vrev64_if_be XL
vst1.64 {XL}, [r1]
bx lr
ENDPROC(pmull_ghash_update_p8)

3
arch/arm64/crypto/aes-ce.S → lib/crypto/arm64/aes-ce.S
View File

@ -1,7 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* linux/arch/arm64/crypto/aes-ce.S - AES cipher for ARMv8 with
* Crypto Extensions
* AES cipher for ARMv8 with Crypto Extensions
*
* Copyright (C) 2013 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
*/

10
lib/crypto/arm64/aes-cipher-core.S
View File

@ -87,11 +87,6 @@
ldp w8, w9, [rk], #16
ldp w10, w11, [rk, #-8]
CPU_BE( rev w4, w4 )
CPU_BE( rev w5, w5 )
CPU_BE( rev w6, w6 )
CPU_BE( rev w7, w7 )
eor w4, w4, w8
eor w5, w5, w9
eor w6, w6, w10
@ -112,11 +107,6 @@ CPU_BE( rev w7, w7 )
3: adr_l tt, \ltab
\round w4, w5, w6, w7, w8, w9, w10, w11, \bsz, b
CPU_BE( rev w4, w4 )
CPU_BE( rev w5, w5 )
CPU_BE( rev w6, w6 )
CPU_BE( rev w7, w7 )
stp w4, w5, [out]
stp w6, w7, [out, #8]
ret

25
arch/arm64/crypto/aes-modes.S → lib/crypto/arm64/aes-modes.S
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* linux/arch/arm64/crypto/aes-modes.S - chaining mode wrappers for AES
* Chaining mode wrappers for AES
*
* Copyright (C) 2013 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
@ -815,9 +815,11 @@ AES_FUNC_START(aes_xts_decrypt)
b .Lxtsdecctsout
AES_FUNC_END(aes_xts_decrypt)
#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC_MACS)
/*
* aes_mac_update(u8 const in[], u32 const rk[], int rounds,
* int blocks, u8 dg[], int enc_before, int enc_after)
* void aes_mac_update(u8 const in[], u32 const rk[], int rounds,
* size_t blocks, u8 dg[], int enc_before,
* int enc_after);
*/
AES_FUNC_START(aes_mac_update)
ld1 {v0.16b}, [x4] /* get dg */
@ -827,7 +829,7 @@ AES_FUNC_START(aes_mac_update)
encrypt_block v0, w2, x1, x7, w8
.Lmacloop4x:
subs w3, w3, #4
subs x3, x3, #4
bmi .Lmac1x
ld1 {v1.16b-v4.16b}, [x0], #64 /* get next pt block */
eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */
@ -837,22 +839,21 @@ AES_FUNC_START(aes_mac_update)
eor v0.16b, v0.16b, v3.16b
encrypt_block v0, w2, x1, x7, w8
eor v0.16b, v0.16b, v4.16b
cmp w3, wzr
csinv x5, x6, xzr, eq
cmp x3, xzr
csinv w5, w6, wzr, eq
cbz w5, .Lmacout
encrypt_block v0, w2, x1, x7, w8
st1 {v0.16b}, [x4] /* return dg */
cond_yield .Lmacout, x7, x8
b .Lmacloop4x
.Lmac1x:
add w3, w3, #4
add x3, x3, #4
.Lmacloop:
cbz w3, .Lmacout
cbz x3, .Lmacout
ld1 {v1.16b}, [x0], #16 /* get next pt block */
eor v0.16b, v0.16b, v1.16b /* ..and xor with dg */
subs w3, w3, #1
csinv x5, x6, xzr, eq
subs x3, x3, #1
csinv w5, w6, wzr, eq
cbz w5, .Lmacout
.Lmacenc:
@ -861,6 +862,6 @@ AES_FUNC_START(aes_mac_update)
.Lmacout:
st1 {v0.16b}, [x4] /* return dg */
mov w0, w3
ret
AES_FUNC_END(aes_mac_update)
#endif /* CONFIG_CRYPTO_LIB_AES_CBC_MACS */

2
arch/arm64/crypto/aes-neon.S → lib/crypto/arm64/aes-neon.S
View File

@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* linux/arch/arm64/crypto/aes-neon.S - AES cipher for ARMv8 NEON
* AES cipher for ARMv8 NEON
*
* Copyright (C) 2013 - 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/

75
lib/crypto/arm64/aes.h
View File

@ -11,6 +11,7 @@
#include <linux/unaligned.h>
#include <linux/cpufeature.h>
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_aes);
struct aes_block {
@ -28,6 +29,9 @@ asmlinkage void __aes_ce_decrypt(const u32 inv_rk[], u8 out[AES_BLOCK_SIZE],
asmlinkage u32 __aes_ce_sub(u32 l);
asmlinkage void __aes_ce_invert(struct aes_block *out,
const struct aes_block *in);
asmlinkage void neon_aes_mac_update(u8 const in[], u32 const rk[], int rounds,
size_t blocks, u8 dg[], int enc_before,
int enc_after);
/*
* Expand an AES key using the crypto extensions if supported and usable or
@ -48,8 +52,7 @@ static void aes_expandkey_arm64(u32 rndkeys[], u32 *inv_rndkeys,
struct aes_block *key_enc, *key_dec;
int i, j;
if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) ||
!static_branch_likely(&have_aes) || unlikely(!may_use_simd())) {
if (!static_branch_likely(&have_aes) || unlikely(!may_use_simd())) {
aes_expandkey_generic(rndkeys, inv_rndkeys, in_key, key_len);
return;
}
@ -126,12 +129,40 @@ int ce_aes_expandkey(struct crypto_aes_ctx *ctx, const u8 *in_key,
}
EXPORT_SYMBOL(ce_aes_expandkey);
EXPORT_SYMBOL_NS_GPL(neon_aes_ecb_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_ecb_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_cbc_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_cbc_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_cbc_cts_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_cbc_cts_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_ctr_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_xctr_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_xts_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_xts_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_essiv_cbc_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(neon_aes_essiv_cbc_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_ecb_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_ecb_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_cbc_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_cbc_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_cbc_cts_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_cbc_cts_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_ctr_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_xctr_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_xts_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_xts_decrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_essiv_cbc_encrypt, "CRYPTO_INTERNAL");
EXPORT_SYMBOL_NS_GPL(ce_aes_essiv_cbc_decrypt, "CRYPTO_INTERNAL");
#if IS_MODULE(CONFIG_CRYPTO_AES_ARM64_CE_CCM)
EXPORT_SYMBOL_NS_GPL(ce_aes_mac_update, "CRYPTO_INTERNAL");
#endif
static void aes_encrypt_arch(const struct aes_enckey *key,
u8 out[AES_BLOCK_SIZE],
const u8 in[AES_BLOCK_SIZE])
{
if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
static_branch_likely(&have_aes) && likely(may_use_simd())) {
if (static_branch_likely(&have_aes) && likely(may_use_simd())) {
scoped_ksimd()
__aes_ce_encrypt(key->k.rndkeys, out, in, key->nrounds);
} else {
@ -143,8 +174,7 @@ static void aes_decrypt_arch(const struct aes_key *key,
u8 out[AES_BLOCK_SIZE],
const u8 in[AES_BLOCK_SIZE])
{
if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
static_branch_likely(&have_aes) && likely(may_use_simd())) {
if (static_branch_likely(&have_aes) && likely(may_use_simd())) {
scoped_ksimd()
__aes_ce_decrypt(key->inv_k.inv_rndkeys, out, in,
key->nrounds);
@ -154,11 +184,36 @@ static void aes_decrypt_arch(const struct aes_key *key,
}
}
#ifdef CONFIG_KERNEL_MODE_NEON
#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC_MACS)
#define aes_cbcmac_blocks_arch aes_cbcmac_blocks_arch
static bool aes_cbcmac_blocks_arch(u8 h[AES_BLOCK_SIZE],
const struct aes_enckey *key, const u8 *data,
size_t nblocks, bool enc_before,
bool enc_after)
{
if (static_branch_likely(&have_neon) && likely(may_use_simd())) {
scoped_ksimd() {
if (static_branch_likely(&have_aes))
ce_aes_mac_update(data, key->k.rndkeys,
key->nrounds, nblocks, h,
enc_before, enc_after);
else
neon_aes_mac_update(data, key->k.rndkeys,
key->nrounds, nblocks, h,
enc_before, enc_after);
}
return true;
}
return false;
}
#endif /* CONFIG_CRYPTO_LIB_AES_CBC_MACS */
#define aes_mod_init_arch aes_mod_init_arch
static void aes_mod_init_arch(void)
{
if (cpu_have_named_feature(AES))
static_branch_enable(&have_aes);
if (cpu_have_named_feature(ASIMD)) {
static_branch_enable(&have_neon);
if (cpu_have_named_feature(AES))
static_branch_enable(&have_aes);
}
}
#endif /* CONFIG_KERNEL_MODE_NEON */

16
lib/crypto/arm64/chacha-neon-core.S
View File

@ -531,10 +531,6 @@ SYM_FUNC_START(chacha_4block_xor_neon)
add v3.4s, v3.4s, v19.4s
add a2, a2, w8
add a3, a3, w9
CPU_BE( rev a0, a0 )
CPU_BE( rev a1, a1 )
CPU_BE( rev a2, a2 )
CPU_BE( rev a3, a3 )
ld4r {v24.4s-v27.4s}, [x0], #16
ld4r {v28.4s-v31.4s}, [x0]
@ -555,10 +551,6 @@ CPU_BE( rev a3, a3 )
add v7.4s, v7.4s, v23.4s
add a6, a6, w8
add a7, a7, w9
CPU_BE( rev a4, a4 )
CPU_BE( rev a5, a5 )
CPU_BE( rev a6, a6 )
CPU_BE( rev a7, a7 )
// x8[0-3] += s2[0]
// x9[0-3] += s2[1]
@ -576,10 +568,6 @@ CPU_BE( rev a7, a7 )
add v11.4s, v11.4s, v27.4s
add a10, a10, w8
add a11, a11, w9
CPU_BE( rev a8, a8 )
CPU_BE( rev a9, a9 )
CPU_BE( rev a10, a10 )
CPU_BE( rev a11, a11 )
// x12[0-3] += s3[0]
// x13[0-3] += s3[1]
@ -597,10 +585,6 @@ CPU_BE( rev a11, a11 )
add v15.4s, v15.4s, v31.4s
add a14, a14, w8
add a15, a15, w9
CPU_BE( rev a12, a12 )
CPU_BE( rev a13, a13 )
CPU_BE( rev a14, a14 )
CPU_BE( rev a15, a15 )
// interleave 32-bit words in state n, n+1
ldp w6, w7, [x2], #64

16
lib/crypto/arm64/chacha.h
View File

@ -36,9 +36,9 @@ asmlinkage void hchacha_block_neon(const struct chacha_state *state,
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
int bytes, int nrounds)
unsigned int bytes, int nrounds)
{
while (bytes > 0) {
while (bytes) {
int l = min(bytes, CHACHA_BLOCK_SIZE * 5);
if (l <= CHACHA_BLOCK_SIZE) {
@ -76,16 +76,8 @@ static void chacha_crypt_arch(struct chacha_state *state, u8 *dst,
!crypto_simd_usable())
return chacha_crypt_generic(state, dst, src, bytes, nrounds);
do {
unsigned int todo = min_t(unsigned int, bytes, SZ_4K);
scoped_ksimd()
chacha_doneon(state, dst, src, todo, nrounds);
bytes -= todo;
src += todo;
dst += todo;
} while (bytes);
scoped_ksimd()
chacha_doneon(state, dst, src, bytes, nrounds);
}
#define chacha_mod_init_arch chacha_mod_init_arch

121
lib/crypto/arm64/gf128hash.h Normal file
View File

@ -0,0 +1,121 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* GHASH and POLYVAL, arm64 optimized
*
* Copyright 2025 Google LLC
*/
#include <asm/simd.h>
#include <linux/cpufeature.h>
#define NUM_H_POWERS 8
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_asimd);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
asmlinkage void pmull_ghash_update_p8(size_t blocks, struct polyval_elem *dg,
const u8 *src,
const struct polyval_elem *h);
asmlinkage void polyval_mul_pmull(struct polyval_elem *a,
const struct polyval_elem *b);
asmlinkage void polyval_blocks_pmull(struct polyval_elem *acc,
const struct polyval_key *key,
const u8 *data, size_t nblocks);
#define polyval_preparekey_arch polyval_preparekey_arch
static void polyval_preparekey_arch(struct polyval_key *key,
const u8 raw_key[POLYVAL_BLOCK_SIZE])
{
static_assert(ARRAY_SIZE(key->h_powers) == NUM_H_POWERS);
memcpy(&key->h_powers[NUM_H_POWERS - 1], raw_key, POLYVAL_BLOCK_SIZE);
if (static_branch_likely(&have_pmull) && may_use_simd()) {
scoped_ksimd() {
for (int i = NUM_H_POWERS - 2; i >= 0; i--) {
key->h_powers[i] = key->h_powers[i + 1];
polyval_mul_pmull(
&key->h_powers[i],
&key->h_powers[NUM_H_POWERS - 1]);
}
}
} else {
for (int i = NUM_H_POWERS - 2; i >= 0; i--) {
key->h_powers[i] = key->h_powers[i + 1];
polyval_mul_generic(&key->h_powers[i],
&key->h_powers[NUM_H_POWERS - 1]);
}
}
}
static void polyval_mul_arm64(struct polyval_elem *a,
const struct polyval_elem *b)
{
if (static_branch_likely(&have_asimd) && may_use_simd()) {
static const u8 zeroes[GHASH_BLOCK_SIZE];
scoped_ksimd() {
if (static_branch_likely(&have_pmull)) {
polyval_mul_pmull(a, b);
} else {
/*
* Note that this is indeed equivalent to a
* POLYVAL multiplication, since it takes the
* accumulator and key in POLYVAL format, and
* byte-swapping a block of zeroes is a no-op.
*/
pmull_ghash_update_p8(1, a, zeroes, b);
}
}
} else {
polyval_mul_generic(a, b);
}
}
#define ghash_mul_arch ghash_mul_arch
static void ghash_mul_arch(struct polyval_elem *acc,
const struct ghash_key *key)
{
polyval_mul_arm64(acc, &key->h);
}
#define polyval_mul_arch polyval_mul_arch
static void polyval_mul_arch(struct polyval_elem *acc,
const struct polyval_key *key)
{
polyval_mul_arm64(acc, &key->h_powers[NUM_H_POWERS - 1]);
}
#define ghash_blocks_arch ghash_blocks_arch
static void ghash_blocks_arch(struct polyval_elem *acc,
const struct ghash_key *key,
const u8 *data, size_t nblocks)
{
if (static_branch_likely(&have_asimd) && may_use_simd()) {
scoped_ksimd()
pmull_ghash_update_p8(nblocks, acc, data, &key->h);
} else {
ghash_blocks_generic(acc, &key->h, data, nblocks);
}
}
#define polyval_blocks_arch polyval_blocks_arch
static void polyval_blocks_arch(struct polyval_elem *acc,
const struct polyval_key *key,
const u8 *data, size_t nblocks)
{
if (static_branch_likely(&have_pmull) && may_use_simd()) {
scoped_ksimd()
polyval_blocks_pmull(acc, key, data, nblocks);
} else {
polyval_blocks_generic(acc, &key->h_powers[NUM_H_POWERS - 1],
data, nblocks);
}
}
#define gf128hash_mod_init_arch gf128hash_mod_init_arch
static void gf128hash_mod_init_arch(void)
{
if (cpu_have_named_feature(ASIMD)) {
static_branch_enable(&have_asimd);
if (cpu_have_named_feature(PMULL))
static_branch_enable(&have_pmull);
}
}

220
lib/crypto/arm64/ghash-neon-core.S Normal file
View File

@ -0,0 +1,220 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Accelerated GHASH implementation with ARMv8 ASIMD instructions.
*
* Copyright (C) 2014 - 2018 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
#include <linux/linkage.h>
#include <asm/assembler.h>
SHASH .req v0
SHASH2 .req v1
T1 .req v2
T2 .req v3
XM .req v5
XL .req v6
XH .req v7
IN1 .req v7
k00_16 .req v8
k32_48 .req v9
t3 .req v10
t4 .req v11
t5 .req v12
t6 .req v13
t7 .req v14
t8 .req v15
t9 .req v16
perm1 .req v17
perm2 .req v18
perm3 .req v19
sh1 .req v20
sh2 .req v21
sh3 .req v22
sh4 .req v23
ss1 .req v24
ss2 .req v25
ss3 .req v26
ss4 .req v27
.text
.macro __pmull_p8, rq, ad, bd
ext t3.8b, \ad\().8b, \ad\().8b, #1 // A1
ext t5.8b, \ad\().8b, \ad\().8b, #2 // A2
ext t7.8b, \ad\().8b, \ad\().8b, #3 // A3
__pmull_p8_\bd \rq, \ad
.endm
.macro __pmull2_p8, rq, ad, bd
tbl t3.16b, {\ad\().16b}, perm1.16b // A1
tbl t5.16b, {\ad\().16b}, perm2.16b // A2
tbl t7.16b, {\ad\().16b}, perm3.16b // A3
__pmull2_p8_\bd \rq, \ad
.endm
.macro __pmull_p8_SHASH, rq, ad
__pmull_p8_tail \rq, \ad\().8b, SHASH.8b, 8b,, sh1, sh2, sh3, sh4
.endm
.macro __pmull_p8_SHASH2, rq, ad
__pmull_p8_tail \rq, \ad\().8b, SHASH2.8b, 8b,, ss1, ss2, ss3, ss4
.endm
.macro __pmull2_p8_SHASH, rq, ad
__pmull_p8_tail \rq, \ad\().16b, SHASH.16b, 16b, 2, sh1, sh2, sh3, sh4
.endm
.macro __pmull_p8_tail, rq, ad, bd, nb, t, b1, b2, b3, b4
pmull\t t3.8h, t3.\nb, \bd // F = A1*B
pmull\t t4.8h, \ad, \b1\().\nb // E = A*B1
pmull\t t5.8h, t5.\nb, \bd // H = A2*B
pmull\t t6.8h, \ad, \b2\().\nb // G = A*B2
pmull\t t7.8h, t7.\nb, \bd // J = A3*B
pmull\t t8.8h, \ad, \b3\().\nb // I = A*B3
pmull\t t9.8h, \ad, \b4\().\nb // K = A*B4
pmull\t \rq\().8h, \ad, \bd // D = A*B
eor t3.16b, t3.16b, t4.16b // L = E + F
eor t5.16b, t5.16b, t6.16b // M = G + H
eor t7.16b, t7.16b, t8.16b // N = I + J
uzp1 t4.2d, t3.2d, t5.2d
uzp2 t3.2d, t3.2d, t5.2d
uzp1 t6.2d, t7.2d, t9.2d
uzp2 t7.2d, t7.2d, t9.2d
// t3 = (L) (P0 + P1) << 8
// t5 = (M) (P2 + P3) << 16
eor t4.16b, t4.16b, t3.16b
and t3.16b, t3.16b, k32_48.16b
// t7 = (N) (P4 + P5) << 24
// t9 = (K) (P6 + P7) << 32
eor t6.16b, t6.16b, t7.16b
and t7.16b, t7.16b, k00_16.16b
eor t4.16b, t4.16b, t3.16b
eor t6.16b, t6.16b, t7.16b
zip2 t5.2d, t4.2d, t3.2d
zip1 t3.2d, t4.2d, t3.2d
zip2 t9.2d, t6.2d, t7.2d
zip1 t7.2d, t6.2d, t7.2d
ext t3.16b, t3.16b, t3.16b, #15
ext t5.16b, t5.16b, t5.16b, #14
ext t7.16b, t7.16b, t7.16b, #13
ext t9.16b, t9.16b, t9.16b, #12
eor t3.16b, t3.16b, t5.16b
eor t7.16b, t7.16b, t9.16b
eor \rq\().16b, \rq\().16b, t3.16b
eor \rq\().16b, \rq\().16b, t7.16b
.endm
.macro __pmull_pre_p8
ext SHASH2.16b, SHASH.16b, SHASH.16b, #8
eor SHASH2.16b, SHASH2.16b, SHASH.16b
// k00_16 := 0x0000000000000000_000000000000ffff
// k32_48 := 0x00000000ffffffff_0000ffffffffffff
movi k32_48.2d, #0xffffffff
mov k32_48.h[2], k32_48.h[0]
ushr k00_16.2d, k32_48.2d, #32
// prepare the permutation vectors
mov_q x5, 0x080f0e0d0c0b0a09
movi T1.8b, #8
dup perm1.2d, x5
eor perm1.16b, perm1.16b, T1.16b
ushr perm2.2d, perm1.2d, #8
ushr perm3.2d, perm1.2d, #16
ushr T1.2d, perm1.2d, #24
sli perm2.2d, perm1.2d, #56
sli perm3.2d, perm1.2d, #48
sli T1.2d, perm1.2d, #40
// precompute loop invariants
tbl sh1.16b, {SHASH.16b}, perm1.16b
tbl sh2.16b, {SHASH.16b}, perm2.16b
tbl sh3.16b, {SHASH.16b}, perm3.16b
tbl sh4.16b, {SHASH.16b}, T1.16b
ext ss1.8b, SHASH2.8b, SHASH2.8b, #1
ext ss2.8b, SHASH2.8b, SHASH2.8b, #2
ext ss3.8b, SHASH2.8b, SHASH2.8b, #3
ext ss4.8b, SHASH2.8b, SHASH2.8b, #4
.endm
.macro __pmull_reduce_p8
eor XM.16b, XM.16b, T1.16b
mov XL.d[1], XM.d[0]
mov XH.d[0], XM.d[1]
shl T1.2d, XL.2d, #57
shl T2.2d, XL.2d, #62
eor T2.16b, T2.16b, T1.16b
shl T1.2d, XL.2d, #63
eor T2.16b, T2.16b, T1.16b
ext T1.16b, XL.16b, XH.16b, #8
eor T2.16b, T2.16b, T1.16b
mov XL.d[1], T2.d[0]
mov XH.d[0], T2.d[1]
ushr T2.2d, XL.2d, #1
eor XH.16b, XH.16b, XL.16b
eor XL.16b, XL.16b, T2.16b
ushr T2.2d, T2.2d, #6
ushr XL.2d, XL.2d, #1
.endm
/*
* void pmull_ghash_update_p8(size_t blocks, struct polyval_elem *dg,
* const u8 *src,
* const struct polyval_elem *h)
*/
SYM_FUNC_START(pmull_ghash_update_p8)
ld1 {SHASH.2d}, [x3]
ld1 {XL.2d}, [x1]
__pmull_pre_p8
0: ld1 {T1.2d}, [x2], #16
sub x0, x0, #1
/* multiply XL by SHASH in GF(2^128) */
rev64 T1.16b, T1.16b
ext T2.16b, XL.16b, XL.16b, #8
ext IN1.16b, T1.16b, T1.16b, #8
eor T1.16b, T1.16b, T2.16b
eor XL.16b, XL.16b, IN1.16b
__pmull2_p8 XH, XL, SHASH // a1 * b1
eor T1.16b, T1.16b, XL.16b
__pmull_p8 XL, XL, SHASH // a0 * b0
__pmull_p8 XM, T1, SHASH2 // (a1 + a0)(b1 + b0)
eor T2.16b, XL.16b, XH.16b
ext T1.16b, XL.16b, XH.16b, #8
eor XM.16b, XM.16b, T2.16b
__pmull_reduce_p8
eor T2.16b, T2.16b, XH.16b
eor XL.16b, XL.16b, T2.16b
cbnz x0, 0b
st1 {XL.2d}, [x1]
ret
SYM_FUNC_END(pmull_ghash_update_p8)

14
lib/crypto/arm64/poly1305.h
View File

@ -27,17 +27,11 @@ static void poly1305_blocks(struct poly1305_block_state *state, const u8 *src,
unsigned int len, u32 padbit)
{
if (static_branch_likely(&have_neon) && likely(may_use_simd())) {
do {
unsigned int todo = min_t(unsigned int, len, SZ_4K);
scoped_ksimd()
poly1305_blocks_neon(state, src, todo, padbit);
len -= todo;
src += todo;
} while (len);
} else
scoped_ksimd()
poly1305_blocks_neon(state, src, len, padbit);
} else {
poly1305_blocks_arm64(state, src, len, padbit);
}
}
#define poly1305_mod_init_arch poly1305_mod_init_arch

80
lib/crypto/arm64/polyval.h
View File

@ -1,80 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* POLYVAL library functions, arm64 optimized
*
* Copyright 2025 Google LLC
*/
#include <asm/simd.h>
#include <linux/cpufeature.h>
#define NUM_H_POWERS 8
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
asmlinkage void polyval_mul_pmull(struct polyval_elem *a,
const struct polyval_elem *b);
asmlinkage void polyval_blocks_pmull(struct polyval_elem *acc,
const struct polyval_key *key,
const u8 *data, size_t nblocks);
static void polyval_preparekey_arch(struct polyval_key *key,
const u8 raw_key[POLYVAL_BLOCK_SIZE])
{
static_assert(ARRAY_SIZE(key->h_powers) == NUM_H_POWERS);
memcpy(&key->h_powers[NUM_H_POWERS - 1], raw_key, POLYVAL_BLOCK_SIZE);
if (static_branch_likely(&have_pmull) && may_use_simd()) {
scoped_ksimd() {
for (int i = NUM_H_POWERS - 2; i >= 0; i--) {
key->h_powers[i] = key->h_powers[i + 1];
polyval_mul_pmull(
&key->h_powers[i],
&key->h_powers[NUM_H_POWERS - 1]);
}
}
} else {
for (int i = NUM_H_POWERS - 2; i >= 0; i--) {
key->h_powers[i] = key->h_powers[i + 1];
polyval_mul_generic(&key->h_powers[i],
&key->h_powers[NUM_H_POWERS - 1]);
}
}
}
static void polyval_mul_arch(struct polyval_elem *acc,
const struct polyval_key *key)
{
if (static_branch_likely(&have_pmull) && may_use_simd()) {
scoped_ksimd()
polyval_mul_pmull(acc, &key->h_powers[NUM_H_POWERS - 1]);
} else {
polyval_mul_generic(acc, &key->h_powers[NUM_H_POWERS - 1]);
}
}
static void polyval_blocks_arch(struct polyval_elem *acc,
const struct polyval_key *key,
const u8 *data, size_t nblocks)
{
if (static_branch_likely(&have_pmull) && may_use_simd()) {
do {
/* Allow rescheduling every 4 KiB. */
size_t n = min_t(size_t, nblocks,
4096 / POLYVAL_BLOCK_SIZE);
scoped_ksimd()
polyval_blocks_pmull(acc, key, data, n);
data += n * POLYVAL_BLOCK_SIZE;
nblocks -= n;
} while (nblocks);
} else {
polyval_blocks_generic(acc, &key->h_powers[NUM_H_POWERS - 1],
data, nblocks);
}
}
#define polyval_mod_init_arch polyval_mod_init_arch
static void polyval_mod_init_arch(void)
{
if (cpu_have_named_feature(PMULL))
static_branch_enable(&have_pmull);
}

22
lib/crypto/arm64/sha1-ce-core.S
View File

@ -62,10 +62,10 @@
.endm
/*
* size_t __sha1_ce_transform(struct sha1_block_state *state,
* const u8 *data, size_t nblocks);
* void sha1_ce_transform(struct sha1_block_state *state,
* const u8 *data, size_t nblocks);
*/
SYM_FUNC_START(__sha1_ce_transform)
SYM_FUNC_START(sha1_ce_transform)
/* load round constants */
loadrc k0.4s, 0x5a827999, w6
loadrc k1.4s, 0x6ed9eba1, w6
@ -80,10 +80,10 @@ SYM_FUNC_START(__sha1_ce_transform)
0: ld1 {v8.4s-v11.4s}, [x1], #64
sub x2, x2, #1
CPU_LE( rev32 v8.16b, v8.16b )
CPU_LE( rev32 v9.16b, v9.16b )
CPU_LE( rev32 v10.16b, v10.16b )
CPU_LE( rev32 v11.16b, v11.16b )
rev32 v8.16b, v8.16b
rev32 v9.16b, v9.16b
rev32 v10.16b, v10.16b
rev32 v11.16b, v11.16b
add t0.4s, v8.4s, k0.4s
mov dg0v.16b, dgav.16b
@ -116,15 +116,11 @@ CPU_LE( rev32 v11.16b, v11.16b )
add dgbv.2s, dgbv.2s, dg1v.2s
add dgav.4s, dgav.4s, dg0v.4s
/* return early if voluntary preemption is needed */
cond_yield 1f, x5, x6
/* handled all input blocks? */
cbnz x2, 0b
/* store new state */
1: st1 {dgav.4s}, [x0]
st1 {dgav.4s}, [x0]
str dgb, [x0, #16]
mov x0, x2
ret
SYM_FUNC_END(__sha1_ce_transform)
SYM_FUNC_END(sha1_ce_transform)

15
lib/crypto/arm64/sha1.h
View File

@ -9,22 +9,15 @@
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_ce);
asmlinkage size_t __sha1_ce_transform(struct sha1_block_state *state,
const u8 *data, size_t nblocks);
asmlinkage void sha1_ce_transform(struct sha1_block_state *state,
const u8 *data, size_t nblocks);
static void sha1_blocks(struct sha1_block_state *state,
const u8 *data, size_t nblocks)
{
if (static_branch_likely(&have_ce) && likely(may_use_simd())) {
do {
size_t rem;
scoped_ksimd()
rem = __sha1_ce_transform(state, data, nblocks);
data += (nblocks - rem) * SHA1_BLOCK_SIZE;
nblocks = rem;
} while (nblocks);
scoped_ksimd()
sha1_ce_transform(state, data, nblocks);
} else {
sha1_blocks_generic(state, data, nblocks);
}

55
lib/crypto/arm64/sha256-ce.S
View File

@ -79,11 +79,11 @@
.endm
/*
* size_t __sha256_ce_transform(struct sha256_block_state *state,
* const u8 *data, size_t nblocks);
* void sha256_ce_transform(struct sha256_block_state *state,
* const u8 *data, size_t nblocks);
*/
.text
SYM_FUNC_START(__sha256_ce_transform)
SYM_FUNC_START(sha256_ce_transform)
load_round_constants x8
@ -94,10 +94,10 @@ SYM_FUNC_START(__sha256_ce_transform)
0: ld1 {v16.4s-v19.4s}, [x1], #64
sub x2, x2, #1
CPU_LE( rev32 v16.16b, v16.16b )
CPU_LE( rev32 v17.16b, v17.16b )
CPU_LE( rev32 v18.16b, v18.16b )
CPU_LE( rev32 v19.16b, v19.16b )
rev32 v16.16b, v16.16b
rev32 v17.16b, v17.16b
rev32 v18.16b, v18.16b
rev32 v19.16b, v19.16b
add t0.4s, v16.4s, v0.4s
mov dg0v.16b, dgav.16b
@ -127,17 +127,13 @@ CPU_LE( rev32 v19.16b, v19.16b )
add dgav.4s, dgav.4s, dg0v.4s
add dgbv.4s, dgbv.4s, dg1v.4s
/* return early if voluntary preemption is needed */
cond_yield 1f, x5, x6
/* handled all input blocks? */
cbnz x2, 0b
/* store new state */
1: st1 {dgav.4s, dgbv.4s}, [x0]
mov x0, x2
st1 {dgav.4s, dgbv.4s}, [x0]
ret
SYM_FUNC_END(__sha256_ce_transform)
SYM_FUNC_END(sha256_ce_transform)
.unreq dga
.unreq dgav
@ -293,14 +289,14 @@ SYM_FUNC_START(sha256_ce_finup2x)
ld1 {v20.4s-v23.4s}, [data2], #64
.Lfinup2x_loop_have_data:
// Convert the words of the data blocks from big endian.
CPU_LE( rev32 v16.16b, v16.16b )
CPU_LE( rev32 v17.16b, v17.16b )
CPU_LE( rev32 v18.16b, v18.16b )
CPU_LE( rev32 v19.16b, v19.16b )
CPU_LE( rev32 v20.16b, v20.16b )
CPU_LE( rev32 v21.16b, v21.16b )
CPU_LE( rev32 v22.16b, v22.16b )
CPU_LE( rev32 v23.16b, v23.16b )
rev32 v16.16b, v16.16b
rev32 v17.16b, v17.16b
rev32 v18.16b, v18.16b
rev32 v19.16b, v19.16b
rev32 v20.16b, v20.16b
rev32 v21.16b, v21.16b
rev32 v22.16b, v22.16b
rev32 v23.16b, v23.16b
.Lfinup2x_loop_have_bswapped_data:
// Save the original state for each block.
@ -340,11 +336,8 @@ CPU_LE( rev32 v23.16b, v23.16b )
sub w8, len, #64 // w8 = len - 64
add data1, data1, w8, sxtw // data1 += len - 64
add data2, data2, w8, sxtw // data2 += len - 64
CPU_LE( mov x9, #0x80 )
CPU_LE( fmov d16, x9 )
CPU_BE( movi v16.16b, #0 )
CPU_BE( mov x9, #0x8000000000000000 )
CPU_BE( mov v16.d[1], x9 )
mov x9, #0x80
fmov d16, x9
movi v17.16b, #0
stp q16, q17, [sp, #64]
stp q17, q17, [sp, #96]
@ -352,7 +345,7 @@ CPU_BE( mov v16.d[1], x9 )
cmp len, #56
b.ge 1f // will count spill into its own block?
lsl count, count, #3
CPU_LE( rev count, count )
rev count, count
str count, [x9, #56]
mov final_step, #2 // won't need count-only block
b 2f
@ -397,10 +390,10 @@ CPU_LE( rev count, count )
.Lfinup2x_done:
// Write the two digests with all bytes in the correct order.
CPU_LE( rev32 state0_a.16b, state0_a.16b )
CPU_LE( rev32 state1_a.16b, state1_a.16b )
CPU_LE( rev32 state0_b.16b, state0_b.16b )
CPU_LE( rev32 state1_b.16b, state1_b.16b )
rev32 state0_a.16b, state0_a.16b
rev32 state1_a.16b, state1_a.16b
rev32 state0_b.16b, state0_b.16b
rev32 state1_b.16b, state1_b.16b
st1 {state0_a.4s-state1_a.4s}, [out1]
st1 {state0_b.4s-state1_b.4s}, [out2]
add sp, sp, #128

Some files were not shown because too many files have changed in this diff Show More