mirror-linux/include/net
Eric Dumazet 133c4c0d37 tcp: defer regular ACK while processing socket backlog
This idea came after a particular workload requested
the quickack attribute set on routes, and a performance
drop was noticed for large bulk transfers.

For high throughput flows, it is best to use one cpu
running the user thread issuing socket system calls,
and a separate cpu to process incoming packets from BH context.
(With TSO/GRO, bottleneck is usually the 'user' cpu)

Problem is the user thread can spend a lot of time while holding
the socket lock, forcing BH handler to queue most of incoming
packets in the socket backlog.

Whenever the user thread releases the socket lock, it must first
process all accumulated packets in the backlog, potentially
adding latency spikes. Due to flood mitigation, having too many
packets in the backlog increases chance of unexpected drops.

Backlog processing unfortunately shifts a fair amount of cpu cycles
from the BH cpu to the 'user' cpu, thus reducing max throughput.

This patch takes advantage of the backlog processing,
and the fact that ACK are mostly cumulative.

The idea is to detect we are in the backlog processing
and defer all eligible ACK into a single one,
sent from tcp_release_cb().

This saves cpu cycles on both sides, and network resources.

Performance of a single TCP flow on a 200Gbit NIC:

- Throughput is increased by 20% (100Gbit -> 120Gbit).
- Number of generated ACK per second shrinks from 240,000 to 40,000.
- Number of backlog drops per second shrinks from 230 to 0.

Benchmark context:
 - Regular netperf TCP_STREAM (no zerocopy)
 - Intel(R) Xeon(R) Platinum 8481C (Saphire Rapids)
 - MAX_SKB_FRAGS = 17 (~60KB per GRO packet)

This feature is guarded by a new sysctl, and enabled by default:
 /proc/sys/net/ipv4/tcp_backlog_ack_defer

Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Acked-by: Dave Taht <dave.taht@gmail.com>
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
2023-09-12 19:10:01 +02:00
..
9p 9p: Add additional debug flags and open modes 2023-03-27 02:33:48 +00:00
bluetooth Bluetooth: HCI: Introduce HCI_QUIRK_BROKEN_LE_CODED 2023-08-24 12:23:46 -07:00
caif net: caif: Remove unused declaration cfsrvl_ctrlcmd() 2023-08-10 18:24:48 -07:00
iucv net/af_iucv: Use struct_group() to zero struct iucv_sock region 2021-11-19 11:52:25 +00:00
mana net: mana: Add gdma stats to ethtool output for mana 2023-08-11 09:53:59 +01:00
netfilter Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-08-24 10:51:39 -07:00
netns tcp: defer regular ACK while processing socket backlog 2023-09-12 19:10:01 +02:00
nfc NFC: add NCI_UNREG flag to eliminate the race 2021-11-17 20:17:05 -08:00
page_pool docs: net: page_pool: de-duplicate the intro comment 2023-08-08 16:09:10 -07:00
phonet net: ioctl: Use kernel memory on protocol ioctl callbacks 2023-06-15 22:33:26 -07:00
sctp sctp: Remove unused declaration sctp_backlog_migrate() 2023-08-10 19:31:52 -07:00
tc_act net/sched: act_connmark: transition to percpu stats and rcu 2023-02-16 10:39:28 +01:00
6lowpan.h
Space.h net: Space.h: Remove unused function declarations 2023-08-03 18:10:10 -07:00
act_api.h net/sched: Rename user cookie and act cookie 2023-02-20 16:46:10 -08:00
addrconf.h ipv6: constify inet6_mc_check() 2023-03-17 08:56:37 +00:00
af_ieee802154.h
af_rxrpc.h rxrpc: Fix timeout of a call that hasn't yet been granted a channel 2023-05-01 07:43:19 +01:00
af_unix.h af_unix: preserve const qualifier in unix_sk() 2023-03-18 12:23:33 +00:00
af_vsock.h af_vsock: Remove unused declaration vsock_release_pending()/vsock_init_tap() 2023-08-04 15:34:03 -07:00
ah.h
amt.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
arp.h neighbour: switch to standard rcu, instead of rcu_bh 2023-03-21 21:32:18 -07:00
atmclip.h
ax25.h x25: preserve const qualifier in [a]x25_sk() 2023-03-18 12:23:34 +00:00
ax88796.h ax88796: Fix some typo in a comment 2022-08-09 22:14:02 -07:00
bareudp.h bareudp: Move definition of struct bareudp_conf to bareudp.c 2021-12-13 12:34:09 +00:00
bond_3ad.h bonding: 3ad: Remove unused declaration bond_3ad_update_lacp_active() 2023-07-28 18:06:30 -07:00
bond_alb.h bonding (gcc13): synchronize bond_{a,t}lb_xmit() types 2022-11-02 20:38:13 -07:00
bond_options.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
bonding.h bonding: fix macvlan over alb bond support 2023-08-24 10:07:13 +02:00
bpf_sk_storage.h bpf: struct sock is declared twice in bpf_sk_storage header 2021-03-26 17:43:55 +01:00
busy_poll.h net: invert the netdevice.h vs xdp.h dependency 2023-08-03 08:38:07 -07:00
calipso.h
cfg80211-wext.h wifi: cfg80211: Avoid clashing function prototypes 2022-11-16 11:31:47 +02:00
cfg80211.h wifi: cfg80211: remove dead/unused enum value 2023-08-22 21:40:40 +02:00
cfg802154.h net: cfg802154: fix kernel-doc notation warnings 2023-07-14 20:39:29 -07:00
checksum.h net: checksum: drop the linux/uaccess.h include 2023-01-27 11:19:46 +00:00
cipso_ipv4.h
cls_cgroup.h
codel.h codel: fix kernel-doc notation warnings 2023-07-14 20:39:29 -07:00
codel_impl.h codel: remove unnecessary sock.h include 2021-12-22 15:03:47 -08:00
codel_qdisc.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
compat.h net: copy from user before calling __get_compat_msghdr 2022-07-24 18:39:17 -06:00
datalink.h net: datalink: Remove unused declarations 2023-07-27 17:17:32 -07:00
dcbevent.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
dcbnl.h net: dcb: add helper functions to retrieve PCP and DSCP rewrite maps 2023-01-20 09:33:22 +00:00
devlink.h devlink: Expose port function commands to control IPsec packet offloads 2023-08-27 17:08:45 -07:00
dropreason-core.h net: add skb_queue_purge_reason and __skb_queue_purge_reason 2023-08-19 15:30:15 +01:00
dropreason.h net: openvswitch: add last-action drop reason 2023-08-14 08:01:06 +01:00
dsa.h net: dsa: remove legacy_pre_march2020 detection 2023-07-18 09:47:08 +02:00
dsa_stubs.h net: dsa: replace NETDEV_PRE_CHANGE_HWTSTAMP notifier with a stub 2023-04-09 15:35:49 +01:00
dsfield.h
dst.h net: dst: remove unnecessary input parameter in dst_alloc and dst_init 2023-09-12 11:42:25 +02:00
dst_cache.h wireguard: device: reset peer src endpoint when netns exits 2021-11-29 19:50:45 -08:00
dst_metadata.h xfrm: interface: Add unstable helpers for setting/getting XFRM metadata from TC-BPF 2022-12-05 21:58:27 -08:00
dst_ops.h net: remove unnecessary input parameter 'how' in ifdown function 2023-08-22 13:19:02 +02:00
erspan.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
esp.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
espintcp.h
ethoc.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
failover.h net: failover: add net device refcount tracker 2021-12-06 16:06:02 -08:00
fib_notifier.h
fib_rules.h fib: expand fib_rule_policy 2021-12-16 07:18:35 -08:00
firewire.h firewire: net: Make use of get_unaligned_be48(), put_unaligned_be48() 2022-07-28 22:21:54 -07:00
flow.h ipv4: Drop tos parameter from flowi4_update_output() 2023-06-02 10:52:38 +01:00
flow_dissector.h net: flow_dissector: Add IPSEC dissector 2023-08-02 10:09:31 +01:00
flow_offload.h tc: flower: Enable offload support IPSEC SPI field. 2023-08-02 10:09:32 +01:00
fou.h bpf,fou: Add bpf_skb_{set,get}_fou_encap kfuncs 2023-04-12 16:40:39 -07:00
fq.h net: fq: Remove unused typedef fq_flow_get_default_t 2023-08-08 15:58:23 -07:00
fq_impl.h wifi: mac80211: add support for restricting netdev features per vif 2022-12-01 15:09:10 +01:00
garp.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
gen_stats.h net: sched: Remove Qdisc::running sequence counter 2021-10-18 12:54:41 +01:00
genetlink.h genetlink: add genlmsg_iput() API 2023-08-15 15:01:03 -07:00
geneve.h net: geneve: fix array of flexible structures warnings 2022-10-31 10:43:04 +00:00
gre.h ip_gre: add csum offload support for gre header 2021-01-29 20:39:14 -08:00
gro.h net: gro: fix misuse of CB in udp socket lookup 2023-07-29 17:10:27 +01:00
gro_cells.h
gso.h net: move gso declarations and functions to their own files 2023-06-10 00:11:41 -07:00
gtp.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
gue.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
handshake.h net/handshake: Add helpers for parsing incoming TLS Alerts 2023-07-28 14:07:59 -07:00
hwbm.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
icmp.h ipv6: ICMPV6: add response to ICMPV6 RFC 8335 PROBE messages 2021-06-28 14:29:45 -07:00
ieee80211_radiotap.h wifi: radiotap: fix kernel-doc notation warnings 2023-08-22 21:40:40 +02:00
ieee802154_netdev.h mac802154: Handle received BEACON_REQ 2023-03-23 21:51:30 +01:00
if_inet6.h ipv6: fix locking issues with loops over idev->addr_list 2022-04-06 22:09:39 -07:00
ife.h
inet6_connection_sock.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
inet6_hashtables.h net: Fix slab-out-of-bounds in inet[6]_steal_sock 2023-08-15 13:57:51 -07:00
inet_common.h net: factor out __inet_listen_sk() helper 2023-08-14 07:06:13 +01:00
inet_connection_sock.h inet: move inet->is_icsk to inet->inet_flags 2023-08-16 11:09:17 +01:00
inet_dscp.h ipv6: Define dscp_t and stop taking ECN bits into account in fib6-rules 2022-02-07 20:12:45 -08:00
inet_ecn.h net: add skb_get_dsfield() helper 2021-10-15 11:33:08 +01:00
inet_frag.h inet: frags: eliminate kernel-doc warning 2023-07-14 20:39:29 -07:00
inet_hashtables.h net: Fix slab-out-of-bounds in inet[6]_steal_sock 2023-08-15 13:57:51 -07:00
inet_sock.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-08-24 10:51:39 -07:00
inet_timewait_sock.h tcp: Add TIME_WAIT sockets in bhash2. 2022-12-30 07:25:52 +00:00
inetpeer.h
ioam6.h treewide: Replace zero-length arrays with flexible-array members 2022-02-17 07:00:39 -06:00
ip.h ipv4: ignore dst hint for multipath routes 2023-09-01 08:11:51 +01:00
ip6_checksum.h net: move gro definitions to include/net/gro.h 2021-11-16 13:16:54 +00:00
ip6_fib.h net: fib: avoid warn splat in flow dissector 2023-08-31 12:33:03 +02:00
ip6_route.h IPv6: add extack info for IPv6 address add/delete 2023-07-28 11:01:56 +01:00
ip6_tunnel.h ip_gre, ip6_gre: Fix race condition on o_seqno in collect_md mode 2022-04-25 11:40:45 +01:00
ip_fib.h net: fib: avoid warn splat in flow dissector 2023-08-31 12:33:03 +02:00
ip_tunnels.h ip_tunnels: use DEV_STATS_INC() 2023-09-06 06:05:59 +01:00
ip_vs.h ipvs: Correct spelling in comments 2023-04-22 01:39:41 +02:00
ipcomp.h xfrm: ipcomp: add extack to ipcomp{4,6}_init_state 2022-09-29 07:18:00 +02:00
ipconfig.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
ipv6.h sysctl-6.6-rc1 2023-08-29 17:39:15 -07:00
ipv6_frag.h net: dropreason: add SKB_DROP_REASON_FRAG_REASM_TIMEOUT 2022-10-31 20:14:27 -07:00
ipv6_stubs.h bpf: Change bpf_getsockopt(SOL_IPV6) to reuse do_ipv6_getsockopt() 2022-09-02 20:34:32 -07:00
iw_handler.h wifi: wext: Remove unused declaration dev_get_wireless_info() 2023-08-22 21:40:40 +02:00
kcm.h kcm: Send multiple frags in one sendmsg() 2023-06-12 21:13:23 -07:00
l3mdev.h
lag.h
lapb.h net: lapb: Make "lapb_t1timer_running" able to detect an already running timer 2021-03-23 14:14:50 -07:00
lib80211.h
llc.h llc: fix out-of-bound array index in llc_sk_dev_hash() 2021-11-07 19:25:29 +00:00
llc_c_ac.h net: llc: Remove unused function declarations 2023-08-04 15:33:17 -07:00
llc_c_ev.h net: llc: Remove unused function declarations 2023-08-04 15:33:17 -07:00
llc_c_st.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
llc_conn.h llc: Check netns in llc_estab_match() and llc_listener_match(). 2023-07-20 10:46:28 +02:00
llc_if.h llc/snap: constify dev_addr passing 2021-10-13 09:40:46 -07:00
llc_pdu.h net: llc: fix kernel-doc notation warnings 2023-07-14 20:39:29 -07:00
llc_s_ac.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
llc_s_ev.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
llc_s_st.h add missing includes and forward declarations to networking includes under linux/ 2022-07-28 11:29:36 +02:00
llc_sap.h
lwtunnel.h lwt: Check LWTUNNEL_XMIT_CONTINUE strictly 2023-08-18 16:05:26 +02:00
mac80211.h wireless-next patches for v6.6 2023-08-25 18:35:09 -07:00
mac802154.h mac802154: Drop IEEE802154_HW_RX_DROP_BAD_CKSUM 2022-10-12 12:57:19 +02:00
macsec.h macsec: add functions to get macsec real netdevice and check offload 2023-08-20 12:35:23 +03:00
mctp.h mctp: Reorder fields in 'struct mctp_route' 2023-06-20 20:06:16 -07:00
mctpdevice.h mctp: Pass flow data & flow release events to drivers 2021-10-29 13:23:51 +01:00
mip6.h
mld.h mld: add new workqueues for process mld events 2021-03-26 15:14:56 -07:00
mpls.h
mpls_iptunnel.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
mptcp.h mptcp: add struct mptcp_sched_ops 2023-08-22 17:31:18 -07:00
mrp.h mrp: introduce active flags to prevent UAF when applicant uninit 2022-11-18 12:14:55 +00:00
ncsi.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
ndisc.h ndisc: Remove unused ndisc_ifinfo_sysctl_strategy() declaration 2023-08-07 08:53:55 +01:00
neighbour.h neighbour: Remove unused function declaration pneigh_for_each() 2023-08-07 08:53:54 +01:00
net_debug.h net: add CONFIG_DEBUG_NET 2022-05-11 12:43:10 +01:00
net_failover.h
net_namespace.h sysctl-6.6-rc1 2023-08-29 17:39:15 -07:00
net_ratelimit.h
net_trackers.h net: add networking namespace refcount tracker 2021-12-10 06:38:26 -08:00
netdev_queues.h net: add macro netif_subqueue_completed_wake 2023-04-18 12:59:01 +02:00
netdev_rx_queue.h net: move struct netdev_rx_queue out of netdevice.h 2023-08-03 08:38:07 -07:00
netevent.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
netlabel.h
netlink.h netlink: allow be16 and be32 types in all uint policy checks 2023-07-27 13:45:51 +02:00
netprio_cgroup.h
netrom.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
nexthop.h ipv6: remove nexthop_fib6_nh_bh() 2023-05-11 18:07:05 -07:00
nl802154.h ieee802154: Add support for user beaconing requests 2023-01-28 13:51:22 +01:00
nsh.h net: NSH: fix kernel-doc notation warning 2023-07-14 20:39:29 -07:00
p8022.h net: 802: Remove unused function declarations 2023-08-04 15:33:50 -07:00
pie.h pie: fix kernel-doc notation warning 2023-07-14 20:39:30 -07:00
ping.h net/ipv4: ping_group_range: allow GID from 2147483648 to 4294967294 2023-06-02 09:55:22 +01:00
pkt_cls.h net: pkt_cls: Remove unused inline helpers 2023-08-07 08:53:54 +01:00
pkt_sched.h net/sched: make psched_mtu() RTNL-less safe 2023-07-12 15:59:33 -07:00
pptp.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
protocol.h tcp/udp: Make early_demux back namespacified. 2022-07-15 18:50:35 -07:00
psample.h psample: Add a fwd declaration for skbuff 2021-08-09 15:34:21 -07:00
psnap.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
raw.h Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2023-04-06 12:01:20 -07:00
rawv6.h ipv6: raw: constify raw_v6_match() socket argument 2023-03-17 08:56:37 +00:00
red.h treewide: use get_random_u32() when possible 2022-10-11 17:42:58 -06:00
regulatory.h wifi: cfg80211: fix regulatory disconnect with OCB/NAN 2023-06-19 12:05:29 +02:00
request_sock.h tcp: Use BPF timeout setting for SYN ACK RTO 2022-02-02 14:45:18 +00:00
rose.h net: rose: add netdev ref tracker to 'struct rose_sock' 2022-08-01 11:59:23 -07:00
route.h inet: move inet->transparent to inet->inet_flags 2023-08-16 11:09:17 +01:00
rpl.h ipv6: rpl: Remove pskb(_may)?_pull() in ipv6_rpl_srh_rcv(). 2023-06-19 11:32:58 -07:00
rsi_91x.h rsi: remove kernel-doc comment marker 2023-07-14 20:39:30 -07:00
rtnetlink.h net: validate veth and vxcan peer ifindexes 2023-08-20 11:40:03 +01:00
rtnh.h
sch_generic.h net/sched: wrap open coded Qdics class filter counter 2023-08-01 10:47:24 +02:00
scm.h af_unix: Fix msg_controllen test in scm_pidfd_recv() for MSG_CMSG_COMPAT. 2023-09-04 11:00:17 +01:00
secure_seq.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
seg6.h udp6: Use Segment Routing Header for dest address if present 2022-01-04 12:17:35 +00:00
seg6_hmac.h
seg6_local.h
selftests.h net: selftest: fix build issue if INET is disabled 2021-04-28 14:06:45 -07:00
slhc_vj.h
smc.h net/smc: Introduce explicit check for v2 support 2023-03-15 08:18:35 +00:00
snmp.h
sock.h net: sock_release_ownership() cleanup 2023-09-12 19:10:01 +02:00
sock_reuseport.h soreuseport: Fix socket selection for SO_INCOMING_CPU. 2022-10-25 11:35:16 +02:00
stp.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
strparser.h tls: rx: remove the message decrypted tracking 2022-07-18 11:24:10 +01:00
switchdev.h net: switchdev: Remove unused declaration switchdev_port_fwd_mark_set() 2023-08-09 13:12:15 -07:00
tc_wrapper.h net/sched: Retire rsvp classifier 2023-02-16 09:27:07 +01:00
tcp.h - Some swap cleanups from Ma Wupeng ("fix WARN_ON in add_to_avail_list") 2023-08-29 14:25:26 -07:00
tcp_states.h
tcx.h bpf: Add fd-based tcx multi-prog infra with link support 2023-07-19 10:07:27 -07:00
timewait_sock.h
tipc.h
tls.h tls: move tls_cipher_size_desc to net/tls/tls.h 2023-08-27 17:17:41 -07:00
tls_prot.h net/tls: Add TLS Alert definitions 2023-07-28 14:07:59 -07:00
tls_toe.h
transp_v6.h inet6: Remove unused function declaration udpv6_connect() 2023-08-01 15:06:27 -07:00
tso.h net: tso: inline tso_count_descs() 2022-12-12 15:04:39 -08:00
tun_proto.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
udp.h udp/udplite: Remove unused function declarations udp{,lite}_get_port() 2023-08-07 08:53:55 +01:00
udp_tunnel.h net: Change the udp encap_err_rcv to allow use of {ip,ipv6}_icmp_error() 2022-11-08 16:42:28 +00:00
udplite.h udp/udplite: Remove unused function declarations udp{,lite}_get_port() 2023-08-07 08:53:55 +01:00
vsock_addr.h
vxlan.h vxlan: Fix nexthop hash size 2023-08-02 10:58:26 +01:00
wext.h
x25.h x25: preserve const qualifier in [a]x25_sk() 2023-03-18 12:23:34 +00:00
x25device.h
xdp.h net: invert the netdevice.h vs xdp.h dependency 2023-08-03 08:38:07 -07:00
xdp_priv.h net: add missing includes and forward declarations under net/ 2022-07-22 12:53:22 +01:00
xdp_sock.h xsk: introduce wrappers and helpers for supporting multi-buffer in Tx path 2023-07-19 09:56:49 -07:00
xdp_sock_drv.h xsk: support mbuf on ZC RX 2023-07-19 09:56:49 -07:00
xfrm.h xfrm: don't skip free of empty state in acquire policy 2023-08-01 12:04:43 +02:00
xsk_buff_pool.h xsk: support mbuf on ZC RX 2023-07-19 09:56:49 -07:00