Linux. Network Tuning

Tuning of the network options prevents data loss and maximum bandwidth.

System buffer

The buffer size options should be defined in the /etc/sysctl.conf file. Recommended to use next values for 1G ethernet adapters:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.udp_mem = 8388608 12582912 16777216
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 65536 8388608
net.core.wmem_default = 16777216
net.core.rmem_default = 16777216
net.ipv4.tcp_tw_recycle = 0

For 10G ethernet adapters:

net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.udp_mem = 8388608 16777216 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.core.wmem_default = 33554432
net.core.rmem_default = 33554432
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_tw_recycle = 0

For 40G ethernet adapters:

net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.udp_mem = 8388608 33554432 67108864
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.wmem_default = 67108864
net.core.rmem_default = 67108864
net.ipv4.tcp_congestion_control=htcp
net.ipv4.tcp_tw_recycle = 0

To apply changes restart system or launch:

sysctl -p

You can verify the current values with next command:

sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default net.core.wmem_max net.ipv4.udp_mem net.ipv4.tcp_wmem

The size of the buffer of the network card

[root@astra ~]# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:		4096
RX Mini:	0
RX Jumbo:	0
TX:		4096
Current hardware settings:
RX:		4096
RX Mini:	0
RX Jumbo:	0
TX:		256

Here we can see the rx-buffer increased by the maximum. Usually it is quite difficult to find the value. The most optimal is some "average" value. With a high-frequency and multi-core processor (>3GHz), you can get closer to the maximum/maximum buffer. Example of a command to increase the buffer:

ethtool -G eth1 rx 2048

How to check information about losses

Keywords words: missed, dropped, fifo, error, rx.

ip -s -s link show eth1

You need to look at RX Errors. Some network cards provide more detailed information about the nature of the loss:

ethtool -S eth1

Losses can be not only on the network cards of your server. They can also be on the network equipment port. You can learn how to see it from the documentation of the network equipment manufacturer.

rp_filter

rp_filter - is a technique for the purposes of ensuring loop-free forwarding of multicast packets in multicast routing.

If your server has several network interfaces, recommend to set routes for multicast groups. If this is not applicable and interface is defined in the source or destination address you should disable rp_filter.

Append into the /etc/sysctl.conf file next lines:

net.ipv4.conf.eth0.rp_filter = 2

First two line is identical for any servers. Third line is depend of the interfaces names. Append lines, like a third, for each interface and replace eth0 with the name of interfaces.

To apply changes restart system or launch:

sysctl -p