Linux. Network Tuning

    Tuning of the network options prevents data loss and maximum bandwidth.

    System buffer

    The buffer size options should be defined in the /etc/sysctl.conf file. Recommended to use next values for 1G ethernet adapters:

    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.udp_mem = 8388608 12582912 16777216
    net.ipv4.tcp_rmem = 4096 87380 8388608
    net.ipv4.tcp_wmem = 4096 65536 8388608
    net.core.wmem_default = 16777216
    net.core.rmem_default = 16777216
    net.ipv4.tcp_tw_recycle = 0
    

    For 10G ethernet adapters:

    net.core.rmem_max = 67108864
    net.core.wmem_max = 67108864
    net.ipv4.udp_mem = 8388608 16777216 33554432
    net.ipv4.tcp_rmem = 4096 87380 33554432
    net.ipv4.tcp_wmem = 4096 65536 33554432
    net.core.wmem_default = 33554432
    net.core.rmem_default = 33554432
    net.ipv4.tcp_congestion_control=htcp
    net.ipv4.tcp_tw_recycle = 0
    

    For 40G ethernet adapters:

    net.core.rmem_max = 134217728
    net.core.wmem_max = 134217728
    net.ipv4.udp_mem = 8388608 33554432 67108864
    net.ipv4.tcp_rmem = 4096 87380 67108864
    net.ipv4.tcp_wmem = 4096 65536 67108864
    net.core.wmem_default = 67108864
    net.core.rmem_default = 67108864
    net.ipv4.tcp_congestion_control=htcp
    net.ipv4.tcp_tw_recycle = 0
    

    To apply changes restart system or launch:

    sysctl -p
    

    You can verify the current values with next command:

    sysctl net.core.rmem_default net.core.rmem_max net.core.wmem_default net.core.wmem_max net.ipv4.udp_mem net.ipv4.tcp_wmem
    

    The size of the buffer of the network card

    [root@astra ~]# ethtool -g eth1
    Ring parameters for eth1:
    Pre-set maximums:
    RX:		4096
    RX Mini:	0
    RX Jumbo:	0
    TX:		4096
    Current hardware settings:
    RX:		4096
    RX Mini:	0
    RX Jumbo:	0
    TX:		256
    

    Here we can see the rx-buffer increased by the maximum. Usually it is quite difficult to find the value. The most optimal is some "average" value. With a high-frequency and multi-core processor (>3GHz), you can get closer to the maximum/maximum buffer. Example of a command to increase the buffer:

    ethtool -G eth1 rx 2048
    

    How to check information about losses

    Keywords words: missed, dropped, fifo, error, rx.

    ip -s -s link show eth1
    

    You need to look at RX Errors. Some network cards provide more detailed information about the nature of the loss:

    ethtool -S eth1
    

    Losses can be not only on the network cards of your server. They can also be on the network equipment port. You can learn how to see it from the documentation of the network equipment manufacturer.

    rp_filter

    rp_filter - is a technique for the purposes of ensuring loop-free forwarding of multicast packets in multicast routing.

    If your server has several network interfaces, recommend to set routes for multicast groups. If this is not applicable and interface is defined in the source or destination address you should disable rp_filter.

    Append into the /etc/sysctl.conf file next lines:

    net.ipv4.conf.eth0.rp_filter = 2
    

    First two line is identical for any servers. Third line is depend of the interfaces names. Append lines, like a third, for each interface and replace eth0 with the name of interfaces.

    To apply changes restart system or launch:

    sysctl -p