Reducing Packet Drops on Storage Nodes by Tuning RX Ring Buffer and Socket Buffers

Overview

This article provides guidance for tuning the RX ring buffer size and socket buffer size on a storage node to resolve issues related to dropped packets. Adjusting these parameters can help accommodate high incoming traffic and reduce packet loss.

Prerequisites

• Access to the storage node with sufficient privileges to execute system commands.
• Basic understanding of Linux command-line operations.
• Ensure you have a backup of the current configuration before making changes.

Steps

Step 1: Check Current Network Settings

Begin by inspecting the current network settings for the interface ens3 to assess the RX ring buffer size and current packet drop rate.

• Open a terminal on the storage node.
• Run the following command:
  ip -s link show ens3
• Note the dropped packets count for analysis.

Step 2: Identify RX Ring Buffer Usage

Check if the RX ring buffer is full by using the tc command:

• Execute the following command:
  tc -s qdisc show dev ens3
• Look for the output indicating the backlog and drops values. If the backlog is close to the RX ring buffer size, it may indicate that the buffer is full.

Step 3: Review RX Ring Buffer Size

Next, review the current RX ring buffer size using the ethtool command.

• Execute the following command:
  ethtool -g ens3
• Observe the current RX ring buffer setting, which should display a value of 1024.
• Since the maximum preset value is 8192, consider increasing the RX ring buffer size.

Step 4: Increase RX Ring Buffer Size

To adjust the RX ring buffer size, follow these instructions:

• Increase the RX ring buffer to a more optimal size (e.g., 4096):
  sudo ethtool -G ens3 rx 4096
• Verify the new setting by running:
  ethtool -g ens3
• Confirm that the RX buffer size has been updated.

Step 5: Adjust Socket Buffer Sizes

Now, adjust the socket buffer sizes to accommodate higher incoming traffic.

• Open the /etc/sysctl.conf file in a text editor:
  sudo vi /etc/sysctl.conf
• Add or modify the following lines to set the maximum receive and send buffer sizes:
  net.core.rmem_max = 16777216 # 16 MB for receive buffer net.core.wmem_max = 16777216 # 16 MB for send buffer
• Save and close the file.

Step 6: Apply New Settings

After making changes to the /etc/sysctl.conf, apply the new settings with the following command:

Step 7: Monitor Packet Drops

Finally, monitor the interface for any further packet drops.

• Continuously check the network statistics:
  watch -n 5 ip -s link show ens3
• Observe the dropped packets count to ensure it decreases over time.

Additional Resources

• Ceph Documentation
• Linux Networking Documentation
• Ethtool Documentation