Overview

This article provides guidance for tuning the RX ring buffer size and socket buffer size on a storage node to resolve issues related to dropped packets. Adjusting these parameters can help accommodate high incoming traffic and reduce packet loss.

Prerequisites

• Access to the storage node with sufficient privileges to execute system commands.
• Basic understanding of Linux command-line operations.
• Ensure you have a backup of the current configuration before making changes.

Steps

Step 1: Check Current Network Settings

Begin by inspecting the current network settings for the interface ens3 to assess the RX ring buffer size and current packet drop rate.

• Open a terminal on the storage node.
• Run the following command:
  ip -s link show ens3
• Note the dropped packets count for analysis.

Step 2: Identify RX Ring Buffer Usage

Check if the RX ring buffer is full by using the tc command:

• Execute the following command:
  tc -s qdisc show dev ens3
• Look for the output indicating the backlog and drops values. If the backlog is close to the RX ring buffer size, it may indicate that the buffer is full.

Step 3: Review RX Ring Buffer Size

Next, review the current RX ring buffer size using the ethtool command.

• Execute the following command:
  ethtool -g ens3
• Observe the current RX ring buffer setting, which should display a value of 1024.
• Since the maximum preset value is 8192, consider increasing the RX ring buffer size.

Step 4: Increase RX Ring Buffer Size

To adjust the RX ring buffer size, follow these instructions:

• Increase the RX ring buffer to a more optimal size (e.g., 4096):
  sudo ethtool -G ens3 rx 4096
• Verify the new setting by running:
  ethtool -g ens3
• Confirm that the RX buffer size has been updated.

Step 5: Adjust Socket Buffer Sizes

Now, adjust the socket buffer sizes to accommodate higher incoming traffic.

• Open the /etc/sysctl.conf file in a text editor:
  sudo vi /etc/sysctl.conf
• Add or modify the following lines to set the maximum receive and send buffer sizes:
  net.core.rmem_max = 16777216 # 16 MB for receive buffer net.core.wmem_max = 16777216 # 16 MB for send buffer
• Save and close the file.

Step 6: Apply New Settings

After making changes to the /etc/sysctl.conf, apply the new settings with the following command:

Step 7: Monitor Packet Drops

Finally, monitor the interface for any further packet drops.

• Continuously check the network statistics:
  watch -n 5 ip -s link show ens3
• Observe the dropped packets count to ensure it decreases over time.

Additional Resources

• Ceph Documentation
• Linux Networking Documentation
• Ethtool Documentation