Hardening the TCP/IP stack to SYN attacks

[aelius]

Most people know how problematic protection against SYN denial of service attacks can be. Several methods, more or less effective, are usually used. In almost every case proper filtering of packets is a viable solution. In addition to creating packet filters, the modification of the TCP/IP stack of a given operating system can be performed by an administrator. This method, the tuning of the TCP/IP stack in various operating systems, will be described in depth in this article.

While SYN attacks may not be entirely preventable, tuning the TCP/IP stack will help reduce the impact of SYN attacks while still allowing legitimate client traffic through. It should be noted that some SYN attacks do not always attempt to upset servers, but instead try to consume all of the bandwidth of your Internet connection. This kind of flood is outside the scope of scope of this article, as is the filtering of packets which has been discussed elsewhere.

What can an administrator do when his servers are under a classic, non-bandwidth flooding SYN attack? One of most important steps is to enable the operating system's built-in protection mechanisms like SYN cookies or SynAttackProtect. Additionally, in some cases it is worth tuning parameters of the TCP/IP stack. Changing the default values of stack variables can be another layer of protection and help better secure your hosts. In this paper I will concentrate on:

- Increasing the queue of half-open connections (in the SYN RECEIVED state).

- Decreasing the time period of keeping a pending connection in the SYN RECEIVED state in the queue. This method is accomplished by decreasing the time of the first packet retransmission and by either decreasing the number of packet retransmissions or by turning off packet retransmissions entirely. The process of packet retransmissions is performed by a server when it doesn't receive an ACK packet from a client. A Packet with the ACK flag finalizes the process of the three-way handshake.

Note that an attacker can simply send more packets with the SYN flag set and then the above tasks will not solve the problem. However, we can still increase the likelihood of creating a full connection with legitimate clients by performing the above operations.

We should remember that our modification of variables will change the behavior of the TCP/IP stack. In some cases the values can be too strict. So, after the modification we have to make sure that our server can properly communicate with other hosts. For example, the disabling of packet retransmissions in some environments with low bandwidth can cause a legitimate request to fail. In this article you will find a description of the TCP/IP variables for the fallowing operating systems: Microsoft Windows 2000, RedHat Linux 7.3, Sun Solaris 8 and HP-UX 11.00. These variables are similar or the same in current releases.

Definitions: SYN flooding and SYN spoofing

A SYN flood is a type of Denial of Service attack. We can say that a victim host is under a SYN flooding attack when an attacker tries to create a huge amount of connections in the SYN RECEIVED state until the backlog queue has overflowed. The SYN RECEIVED state is created when the victim host receives a connection request (a packet with SYN flag set) and allocates for it some memory resources. A SYN flood attack creates so many half-open connections that the system becomes overwhelmed and cannot handle incoming requests any more.

To increase an effectiveness of a SYN flood attack, an attacker spoofs source IP addresses of SYN packets. In this case the victim host cannot finish the initialization process in a short time because the source IP address can be unreachable. This malicious operation is called a SYN spoofing attack.

We need to know that the process of creating a full connection takes some time. Initially, after receiving a connection request (a packet with SYN flag set), a victim host puts this half-open connection to the backlog queue and sends out the first response (a packet with SYN and ACK flags set). When the victim does not receive a response from a remote host, it tries to retransmit this SYN+ACK packet until it times out, and then finally removes this half-open connection from the backlog queue. In some operating systems this process for a single SYN request can take about 3 minutes! In this document you will learn how to change this behavior. The other important information you need to know is that the operating system can handle only a defined amount of half-open connections in the backlog queue. This amount is controlled by the size of the backlog queue. For instance, the default backlog size is 256 for RedHat 7.3 and 100 for Windows 2000 Professional. When this size is reached, the system will no longer accept incoming connection requests.

Read more: Hardening the TCP/IP stack to SYN attacks | Symantec Connect Community