Search the Community

Tuning the initial congestion window parameter (initcwnd) on the server can have a significant improvement in TCP performance, resulting in faster downloads and faster webpages. In this article, I will start with an introduction to how TCP/IP connections work with regards to HTTP. Then I will go into TCP slow start and show how tuning the initcwnd setting on the (Linux) server can greatly improve page performance. In our follow-up article we show data on the value of the initcwnd setting for the various CDNs: Initcwnd settings of major CDN providers. Three-way handshake Imagine a client wants to request the webpage Example Domain from a server. Here is an over simplified version of the transaction between client and server. The requested page is 6 KB and we assume there is no overhead on the server to generate the page (e.g. it's static content cached in memory) or any other overhead: we live in an ideal world ;-) Step 1: Client sends SYN to server - "How are you? My receive window is 65,535 bytes." Step 2: Server sends SYN, ACK - "Great! How are you? My receive window is 4,236 bytes" Step 3: Client sends ACK, SEQ - "Great as well... Please send me the webpage http://www.example.com/" Step 4: Server sends 3 data packets. Roughly 4 - 4.3 kb (3*MSS1) of data Step 5: Client acknowledges the segment (sends ACK) Step 6: Server sends the remaining bytes to the client 1. MSS = Maximum Segment Size After step 6 the connection can be ended (FIN) or kept alive, but that is irrelevant here, since at this point the browser has already received the data. The above transaction took 3*RTT (Round Trip Time) to finish. If your RTT to a server is 200ms this transaction will take you at least 600ms to complete, no matter how big your bandwidth is. The bigger the file, the more round trips and the longer it takes to download. Congestion control/TCP Slow Start As illustrated in the video and as you have seen in our example transaction in the section above, a server does not necessarily adhere to the client's RWIN (receivers advertised window size). The client told the server it can receive a maximum of 65,535 bytes of un-acknowledged data (before ACK), but the server only sent about 4 KB and then waited for ACK. This is because the initial congestion window (initcwnd) on the server is set to 3. The server is being cautious. Rather than throw a burst of packets into a fresh connection, the server chooses to ease into it gradually, making sure that the entire network route is not congested. The more congested a network is, the higher is the chances for packet loss. Packet loss results in retransmissions which means more round trips, resulting in higher download times. Basically, there are 2 main parameters that affect the amount of data the server can send at the start of a connection: the receivers advertised window size (RWIN) and the value of the initcwnd setting on the server. The initial transfer size will be the lower of the 2, so if the initcwnd value on the server is a lot lower than the RWIN on the computer of the user, the initial transfer size is less then optimal (assuming no network congestion). It is easy to change the initcwnd setting on the server, but not the RWIN. Different OSes have different RWIN settings, as shown in the table below. OS RWIN Linux 2.6.32 3*MSS (usually 5,840) Linux 3.0.0 10*MSS (usually 14,600) Windows NT 5.1 (XP) 65,535^2 Windows NT 6.1 (Windows 7 or Server 2008 R2) 8,192^2 Mac OS X 10.5.8 (Leopard) 65,535^2 Mac OS X 10.6.8 (Snow Leopard) 65,535^2 Apple IOS 4.1 65,535^2 Apple IOS 5.1 65,535^2 2. Some Operating Systems dynamically calculate RWIN based on external factors. The value here is based on SYN packets sent to CDN Planet. The Win flag can also be increased by the client before the transfer actually starts. A you can see from the table, Windows and Mac users would benefit most from servers sending more bytes in the initial transfer (which is almost everybody!) Changing initcwnd Adjusting the value of the initcwnd setting on Linux is simple. Assuming we want to set it to 10: Step 1: check route settings. sajal@sajal-desktop:~$ ip route show 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100 metric 1 169.254.0.0/16 dev eth0 scope link metric 1000 default via 192.168.1.1 dev eth0 proto static sajal@sajal-desktop:~$ Make a note of the line starting with default. Step 2: Change the default settings. Paste the current settings for default and add initcwnd 10 to it. sajal@sajal-desktop:~$ sudo ip route change default via 192.168.1.1 dev eth0 proto static initcwnd 10 Step 3: Verify sajal@sajal-desktop:~$ ip route show 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100 metric 1 169.254.0.0/16 dev eth0 scope link metric 1000 default via 192.168.1.1 dev eth0 proto static initcwnd 10 sajal@sajal-desktop:~$ Results The entire transaction now happened in 400ms, rather than 600ms. That is a big win and this is just for one HTTP request. Modern browsers open 6 connections to a host which means you will see a faster load time for 6 objects Here is a before and after comparison of accessing a 14 KB file (13,516 bytes transfered ~10*MSS) from across the globe with different settings. It is clear that when initcwnd is larger than the payload size, the entire transaction happens in just 2*RTT. The graph shows that the total load time of this object was reduced by ~50% by increasing initcwnd to 10. A great performance gain! Interested in more info and insight on tuning initcwnd? Read Google's paper An Argument for Increasing TCP's Initial Congestion Window. It's a great resource. Source: Tuning initcwnd for optimum performance - CDN Planet