Security assessment of the transmission control protocol (tcp)

[Nytro]

SECURITY ASSESSMENT OF THE

TRANSMISSION CONTROL PROTOCOL (TCP)

Contents
1. Preface.................................................................................................. 5
1.1. Introduction ..................................................................................................................................... 5
1.2. Scope of this document .................................................................................................................. 6
1.3. Organisation of this document........................................................................................................ 7
1.4. Typographical conventions............................................................................................................. 7
1.5 Acknowledgements ......................................................................................................................... 7
1.6. Advice and guidance to vendors .................................................................................................... 8
2. The Transmission Control Protocol................................................... 9
3. TCP header fields .............................................................................. 10
3.1. Source Port................................................................................................................................... 10
3.2. Destination port............................................................................................................................. 18
3.3. Sequence number ........................................................................................................................ 19
3.4. Acknowledgement number........................................................................................................... 20
3.5. Data Offset.................................................................................................................................... 21
3.6. Control bits.................................................................................................................................... 21
3.7. Window ......................................................................................................................................... 25
3.8. Checksum..................................................................................................................................... 26
3.9. Urgent pointer............................................................................................................................... 27
3.10. Options ....................................................................................................................................... 31
3.11. Padding....................................................................................................................................... 33
3.12. Data ............................................................................................................................................ 33
4. Common TCP options....................................................................... 34
4.1. End of Option List (Kind = 0) ........................................................................................................ 34
4.2. No Operation (Kind = 1)................................................................................................................ 34
4.3. Maximum Segment Size (Kind = 2).............................................................................................. 34
4.4. Selective Acknowledgement option.............................................................................................. 36
4.5. MD5 option (Kind=19)................................................................................................................... 38
4.6. Window scale option (Kind = 3).................................................................................................... 39
4.7. Timestamps option (Kind = 8) ...................................................................................................... 40
5. Connection-establishment mechanism........................................... 43
5.1. SYN flood...................................................................................................................................... 43
5.2. Connection forgery ....................................................................................................................... 46
5.3. Connection-flooding attack ........................................................................................................... 47
5.4. Firewall-bypassing techniques ..................................................................................................... 49
3
6. Connection-termination mechanism ............................................... 50
6.1. FIN-WAIT-2 flooding attack .......................................................................................................... 50
7. Buffer management........................................................................... 53
7.1. TCP retransmission buffer............................................................................................................ 53
7.2. TCP segment reassembly buffer .................................................................................................. 56
7.3. Automatic buffer tuning mechanisms ........................................................................................... 58
8. TCP segment reassembly algorithm ............................................... 62
8.1. Problems that arise from ambiguity in the reassembly process................................................... 62
9. TCP congestion control .................................................................... 63
9.1. Congestion control with misbehaving receivers ........................................................................... 64
9.2. Blind DupACK triggering attacks against TCP ............................................................................. 66
9.3. TCP Explicit Congestion Notification (ECN)................................................................................. 79
10. TCP API ............................................................................................ 82
10.1 Passive opens and binding sockets ............................................................................................ 82
10.2. Active opens and binding sockets .............................................................................................. 83
11. Blind in-window attacks.................................................................. 84
11.1. Blind TCP-based connection-reset attacks ................................................................................ 84
11.2. Blind data-injection attacks......................................................................................................... 90
12. Information leaking ......................................................................... 91
12.1. Remote Operating System detection via TCP/IP stack fingerprinting........................................ 91
12.2. System uptime detection ............................................................................................................ 94
13. Covert channels............................................................................... 95
14. TCP port scanning........................................................................... 96
14.1. Traditional connect() scan .......................................................................................................... 96
14.2. SYN scan.................................................................................................................................... 96
14.3. FIN, NULL, and XMAS scans..................................................................................................... 97
14.4. Maimon scan .............................................................................................................................. 98
14.5. Window scan .............................................................................................................................. 98
14.6. ACK scan.................................................................................................................................... 98
4
15. Processing of ICMP error messages by TCP.............................. 100
15.1. Internet Control Message Protocol........................................................................................... 100
15.2. Handling of ICMP error messages ........................................................................................... 101
15.3 Constraints in the possible solutions ......................................................................................... 102
15.4. General countermeasures against ICMP attacks..................................................................... 103
15.5. Blind connection-reset attack ................................................................................................... 104
15.6. Blind throughput-reduction attack............................................................................................. 107
15.7. Blind performance-degrading attack ........................................................................................ 108
16. TCP interaction with the Internet Protocol (IP) ........................... 120
16.1. TCP-based traceroute .............................................................................................................. 120
16.2. Blind TCP data injection through fragmented IP traffic ............................................................ 120
16.3. Broadcast and multicast IP addresses ..................................................................................... 121
17. References ..................................................................................... 122

Download:

http://www.si6networks.com/publications/tn-03-09-security-assessment-TCP.pdf