Interface redundancy on the host with TCP Multipath

TCP and UDP protocols are used exchange data between hosts. They have been used for a decade or longer and are very well documented how they work.

Everyone knows the problem that when you lost your active link on the server all your TCP sessions are going to die as well. Let's say your server has 2 active interfaces. There is no way to move/migrate a TCP session to use another active interface (by default). The other link can't be used automatically as a fail back mechanism.

There are couple of reasons behind why it isn't to works, the simplest one is that the new link used a different IP address. Even if the Linux kernel would start using the new interface and start sending IP/TCP packets sourced with the new IP address these packets wouldn't be recognized on the remote site. The remote site expect tcp segments from one and only one IP source.

Problem

How to provide a link level redundancy on the server to keep a TCP session alive even if one interface experience an error.

Analysis and solution Demonstration

The problem could be see as a more generic issue: how to implement multihoming or link redundancy. There are couple of working solution out there. The simplest example:

• Link bonding(link aggregation) on the server; requires support and proper configuration on the switch and the server

We will look at another one: TCP Multipath. What is cool about this is that it is transparent to your application. It visualizes a session and provide a single TCP session to the application that can benefit from built-in multipath redundancy on the kernel level.

References

http://multipath-tcp.org/
Decoupled from IP, TCP is at last able to support multihomed hosts
https://devcentral.f5.com/articles/multipath-tcp-mptcp
https://devcentral.f5.com/articles/the-evolution-of-tcp

Cisco ASA connection table state description and examples

On ASA in the connection table you can find protocol sessions (TCP, UDP, ICMP and others) that describe the state of the session (like TCP/IP) when the command was run.

In the session you can find all currently managed sessions by the ASA. From this output you can understand as well as from what IPs your clients are coming from and to what services they connect.

Session statutes

 
fw-asa# sh conn  

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
       B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,
       E - outside back connection, F - outside FIN, f - inside FIN,
       G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
       i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
       k - Skinny media, M - SMTP data, m - SIP media, n - GUP
       O - outbound data, P - inside back connection, p - Phone-proxy TFTP connection,
       q - SQL*Net data, R - outside acknowledged FIN,
       R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
       s - awaiting outside SYN, T - SIP, t - SIP transient, U - up,
       V - VPN orphan, W - WAAS,
       X - inspected by service module

Example flags meaning from the session entities

 
UB
U - up,
B - initial SYN from outside,

UO
U - up,
O - outbound data,

UIB
U - up,
I - inbound data,
B - initial SYN from outside,

UIOB
U - up,
I - inbound data,
O - outbound data,
B - initial SYN from outside,

UfIB
U - up,
f - inside FIN,
I - inbound data,
B - initial SYN from outside,

UfrO
U - up,
f - inside FIN,
r - inside acknowledged FIN,
O - outbound data,

UfIOB 
U - up,
f - inside FIN,
I - inbound data,
O - outbound data,
B - initial SYN from outside,

UfFIOB
 the same like UfIOB 
 F - outside FIN,

UfFRIOB
the same like UfFIOB
R - UDP SUNRPC,

UfrIOB
U - up,
f - inside FIN,
r - inside acknowledged FIN
I - inbound data,
O - outbound data,
B - initial SYN from outside,

SaAB
S - awaiting inside SYN,
a - awaiting outside ACK to SYN,
A - awaiting inside ACK to SYN, 
B - initial SYN from outside,

aB
a - awaiting outside ACK to SYN,
B - initial SYN from outside,

Example flow you can find in the ASA firewall connection table

Usually a lot entries with these state.

 
fw-asa# sh conn detail long

flags UfIOB TCP outside:1.165.177.125/1965 (1.165.177.125/1965) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 52m38s, uptime 54m21s, timeout 1h0m, bytes 3063
flags UfIOB TCP outside:1.172.130.64/1485 (1.172.130.64/1485) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 41m38s, uptime 43m12s, timeout 1h0m, bytes 3063

flags UB TCP outside:1.189.22.195/16208 (1.189.22.195/16208) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UB, idle 45m6s, uptime 48m17s, timeout 1h0m, bytes 0
flags UB TCP outside:1.56.45.22/24654 (1.56.45.22/24654) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UB, idle 45m54s, uptime 49m4s, timeout 1h0m, bytes 0

Common but less frequent state

 
flags UfFIOB TCP outside:1.55.216.14/14104 (1.55.216.14/14104) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFIOB, idle 41m51s, uptime 43m24s, timeout 1h0m, bytes 3002
flags UfFIOB TCP outside:110.81.84.50/20230 (110.81.84.50/20230) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFIOB, idle 52m55s, uptime 54m28s, timeout 1h0m, bytes 3063

flags UfFRIOB TCP outside:109.109.38.148/4760 (109.109.38.148/4760) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFRIOB, idle 3s, uptime 15s, timeout 5m0s, bytes 2261
flags UfFRIOB TCP outside:112.12.221.155/3753 (112.12.221.155/3753) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFRIOB, idle 0s, uptime 0s, timeout 5m0s, bytes 1008

flags UfIB TCP outside:121.35.47.128/1481 (121.35.47.128/1481) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIB, idle 23m54s, uptime 26m28s, timeout 1h0m, bytes 1106
flags UfIB TCP outside:183.11.2.56/4589 (183.11.2.56/4589) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIB, idle 47m15s, uptime 49m48s, timeout 1h0m, bytes 1106

flags SaAB TCP outside:112.72.135.224/7494 (112.72.135.224/7494) inside:192.168.55.172/4567 (1.2.157.172/4567), flags SaAB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
flags SaAB TCP outside:113.170.107.218/4472 (113.170.107.218/4472) inside:192.168.55.172/4567 (1.2.157.172/4567), flags SaAB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

flags UfrO TCP outside:202.168.215.226/80 (202.168.215.226/80) inside:192.168.55.172/3845 (1.2.157.172/3845), flags UfrO, idle 6s, uptime 8s, timeout 10m0s, bytes 1182

flags UIOB TCP outside:61.187.244.179/9571 (61.187.244.179/9571) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 38m13s, uptime 39m46s, timeout 1h0m, bytes 2897
flags UIOB TCP outside:67.47.251.34/14921 (67.47.251.34/14921) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 48m14s, uptime 49m50s, timeout 1h0m, bytes 3348

TCP outside:1.2.27.69/49856 (1.2.27.69/49856) FW-INSIDE:192.168.100.112/80 (11.22.192.112/80), flags UIB, idle 0s, uptime 0s, timeout 1h0m, bytes 581

flags UO TCP outside:202.168.215.226/80 (202.168.215.226/80) inside:192.168.55.172/3848 (1.2.157.172/3848), flags UO, idle 7s, uptime 7s, timeout 1h0m, bytes 1182

TCP outside:220.135.240.219/61139 (220.135.240.219/61139) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
TCP outside:220.135.240.219/61138 (220.135.240.219/61138) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

# without the 'long' parameter
TCP outside 94.5.94.11:59458 FW-DMZ-LB 192.168.67.79:80, idle 0:04:31, bytes 19424, flags UfrIOB
TCP outside 94.5.94.11:59463 FW-DMZ-LB 192.168.67.72:80, idle 0:04:05, bytes 7181, flags UfrIOB

You can specify additional parameters to filter output for specific connection entries state.

 
fw-asa# sh conn detail  long state tcp_embryonic all

TCP outside:220.135.240.219/61139 (220.135.240.219/61139) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
TCP outside:220.135.240.219/61138 (220.135.240.219/61138) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

fw-asa# sh conn long state data_out

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 0s, uptime 3m48s, timeout 1h0m, bytes 72509
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 1s, uptime 6m12s, timeout 1h0m, bytes 139249
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m37s, uptime 7m14s, timeout 1h0m, bytes 3002
TCP outside:171.118.104.53/64054 (171.118.104.53/64054) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 8s, uptime 7m27s, timeout 1h0m, bytes 98878
TCP outside:219.139.32.90/4141 (219.139.32.90/4141) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 7s, uptime 7m32s, timeout 1h0m, bytes 94113

fw-asa# sh conn long state data_in

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 4s, uptime 3m37s, timeout 1h0m, bytes 44907
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 1s, uptime 6m1s, timeout 1h0m, bytes 137801

fw-asa# sh conn long state finin

TCP outside:138.91.170.208/1264 (138.91.170.208/1264) inside:192.168.55.172/80 (1.2.157.172/80), flags UfFRIOB, idle 0s, uptime 0s, timeout 5m0s, bytes 5052
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 4m45s, uptime 6m21s, timeout 1h0m, bytes 3002
TCP outside:2.176.137.197/1653 (2.176.137.197/1653) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m6s, uptime 6m43s, timeout 1h0m, bytes 3002

fw-asa# sh conn long state up

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 0s, uptime 2m50s, timeout 1h0m, bytes 37914
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 4s, uptime 5m14s, timeout 1h0m, bytes 78789
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 4m39s, uptime 6m16s, timeout 1h0m, bytes 3002
TCP outside:171.118.104.53/64054 (171.118.104.53/64054) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 0s, uptime 6m29s, timeout 1h0m, bytes 89118
TCP outside:219.139.32.90/4141 (219.139.32.90/4141) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 9s, uptime 6m35s, timeout 1h0m, bytes 82689
TCP outside:2.176.137.197/1653 (2.176.137.197/1653) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m1s, uptime 6m38s, timeout 1h0m, bytes 3002
TCP outside:2.176.137.197/1589 (2.176.137.197/1589) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m13s, uptime 6m49s, timeout 1h0m, bytes 3002

Howto tune HTTP browser connection with Connection and Keep-Alive HTTP headers

I've come across this response today. This is diffident from all other I have seen so far. It uses the timeout and max options for the Keep-Alive header that should influences how the TCP connection should be handled.

 
HTTP/1.1 200 OK
Date: Wed, 10 Apr 2013 08:42:48 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 10 Apr 2013 08:42:48 GMT
Cache-Control: no-store,no-cache,must-revalidate
Expires: -1
Content-Length: 1567
Keep-Alive: timeout=5, max=10
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

Looking at the recent HTTP 1.1 RFC rfc2616 we can see that this options is addressed in more details here:

The original HTTP/1.0 form of persistent connections (the Connection: Keep-Alive and Keep-Alive header) is documented in RFC 2068.

Looking at RFC 2068 we see that it specifies in the paragraph  19.7.1.1 The Keep-Alive Header

The Keep-Alive header itself is optional, and is used only if a parameter is being sent. HTTP/1.1 does not define any parameters.

It doesn't specify anything about max or timeout parameters  With Google help I found this RFC draft Hypertext Transfer Protocol (HTTP) Keep-Alive Header. In this one we can see that:

The value of the timeout parameter is a single integer in seconds.
The value of the max parameter counts the number of requests since the connection was created.

How to forcibly kill an established TCP connection in Linux

During one of the repros I needed to find a way to kill an already established TCP session but without killing the process that opened it.

Problem

How to kill an ESTABLISHED TCP connection in Linux.

Solution #1: passive mechanism

We can use a little tool called tcpkill that you can find desription and example usage here link 

tcpkill -i eth0 { expression }

When started it first listens on a wire for any traffic matching your expression filter (that is compatible with tcpdump expressions). This is passive mechanism to learn about the TCP ACK and SEQ numbers. Once it knows the numbers it spoofs a TCP segments and try to reset the TCP session on both sides.

If the TCP session is idle or in generic there is no data being exchanged than this passive mechanism unfortunately is not going to work.

Solution #2: active mechanism

This killcx tool is taking a different approach and tries to generate traffic on the wire to discover what the ACK and SEQ numbers are. Once it forces the peer to replay to its spoofed traffic it kills the TCP session immediately after.

killcx.pl dest_ip:dest_port 

Simple demonstration

1. tcpkill (passive)

On 2 different hosts we simulate a daemon that actively listens on incoming traffic and a client that initiate a connection to a server. Once a TCP session is created the client sent single byte. The commands show the outputs.

root@server:~# nc -v -l 7777                                                                                       
Connection from 164.177.146.87 port 7777 [tcp/*] accepted
a

root@client:~# nc -D 5.79.21.166 7777
a
b

Below are tcpdumps showing TCP handshaking and 1 byte exchange on both hosts.

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:15:59.049739 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:15:59.049803 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:15:59.051394 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:16:00.815434 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:16:00.815493 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:20:38.159839 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [S], seq 2946797440, win 14600, options [mss 1460,sackOK,TS val 726987371 ecr 0,nop,wscale 2], length 0
23:20:38.174837 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [S.], seq 2303834803, ack 2946797441, win 14480, options [mss 1460,sackOK,TS val 350774705 ecr 726987371,nop,wscale 3], length 0
23:20:38.174880 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 726987375 ecr 350774705], length 0
23:20:39.938895 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 726987816 ecr 350774705], length 2
23:20:39.939313 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 3, win 1810, options [nop,nop,TS val 350775146 ecr 726987816], length 0 

At this point from another terminal on the server we can verify that the connection exists and start the tcpkill to try to kill it.

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      1338/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50185    ESTABLISHED 1338/nc
root@server:~# tcpkill -i eth0 -9 port 50185

At this stage nothing is going to happen because neither the client or server sending any data. The tcpkill simply hangs and waits. The open TCP session continue to exists. But as soon as we sent another byte from the client to the server these is being logged on the tcpdump sessions  for the tcpkill, client and server.

tcpkill

root@server:~# tcpkill -i eth0 -9 port 50185
tcpkill: listening on eth0 [port 50185]
164.177.146.87:50185 > 5.79.21.166:7777: R 2303834804:2303834804(0) win 0             
164.177.146.87:50185 > 5.79.21.166:7777: R 2303838454:2303838454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303845754:2303845754(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303856704:2303856704(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303871304:2303871304(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303889554:2303889554(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303911454:2303911454(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303937004:2303937004(0) win 0
164.177.146.87:50185 > 5.79.21.166:7777: R 2303966204:2303966204(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946797445:2946797445(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946799255:2946799255(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946802875:2946802875(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946808305:2946808305(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946815545:2946815545(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946824595:2946824595(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946835455:2946835455(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946848125:2946848125(0) win 0
5.79.21.166:7777 > 164.177.146.87:50185: R 2946862605:2946862605(0) win 0

client

root@client:~# tcpdump -i any -nn port 7777
...
23:21:03.938535 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:21:03.943047 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:21:03.943070 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:21:03.943082 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:21:03.943086 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:21:03.943090 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:21:03.943093 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:21:03.943096 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:21:03.943099 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:21:03.943102 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0 
23:21:03.943106 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0

Server

root@server:~# tcpdump -i any -nn port 7777
...
23:16:24.815402 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [P.], seq 3:5, ack 1, win 3650, options [nop,nop,TS val 726993816 ecr 350775146], length 2
23:16:24.815449 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [.], ack 5, win 1810, options [nop,nop,TS val 350781146 ecr 726993816], length 0
23:16:24.816040 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303834804, win 0, length 0
23:16:24.816080 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303838454, win 0, length 0
23:16:24.816103 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303845754, win 0, length 0
23:16:24.816130 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303856704, win 0, length 0
23:16:24.816153 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303871304, win 0, length 0
23:16:24.816177 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303889554, win 0, length 0
23:16:24.816199 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303911454, win 0, length 0
23:16:24.816225 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303937004, win 0, length 0
23:16:24.816245 IP 5.79.21.166.7777 > 164.177.146.87.50185: Flags [R], seq 2303966204, win 0, length 0
23:16:24.816280 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946797445, win 0, length 0
23:16:24.816314 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946799255, win 0, length 0
23:16:24.816339 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946802875, win 0, length 0
23:16:24.816362 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946808305, win 0, length 0
23:16:24.816384 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946815545, win 0, length 0
23:16:24.816407 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946824595, win 0, length 0
23:16:24.816429 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946835455, win 0, length 0
23:16:24.816451 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946848125, win 0, length 0
23:16:24.816474 IP 164.177.146.87.50185 > 5.79.21.166.7777: Flags [R], seq 2946862605, win 0, length 0

We can see that as soon as the new byte generate 2 TCP segment the tcpkill learns about the SEQ and ACK numbers and generate its RST packet to desynchronize and kill the TCP session.

1. killcx (active)

We start the 2 client and server sessions in a similar way. These are the command output showing what is happening on the wire this time.

server

root@server:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:46:06.228566 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:46:06.228625 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:46:06.233328 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:46:07.960259 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:46:07.960322 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351226932 ecr 727439605], length 0
# at this moment the TCP is estabilished and the server received one byte

# we see this as soon as the killcx is started
23:46:32.411167 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 10, win 65535, length 0
23:46:32.411194 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:46:32.413315 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [R], seq 11257297, win 65535, length 0
23:46:32.414945 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

client

root@client:~# tcpdump -i any -nn port 7777
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
23:50:45.365610 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [S], seq 11257294, win 14600, options [mss 1460,sackOK,TS val 727439173 ecr 0,nop,wscale 2], length 0
23:50:45.370015 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [S.], seq 1319656020, ack 11257295, win 14480, options [mss 1460,sackOK,TS val 351226499 ecr 727439173,nop,wscale 3], length 0
23:50:45.370059 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [.], ack 1, win 3650, options [nop,nop,TS val 727439174 ecr 351226499], length 0
23:50:47.097391 IP 164.177.146.87.50189 > 5.79.21.166.7777: Flags [P.], seq 1:3, ack 1, win 3650, options [nop,nop,TS val 727439605 ecr 351226499], length 2
23:50:47.098888 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 35122693221:20:14439605], length 0
# at this moment the TCP is estabilished and the client sent one byte

# we see this as soon as the killcx is started
23:51:11.550122 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [.], ack 3, win 1810, options [nop,nop,TS val 351233045 ecr 727439605,nop,nop,sack 1 {4283710012:4283710013}], length 0
23:51:11.552854 IP 5.79.21.166.7777 > 164.177.146.87.50189: Flags [R], seq 1319656021, win 65535, length 0

killcx

root@server:~# netstat -tulpan | grep 7777
tcp        0      0 0.0.0.0:7777            0.0.0.0:*               LISTEN      10568/nc
tcp        0      0 5.79.21.166:7777        164.177.146.87:50189    ESTABLISHED 10568/nc

root@manage2:~# ./killcx.pl 164.177.146.87:50189
killcx v1.0.3 - (c)2009-2011 Jerome Bruandet - http://killcx.sourceforge.net/

[PARENT] checking connection with [164.177.146.87:50189]
[PARENT] found connection with [5.79.21.166:7777] (ESTABLISHED)
[PARENT] forking child
[CHILD]  interface not defined, will use [eth0]
[CHILD]  setting up filter to sniff ACK on [eth0] for 5 seconds
[PARENT] sending spoofed SYN to [5.79.21.166:7777] with bogus SeqNum
[CHILD]  hooked ACK from [5.79.21.166:7777]
[CHILD]  found AckNum [11257297] and SeqNum [1319656021]
[CHILD]  sending spoofed RST to [5.79.21.166:7777] with SeqNum [11257297]
[CHILD]  sending RST to remote host as well with SeqNum [1319656021]
[CHILD]  all done, sending USR1 signal to parent [10726] and exiting
[PARENT] received child signal, checking results...
         => success : connection has been closed !

As before the TCP session is established and idle waiting. As soon as we start the killcx it successfully spoofs a SYN packet on behalf of the client and sent it to the server. The server than replays with a valid TCP packet revealing the ACK and SEQ numbers. As soon as this this is on a wire the killcx sniffs this up and sent RST to kill the active session.

References

1. http://killcx.sourceforge.net/
2. http://www.cyberciti.biz/howto/question/linux/kill-tcp-connection-using-linux-netstat.php

Further reading
3. http://serverfault.com/questions/179702/is-it-possible-to-close-a-socket-with-a-shell-command
4. http://superuser.com/questions/127863/manually-closing-a-port-from-commandline