Search This Blog

Wednesday, March 28, 2012

Network data analize for a scenario with two Cloud Server behind a Rackspace Cloud Load Balancer

In this post we are going to take a look at the Rackspace Cloud Load Balancer (CLB) again but this time we aim to analyze a scenario with persistence issues and 2 pool member. Particularly, we want to find out and understand how the session persistence works on CLB. The simple diagram bellow shows our small test cloud topology. We have 3 Cloud Servers (CS) (1 client + 2 pool members) and one load balancer (LB). Our LB is using Round Robin algorithm to distribute the load amount the pool members. The pool members are defined using the internal IP addresses: 10.177.132.15  and 10.177.133.12.



Test scenario #1 without session persistence
  1. The client sends 1st HTTP request to LB1.
  2. The LB1 sends the req to first pool member according to round robin state.
  3. The pool member replays with HTTP 200.
  4. LB1 sends the replay back to the client.
  5. The client sends 2th HTTP request to LB1.
  6. The LB1 sends the req to another pool member.
  7. The pool member replays with HTTP 200.
  8. LB1 sends the replay back to the client.
Because we don't have access to the LB1 we are going to collect concurrent tcpdumps from all CS. We are using the curl to simulate HTTP requests on the client.

# run on client
tcpdump -nn -s0 -i any -w /var/tmp/client.pcap  port 80

# run on server1
tcpdump -nn -s0 -i any -w /var/tmp/server-urado1.pcap  port 80

# run on server2
tcpdump -nn -s0 -i any -w /var/tmp/server-urado2.pcap  port 80

# run on client to simulate HTTP requests
curl -v http://31.222.175.142

The first dumps and data from the test can be seen and found bellow [1].

[root@crado1 tmp]# tshark  -n -r client.pcap
  1   0.000000 31.222.191.246 -> 31.222.175.142 TCP 60925 > 80 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=25369178 TSER=0 WS=4
  2   0.000513 31.222.175.142 -> 31.222.191.246 TCP 80 > 60925 [SYN, ACK] Seq=0 Ack=1 Win=17896 Len=0 MSS=8960 TSV=1091232983 TSER=25369178 WS=9
  3   0.000541 31.222.191.246 -> 31.222.175.142 TCP 60925 > 80 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=25369178 TSER=1091232983
  4   0.000609 31.222.191.246 -> 31.222.175.142 HTTP GET / HTTP/1.1
  5   0.000841 31.222.175.142 -> 31.222.191.246 TCP 80 > 60925 [ACK] Seq=1 Ack=159 Win=19456 Len=0 TSV=1091232983 TSER=25369178
  6   0.008348 31.222.175.142 -> 31.222.191.246 HTTP HTTP/1.1 200 OK  (text/html)
  7   0.008369 31.222.191.246 -> 31.222.175.142 TCP 60925 > 80 [ACK] Seq=159 Ack=301 Win=6912 Len=0 TSV=25369180 TSER=1091232983
  8   0.008651 31.222.191.246 -> 31.222.175.142 TCP 60925 > 80 [FIN, ACK] Seq=159 Ack=301 Win=6912 Len=0 TSV=25369180 TSER=1091232983
  9   0.008841 31.222.175.142 -> 31.222.191.246 TCP 80 > 60925 [FIN, ACK] Seq=301 Ack=160 Win=19456 Len=0 TSV=1091232983 TSER=25369180
 10   0.008855 31.222.191.246 -> 31.222.175.142 TCP 60925 > 80 [ACK] Seq=160 Ack=302 Win=6912 Len=0 TSV=25369180 TSER=1091232983
 11   2.182732 31.222.191.246 -> 31.222.175.142 TCP 60926 > 80 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=25369723 TSER=0 WS=4
 12   2.183220 31.222.175.142 -> 31.222.191.246 TCP 80 > 60926 [SYN, ACK] Seq=0 Ack=1 Win=17896 Len=0 MSS=8960 TSV=1091233201 TSER=25369723 WS=9
 13   2.183240 31.222.191.246 -> 31.222.175.142 TCP 60926 > 80 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=25369723 TSER=1091233201
 14   2.183496 31.222.191.246 -> 31.222.175.142 HTTP GET / HTTP/1.1
 15   2.183658 31.222.175.142 -> 31.222.191.246 TCP 80 > 60926 [ACK] Seq=1 Ack=159 Win=19456 Len=0 TSV=1091233201 TSER=25369724
 16   2.186699 31.222.175.142 -> 31.222.191.246 HTTP HTTP/1.1 200 OK  (text/html)
 17   2.186717 31.222.191.246 -> 31.222.175.142 TCP 60926 > 80 [ACK] Seq=159 Ack=301 Win=6912 Len=0 TSV=25369724 TSER=1091233201
 18   2.188335 31.222.191.246 -> 31.222.175.142 TCP 60926 > 80 [FIN, ACK] Seq=159 Ack=301 Win=6912 Len=0 TSV=25369725 TSER=1091233201
 19   2.188756 31.222.175.142 -> 31.222.191.246 TCP 80 > 60926 [FIN, ACK] Seq=301 Ack=160 Win=19456 Len=0 TSV=1091233201 TSER=25369725
 20   2.188771 31.222.191.246 -> 31.222.175.142 TCP 60926 > 80 [ACK] Seq=160 Ack=302 Win=6912 Len=0 TSV=25369725 TSER=1091233201

We can see that from the client point of view the data comes always from the some IP address. The 2 pool members that we have are not visible.

The dumps bellow show as well that the responses came from different servers. We can confirm as well that neither the req or replay had a cookies header that was passed to the client.

root@urado1:/var/tmp# tshark -n -r server-urado1.pcap -V http
    GET / HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n]
            [Message: GET / HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /
        Request Version: HTTP/1.1
    User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5\r\n
    X-Forwarded-For: 31.222.191.246\r\n
    Accept: */*\r\n
    X-Forwarded-Proto: http\r\n
    Host: 31.222.175.142\r\n
    X-Cluster-Client-Ip: 31.222.191.246\r\n
    \r\n

    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [Message: HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 200
        Response Phrase: OK
    Date: Tue, 27 Mar 2012 22:37:17 GMT\r\n
    Server: Apache/2.2.20 (Ubuntu)\r\n
    Last-Modified: Mon, 26 Mar 2012 21:29:38 GMT\r\n
    ETag: "7c030-2c-4bc2c1245f480"\r\n
    Accept-Ranges: bytes\r\n
    Content-Length: 44\r\n
        [Content length: 44]
    Vary: Accept-Encoding\r\n
    Content-Type: text/html\r\n
    \r\n
Line-based text data: text/html
    It works! urado1\n
    \n            



root@urado2:/var/tmp# tshark -n -r server-urado2.pcap -V http
    GET / HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n]
            [Message: GET / HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /
        Request Version: HTTP/1.1
    User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5\r\n
    X-Forwarded-For: 31.222.191.246\r\n
    Accept: */*\r\n
    X-Forwarded-Proto: http\r\n
    Host: 31.222.175.142\r\n
    X-Cluster-Client-Ip: 31.222.191.246\r\n
    \r\n

    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [Message: HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Version: HTTP/1.1
        Status Code: 200
        Response Phrase: OK
    Date: Tue, 27 Mar 2012 22:37:21 GMT\r\n
    Server: Apache/2.2.20 (Ubuntu)\r\n
    Last-Modified: Mon, 26 Mar 2012 21:30:02 GMT\r\n
    ETag: "5c033-2c-4bc2c13b42a80"\r\n
    Accept-Ranges: bytes\r\n
    Content-Length: 44\r\n
        [Content length: 44]
    Vary: Accept-Encoding\r\n
    Content-Type: text/html\r\n
    \r\n
Line-based text data: text/html
    It works! urado2\n
    \n

Test scenario #2 with session persistence enabled on LB1

The test steps are the same as the one before. The only difference is a configuration change on LB1, where we enable persistence feature. The new tcpdumps were taken as before and can be downloaded from [2]. The easy way to see the differences without repeating ourselves again is to take a look at the curl output. We see directly there all our requests and replays with the HTML payload from the servers.

First GET request without Cookie header. In the response we see this time that the LB is sending one cookie we didn't see before. 

[root@crado1 ~]# curl -v http://31.222.175.142
* About to connect() to 31.222.175.142 port 80
*   Trying 31.222.175.142... connected
* Connected to 31.222.175.142 (31.222.175.142) port 80
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: 31.222.175.142
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache/2.2.20 (Ubuntu)
< Vary: Accept-Encoding
< Content-Type: text/html
< Date: Tue, 27 Mar 2012 22:57:56 GMT
< Accept-Ranges: bytes
< ETag: "7c030-2c-4bc2c1245f480"
< Set-Cookie: X-Mapping-fjhppofk=AEC8609A6667F8E6AC1B323F80BDF8C9; path=/
< Last-Modified: Mon, 26 Mar 2012 21:29:38 GMT
< Content-Length: 44
It works! urado1

* Connection #0 to host 31.222.175.142 left intact
* Closing connection #0

Another similar request. The response contains again Cookie header but with different value. The HTML payload from the 2 requests show that these are data from CS1 and CS2. 

[root@crado1 ~]# curl -v http://31.222.175.142
* About to connect() to 31.222.175.142 port 80
*   Trying 31.222.175.142... connected
* Connected to 31.222.175.142 (31.222.175.142) port 80
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: 31.222.175.142
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache/2.2.20 (Ubuntu)
< Vary: Accept-Encoding
< Content-Type: text/html
< Date: Tue, 27 Mar 2012 22:58:04 GMT
< Accept-Ranges: bytes
< ETag: "5c033-2c-4bc2c13b42a80"
< Set-Cookie: X-Mapping-fjhppofk=0459144121542E9668F8271676341C7B; path=/
< Last-Modified: Mon, 26 Mar 2012 21:30:02 GMT
< Content-Length: 44
It works! urado2

* Connection #0 to host 31.222.175.142 left intact
* Closing connection #0

We sent another request but this time the client supply the Cookie header provided before. We use the value returned for the Server2. Base on the HTML payload we see that the response comes from previously selected pool member. The LB doesn't provide another cookie as it was before. 

[root@crado1 ~]# curl -v -H 'Cookie: X-Mapping-fjhppofk=0459144121542E9668F8271676341C7B' 31.222.175.142 
* About to connect() to 31.222.175.142 port 80
*   Trying 31.222.175.142... connected
* Connected to 31.222.175.142 (31.222.175.142) port 80
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: 31.222.175.142
> Accept: */*
> Cookie: X-Mapping-fjhppofk=0459144121542E9668F8271676341C7B
>
< HTTP/1.1 200 OK
< Date: Tue, 27 Mar 2012 22:58:35 GMT
< Server: Apache/2.2.20 (Ubuntu)
< Last-Modified: Mon, 26 Mar 2012 21:30:02 GMT
< ETag: "5c033-2c-4bc2c13b42a80"
< Accept-Ranges: bytes
< Content-Length: 44
< Vary: Accept-Encoding
< Content-Type: text/html
It works! urado2

* Connection #0 to host 31.222.175.142 left intact
* Closing connection #0

The some request sent again returns the some result - as expected. 

[root@crado1 ~]# curl -v -H 'Cookie: X-Mapping-fjhppofk=0459144121542E9668F8271676341C7B' 31.222.175.142                 
* About to connect() to 31.222.175.142 port 80
*   Trying 31.222.175.142... connected
* Connected to 31.222.175.142 (31.222.175.142) port 80
> GET / HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: 31.222.175.142
> Accept: */*
> Cookie: X-Mapping-fjhppofk=0459144121542E9668F8271676341C7B
>
< HTTP/1.1 200 OK
< Date: Tue, 27 Mar 2012 22:58:52 GMT
< Server: Apache/2.2.20 (Ubuntu)
< Last-Modified: Mon, 26 Mar 2012 21:30:02 GMT
< ETag: "5c033-2c-4bc2c13b42a80"
< Accept-Ranges: bytes
< Content-Length: 44
< Vary: Accept-Encoding
< Content-Type: text/html
It works! urado2

* Connection #0 to host 31.222.175.142 left intact
* Closing connection #0

Summary
The session persistence on the Rackspace Cloud Load Balancer uses HTTP cookie header. The load balancer when it returns the response back to the client it rewrites the IP addresses and inserts a new HTTP Cookie header. As long as the client obeys the Cookie value in his subsequent request it will be load balanced to the some pool member.

References
[1] persistence disabled
[2] persistence enabled

Labels: , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)