Search This Blog

Showing posts with label network. Show all posts
Showing posts with label network. Show all posts

Tuesday, May 27, 2014

How to use F5 Wireshark Plugin for LTM troubleshooting

In this post we are going to look how to use F5 Wireshark Plugin to troubleshoot networking issues on BigIP LTM.
  • Download the and install the plugin in your Wireshark
The full instruction are here F5 Wireshark Plugin. In essence you needed to copy the f5ethtrailer.dll file into C:\Program Files (x86)\wireshark\wireshark16\WiresharkPortable\ and restart my Wireshark.

Once you restart wireshark go to menu Help - About Wireshark, Plugins tab. You should be able to see the plugin listed there if properly installed.

  • The plugin is useful only if you take a capture on LTM with 'noise' information.
The noise is an internal information that TMM is attaching and managing for every packet when is being processed. To have a capture with noise these are the minimal options you need to specify:

tcpdump -w /var/tmp/capture.pcap -s0 -i _interface_:nnn

where the _interface_ can be:
    •  1.1 - example of an physical interface
    • dmz_vlan - a name you gave to your vlan when created
    • 0.0 - is the equivalent of 'any' interface what means capture on all interfaces and all vlans
My favourite syntax is usually something like this:

tcpdump -s0 -nn -w /var/tmp/test1-$(date +%s).pcap -i 0.0:nnn '(host _ip_ and port _port_ ) or arp or not ip' 
  • Open the capture in wireshark as normal
Once you open you will noticed that there is additional section in the packet details.

  • The most useful part of using this plugin is that you can quickly and easily find the client and server site traffic in the capture (It can be a challenging when you have multiple tcp streams and OneConnect profile):
    • Find a single packet of the flow you are interested in (search for VIP or client ip for example).
    • Find the "Flow ID" from the F5 Ethernet trailer (see the picture above for example).
    • Click with right mouse taste on the Flow ID field and select "Prepare as Filter".
    • In the Filter box (on top ) it will pre-populate the syntax for you.
    • Copy the hex value and delete the '.flowid == hex' part and start typing '.'  (dot).
    • It will mediately give you a list of possible options, select anyflowid and copy the hex back as it was originally. Example:
The original filter         : f5ethtrailer.flowid == 0x0d2e6dc0
Filter after modifications  : f5ethtrailer.anyflowid == 0x0d2e6dc0
    • Press Apply button
This filter is gong to find the client and server site flows for you. You can then analyse them packet by packet to find out and understand how and why LTM load balance it to one or another pool member.

References

https://devcentral.f5.com/wiki/advdesignconfig.F5WiresharkPlugin.ashx
https://devcentral.f5.com/questions/tcpdump-with-multiple-pool-members
SOL13637: Capturing internal TMM information with tcpdump

Tuesday, April 8, 2014

How does switch fabric network work

A network engineer can list a number of issues you can potentially run when using STP protocol  in your switch network. Over the years the network industry has created successor protocols like RSTP or MSTP. Both are improvements and offer much better convergence time and respond much quicker to switch topology changes. One of the major disadvantages for networks that relay on STP is the fact that they don't support multipathing. It means once network topology converges there will be blocked path between switches that are elected and managed by STP. This often redundant links can't be used because of a loop risk.

But there are better solutions today on the market to design better layer 2 Ethernet networks (more scalable, with higher throughput and with active link redundancy as an example). The 2 most popular are based on SPB and TRILL protocols. Both of them are used as a foundation in switch fabrics products. To better understand both of them the pictures below provide a side by side comparison. This was taken from Avaya document: Compare and Contrast SPB and TRILL.

Avaya is a SPB promoted so the comparison is a bit waited towards SPB but nevertheless it gives some inside view into both protocols.



References

http://cciethebeginning.wordpress.com/2008/11/20/differences-between-stp-and-rstp/
http://etherealmind.com/spb-attention/
http://en.wikipedia.org/wiki/IEEE_802.1aq
http://en.wikipedia.org/wiki/TRILL_(computing)
http://www.avaya.com/uk/resource/assets/whitepapers/SPB-TRILL_Compare_Contrast-DN4634.pdf
http://nanog.org/meetings/nanog50/presentations/Monday/NANOG50.Talk63.NANOG50_TRILL-SPB-Debate-Roisman.pdf
http://www.ebrahma.com/2012/06/trill-vs-spb-similarities-differences/
http://wikibon.org/wiki/v/Network_Fabrics,_L2_Multipath_and_L3

Friday, March 14, 2014

Interface redundancy on the host with TCP Multipath

TCP and UDP protocols are used exchange data between hosts. They have been used for a decade or longer and are very well documented how they work.

Everyone knows the problem that when you lost your active link on the server all your TCP sessions are going to die as well. Let's say your server has 2 active interfaces. There is no way to move/migrate a TCP session to use another active interface (by default). The other link can't be used automatically as a fail back mechanism.

There are couple of reasons behind why it isn't to works, the simplest one is that the new link used a different IP address. Even if the Linux kernel would start using the new interface and start sending IP/TCP packets sourced with the new IP address these packets wouldn't be recognized on the remote site. The remote site expect tcp segments from one and only one IP source.

Problem

How to provide a link level redundancy on the server to keep a TCP session alive even if one interface experience an error.

Analysis and solution Demonstration

The problem could be see as a more generic issue: how to implement multihoming or link redundancy. There are couple of working solution out there. The simplest example:
  • Link bonding(link aggregation) on the server; requires support and proper configuration on the switch and the server
We will look at another one: TCP Multipath. What is cool about this is that it is transparent to your application. It visualizes a session and provide a single TCP session to the application that can benefit from built-in multipath redundancy on the kernel level.


References

http://multipath-tcp.org/
Decoupled from IP, TCP is at last able to support multihomed hosts
https://devcentral.f5.com/articles/multipath-tcp-mptcp
https://devcentral.f5.com/articles/the-evolution-of-tcp


Sunday, March 2, 2014

How to do URL based load balancing on F5

There are many load balancers out there. Some of them offer a great flexibility to control the traffic by allowing a user to upload a custom script that implement the load balancing algorithm to solve a particular problem.

Problem

How to do HTTP URL based load balancing on F5.

Solution and demonstration

This is an iRule script that inspects the HTTP GET URL string to decided where to load balance it: https://github.com/rtomaszewski/f5/blob/master/lb-based-on-url.tcl.

Create default pool

Create VIP

Create custom pools

Testing

To verify that our iRule is working properly we can enable debugging by changing the iRule variable DEBUG to 1.

Next we can simulate traffic

curl -v http://vip/
curl -v http://vip/url1
curl -v http://vip/url2
curl -v http://vip/url3

And watch the logs on the lb.

tail -f /var/log/ltm

Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url1
Mar  2 15:49:37 local/tmm info tmm[5231]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url1
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url2
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': HTTP::uri eq /url3
Mar  2 15:49:37 local/tmm1 info tmm1[5232]: Rule rule-url-lb-vip-80 <HTTP_REQUEST>: '11.22.33.44': sent traffic to pool pool-vip-80-url3



Reference

https://devcentral.f5.com/wiki/iRules.HomePage.ashx

Sunday, February 23, 2014

How to design a network for Openstack or cloud deployment

Designing a network is a topic within itself and there is no way we can cover all of it in this single post. Cisco has its own certification path path CCDE for these who want to know more.

Cisco design and implementation guide - old best practices

In a very simplistic view back in the old days a network used to be design similar to the picture below (although it is hard to say when the new era started ;)). Every big network had to have a core, distribution(sometimes called aggregation) and access layer. The network was engineered mainly to help with North to South traffic in the data center or in another words to help get the data out and into the data center.


New cloud friendly data center design best practice

With the advent and popularization of new networking devises that support layer 2 routing, commonly know as networks fabrics (more info about TRILL and fabric) the network design has shifted in data centers. The way we design the networks today is to maximize the East to West traffic instead of the North to South ( old design above). The purpose of the new network is to allow more efficiently exchange data between the servers within the rack or data center.

They say that a picture is worth more than a thousand words. To help us to visualize how a new data center network/cloud network is designed these demonstration pictures (taken from Cisco document: Cisco Massively Salable Data Center) will shed some more light on it. Please note that we no longer use the core, distribution or access keyword but instead: spine, leaf or superspine to describe the different network layers :).

Sunday, January 12, 2014

During TLS handshaking server can ignore a cipher from the preference list in ClientHello message

I've noticed recently that when my client tries to negotiated a TLS session its ciphers preference list is being ignored by the server.
  • Test 1
$ openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
  • Test 2
$ openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
  • Test 3
# openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA:AES128-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: C7E7F65B4E927AFDE568E52FDAE52495CE815DAC2958B82C854146CD383FC00BB47ED691840C713CE762B6430FCB3230
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568347
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
  • Test 4
# openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA:DES-CBC-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: 1F0FB31410AAFFC234DC5598FFA2A676C63119A6367453D05578F9B1652DD085DB1D5C016694B9D293F3A2DFF769EF51
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568389
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

We can see in (1) and (2) that the TLS servers supports 2 different ciphers suits.
In (3) and (4) we see that even though the preference list was directly specified by the client the TLS server ignored the settings in (3).

Example ssldump output from (3).
 
# ssldump -A -n -i eth0 port 443 and host my_server_ip
New TCP connection #1: 162.13.0.27(51869) <-> my_server_ip(443)
1 1  0.1013 (0.1013)  C>SV3.1(61)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 23 b5 62 f5 08 38 14 53 d9 ac a4 51 c2 28
          0d e2 0e b6 41 89 2d af a1 50 55 bd a9 35 05 75
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.1984 (0.0970)  S>CV3.1(49)  Handshake
      ServerHello
        Version 3.1
        random[32]=
          2b ae 63 1e 9c da f3 38 b6 f9 dd c6 c0 4f ad 6c
          2c f0 f3 52 67 6e 28 29 8a 24 95 95 a5 67 f2 6f
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_AES_128_CBC_SHA
        compressionMethod                   NULL
1 3  0.1984 (0.0000)  S>CV3.1(674)  Handshake
      Certificate
        certificate[664]=
          50 82 02 94 50 82 01 fd 02 02 03 79 50 0d 06 09
          2a 86 48 86 f7 0d 01 01 05 05 00 50 81 b6 31 0b
          50 11 06 03 55 04 06 13 02 55 53 31 0e 50 0c 06
          03 55 04 08 13 05 54 65 78 61 73 31 14 50 12 06
          03 55 04 07 13 0b 53 61 6e 20 41 6e 74 6f 6e 69
          6f 31 12 50 10 06 03 55 04 0a 13 11 52 61 63 6b
          73 70 61 63 65 31 1e 50 1c 06 03 55 04 0b 13 15
          53 79 73 74 65 6d 20 41 64 6d 69 6e 69 73 74 72
          61 74 69 6f 6e 31 23 50 21 06 03 55 04 03 13 1a
...
          b3 79 0a 37 cd 27 93 af
1 4  0.1984 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
1 5  0.2126 (0.0141)  C>SV3.1(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          33 25 53 17 c2 45 b0 32 8a ca c1 66 39 6e f0 31
          98 96 4c 34 f2 e4 fd b7 0c e6 15 af c7 d3 fc e4
          0d 15 c9 c9 d4 e0 78 5a a1 13 dc 55 8b 5a bc 69
          68 24 f5 d1 50 6a 19 2e 71 9a 66 ee 3a 64 bc 1e
          d8 9a da d4 e0 44 96 b3 43 20 f1 a0 b6 4c 49 8e
          b2 ae 2b a6 12 68 78 19 eb 61 06 1c 34 8a 03 22
          ab 7e ff f8 88 44 89 97 cd 53 06 b0 b6 66 7b 77
          2a c0 0a 15 a3 54 2d 8c 5b 74 bc fe 31 3f 1f 8f
1 6  0.2126 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
1 7  0.2126 (0.0000)  C>SV3.1(48)  Handshake
1 8  0.3100 (0.0974)  S>CV3.1(1)  ChangeCipherSpec
1 9  0.3100 (0.0000)  S>CV3.1(48)  Handshake
1    1.4782 (1.1681)  C>S  TCP FIN
1    1.5740 (0.0958)  S>C  TCP FIN

Problem

Can TLS server ignore the cipher suite list passed from the client during TLS handshaking?

Analysis

The specification how the ClientHello message can be handled by the server during TLS handshaking has changed from TLS v1.0 to v1.2. The extract from the RFC comparing both relevant sections.
  • rfc2246
7.4.1.2. Client hello

   The CipherSuite list, passed from the client to the server in the
   client hello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first). Each CipherSuite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length) and a MAC algorithm.

   The server will select a cipher suite or, if no acceptable choices 
   are presented, return a handshake failure alert and close the 
   connection.
  • rfc5246
7.4.1.2.  Client Hello

   The cipher suite list, passed from the client to the server in the
   ClientHello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first).  Each cipher suite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length), a MAC algorithm, and a PRF.  The server will select a cipher
   suite or, if no acceptable choices are presented, return a handshake
   failure alert and close the connection.

   If the list contains cipher suites the server does not recognize, 
   support, or wish to use, the server MUST ignore those cipher suites,
   and process the remaining ones as usual.

In the yellow parts we can see that a server implementing TLS v1.2 has the ability to ignore ciphers specified by the client in its list ClientHello message. The criteria are very wide. The TLS server can ignore a cipher only because it doesn't wish to use it.

References

http://www.openssl.org/docs/apps/ciphers.html
DES and IDEA ciphers are deprecated in the latest TLS protocol
http://www.ietf.org/rfc/rfc2246.txt - The TLS Protocol, Version 1.0
http://tools.ietf.org/html/rfc5246.txt - The Transport Layer Security (TLS) Protocol, Version 1.2
http://support.f5.com/kb/en-us/solutions/public/10000/200/sol10209.html - ssldump

A definitely interesting reading about ciphers in the web browser to show some parasitical view at the cipher list topic.
https://groups.google.com/forum/#!topic/mozilla.dev.tech.crypto/36na1B2brGU

Monday, December 30, 2013

Physical network diagram generated from dot config language

Is there a good and easy solution to automatically generate logical or physical network diagrams? Trying answer this question I've been playing today with the dot configuration language to test it in the field.

Automatically generated diagram


The DOT config file 

graph physical_net_topology {
 
legend [ label="{ legend | {red | public} |{blue| v1002 inside} | {green | v105 dmz} }", shape=record];


subgraph cluster_sp {
    label="pub switch";

    graph [ fillcolor="burlywood", style="filled"]
    node [shape=record,fillcolor="white", style="filled"]
    edge[style=invis];
    
    node [label="3"] p3 ;
    node [label="2"] p2 ;
    node [label="1"] p1 ;

    { rank=same; p1; p2; p3}
    p1 -- p2 -- p3;
}

subgraph cluster_si {
    label="Internal switch";

    edge[style=invis];
    node [shape=record ];

    node [label="1"] e1 ;
    node [label="2"] e2 ;
    node [label="3"] e3 ;
    node [label="4"] e4 ;
    node [label="5"] e5 ;
    node [label="6"] e6 ;

    { rank=same; e1; e2; e3; e4; e5; e6;}
    e1 -- e2 -- e3 -- e4 -- e5 -- e6;
}

subgraph cluster_fw1 {
    label="FW1 ports";

    graph = [ style = rounded]
    edge[style=invis]
    
    node [shape=record ];
    node [label="1"] f1 ;
    node [label="2"] f2 ;
    node [label="3"] f3 ;
    node [label="4"] f4 ;

  { rank=same; f1; f2; f3; f4; }

  f1 -- f2 -- f3 -- f4'
}

p2 -- f1 [color="red"]
f2 -- e1 [color="blue"]
f3 -- e2 [color="green"]

e3 -- server1 [color="green"]
e4 -- server2 [color="green"]
e6 -- server3 [color="blue"]

}

Results discussion

It took me a good couple of hours to write this config. Even after that I still have only a very basic understanding of how flexible the dot language is. The coding was very laborious. It required a lot of trying and testing if the new generated graph is what you are looking for.

Often the options I was trying didn't have any effect.

The documentation I found and read wasn't explaining all the details so trying and intuition was often your only friend.

References

http://sandbox.kidstrythisathome.com/erdos/
http://graphviz-dev.appspot.com/
http://www.graphviz.org/

http://rtomaszewski.blogspot.co.uk/search/label/diagram

Tuesday, December 17, 2013

Top 10 data center network vendor list

In previous post was saw how handful the Gartner magic quadrants can be when learning and researching particular technology or industry trend:
Below is an another comparison chart, this time targeting vendor network equipment in data centers.


By the way I was investigating this because of the networks vendor selection on this cloud hardware deployment calculator (How to estimate hardware requirements for your private Openstack deployment ) : Dell, Cisco, HP, Arista, Juniper, Brocade.

Sunday, December 15, 2013

ASCII diagram tells more than a full page of text

Before anyone can help you they need to understand you first. Below is an example why a single picture is worth more than than full page of text.

Problem

How long would it take to explain to somebody that:
  • VLAN1 is used ONLY for inter VM communication
  • VMs in VLAN2 and 3 are in isolated network segments that have internet access
  • VMs in VLAN2 and 3 can talk to each other (using the FW as its default gateway to route traffic)
Solution

ASCII diagram:
                                               +---+------------+
                                               |   |            |
          ^                                    | v |------------|
          | internet                           | m |  vm2       |
          |                         vlan1      | 1 |------------|
      +---+---+         +-------+   vlan2      |   |  vm3       |
      | FW    | <-------|switch1| ------------ |----------------|
      +-------+   vlan3 +-------+   vlan3      |virt1 |virt2 |  |
                  vlan2     ^                  |switch|switch|  |
                            |                  |------+------+--|
                            |                  | Hypervisor1    |
                            | trunk            +----------------+
                            | vlan1
                            |
                            |
                            +
                       +-----------+  vlan1            
                       |aggregation|  vlan2    +-------------+
                       |-----------+<--------- | vm1 vm2     |
                       | switch    |  trunk    |-------------|
                       +-----------+           | Hypervisor2 |
                                               +-------------+

References

http://www.asciiflow.com/#Draw
http://www.streamwave.com/web-development/ascii-diagram-tools/
http://nedbatchelder.com/blog/200911/ditaa_diagrams_through_ascii_art.html

Monday, November 18, 2013

Network operating system architecture

In my previous posts we took a look at how hardware networking vendors design and build modern systems. We concentrated mainly on the OS and software integration on the appliance:
Today I found another good article on Arista blog that takes this one step further and again advocates an open operating system design: Linux as a Switch Operating System: Five Lessons Learned. This is what Arista says:
  • It’s okay to leave the door unlocked - get net admin access to the underlying Linux operating system tools
  • Preserve the integrity of the Linux core - keep your product specific changes as small as possible to allow integration with already existing software out there
  • Focus on state, not messages - being only maybe an average programmer I will not comment on this as this is clearly not my area of expertise
  • Keep your hands out of the kernel - why to complicate your code if it doesn't bring any revolutionary benefit; besides it is much easier to find developers who know how to program in linux/libc than in Linux kernel.
  • Provide familiar interfaces to ease adoption - everyone knows the IOS CLI so why to invent something new.
Lesson learned from Arista: reuse, integrate and trust your customers.

The company success would not be possible without a clear and consistent OS and tools set design. From my experience they are the second company (I saw this first on BigIP - F5 load balancers).

Thursday, July 11, 2013

What is AppFlow

In networking one of the words that has recently become popular is 'flow'. We have OpenFlow protocols that is a new emerging standard in network visualization that uses flows to describe what action should happen to a packet in OVS switch. We have sFlow and Netflow that are widely used by networking vendors for statistics information and performance data gathering from network devices. I've come recently across a new term: AppFlow.

Problem

What is AppFlow?

Analisis

In the video below we can learn that AppFlow is a protocols that can be used for application monitoring and visibility. With the industry acceptance of ADS/ADC systems (Application Delivery System and Application Delivery Controller) it is therefor logically rational to extend load balancers functionality to allow them to collect application specific data. The idea is similar how sFlow/Netflow work where additionally to IP and TCP based information we want to collect application specify information as well.


References
  1. http://blogs.citrix.com/2011/05/09/introducing-appflow-netflow-for-apps/
  2. http://www.youtube.com/watch?v=MNbLi-t7ZPE
  3. http://www.appflow.org/
  4. http://www.citrix.com/products/netscaler-application-delivery-controller/features/visibility.html
  5. http://www.citrix.com/products/netscaler-application-delivery-controller/how-it-works/technologies.html

Tuesday, June 18, 2013

How does SFTP work

Every network administrator knows how painful is it to troubleshoot FTP protocol issues. Many of them are related to the FTP design concept of independent control and data channels. Because both FTP data and control channels are handled in separate TCP session often network devices are allowing one TCP session but blocking the other. The end result is that you can't connect to FTP server or you can't download a file or your download hangs and never completes.

There are number of alternatives. On of them is SFTP (Secure File Transfer Protocol). The design of this protocols is very different from FTP even though it shares its name.

In FTP we have concept of separate control and data channels. The client can sent commands over the control TCP session and data transfers happens in the data TCP sessions. Both the control and data TCP session has its own TCP conventions. Passive and active FTP mode make it even more convoluted.

In SFTP there is one session (by default the client connect on port 22 from random source port). The single session is used to exchange a mix of control and data commands. This is still truth that for a file transfer you need a new TCP session but the session is initiated in a similar way on port 22. From network point of view there is not much difference how the first or second TCP session looks. The sessions are independent but connected.

Example tcpdump showing a single file transfer (100* is client, 200* is server):
 
Nr      Time        Source             Destination         Protocol flags      Length      vlan hsrc                  hdst             sport  dport  Window size value Seq#       Ack#       Info

# connecting to the SFTP server
    130 0.017634    100.0.0.1          200.0.0.1           TCP      0x02       66          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     65535             0                     55373 > ssh [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=128 SACK_PERM=1
    131 0.002186    200.0.0.1           100.0.0.1          TCP      0x12       66          Cisco_ee:08:c0        Dell_75:49:e3         22     55373  8192              0          1          ssh > 55373 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
    132 0.000033    100.0.0.1          200.0.0.1           TCP      0x10       54          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32768             1          1          55373 > ssh [ACK] Seq=1 Ack=1 Win=4194304 Len=0
    133 0.004121    200.0.0.1           100.0.0.1          SSHv2    0x18       87          Cisco_ee:08:c0        Dell_75:49:e3         22     55373  258               1          1          Server Protocol: SSH-2.0-1.82_sshlib GlobalSCAPE\r
    134 0.000234    100.0.0.1          200.0.0.1           SSHv2    0x18       97          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32767             1          34         Client Protocol: SSH-2.0-PuTTY_Local:_Jan__8_2012_14:40:35\r
    135 0.000055    100.0.0.1          200.0.0.1           TCP      0x18       566         Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32767             44         34         [TCP segment of a reassembled PDU]
    136 0.000018    100.0.0.1          200.0.0.1           SSHv2    0x18       182         Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32767             556        34         Client: Key Exchange Init
    137 0.002431    200.0.0.1           100.0.0.1          SSHv2    0x18       542         Cisco_ee:08:c0        Dell_75:49:e3         22     55373  258               34         44         Server: Key Exchange Init
    138 0.000001    200.0.0.1           100.0.0.1          TCP      0x10       60          Cisco_ee:08:c0        Dell_75:49:e3         22     55373  256               522        684        ssh > 55373 [ACK] Seq=522 Ack=684 Win=65536 Len=0
    139 0.000076    100.0.0.1          200.0.0.1           SSHv2    0x18       70          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32763             684        522        Client: Diffie-Hellman Key Exchange Init
    140 0.002168    200.0.0.1           100.0.0.1          SSHv2    0x18       334         Cisco_ee:08:c0        Dell_75:49:e3         22     55373  256               522        700        Server: Diffie-Hellman Key Exchange Reply
    141 0.018620    100.0.0.1          200.0.0.1           SSHv2    0x18       326         Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32761             700        802        Client: Diffie-Hellman GEX Init
    142 0.024606    200.0.0.1           100.0.0.1          SSHv2    0x18       902         Cisco_ee:08:c0        Dell_75:49:e3         22     55373  254               802        972        Server: Diffie-Hellman GEX Reply
    144 0.005229    100.0.0.1          200.0.0.1           SSHv2    0x18       70          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32768             972        1650       Encrypted request packet len=16[Malformed Packet]
    145 0.000113    100.0.0.1          200.0.0.1           SSHv2    0x18       142         Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32768             988        1650       Encrypted request packet len=88
    146 0.001946    200.0.0.1           100.0.0.1          TCP      0x10       60          Cisco_ee:08:c0        Dell_75:49:e3         22     55373  254               1650       1076       ssh > 55373 [ACK] Seq=1650 Ack=1076 Win=65024 Len=0
    147 0.000001    200.0.0.1           100.0.0.1          SSHv2    0x18       174         Cisco_ee:08:c0        Dell_75:49:e3         22     55373  254               1650       1076       Encrypted response packet len=120
    148 0.000249    100.0.0.1          200.0.0.1           SSHv2    0x18       158         Dell_75:49:e3         Cisco_ee:08:c0        55373  22     32767             1076       1770       Encrypted request packet len=104
    149 0.002327    200.0.0.1           100.0.0.1          SSHv2    0x18       122         Cisco_ee:08:c0        Dell_75:49:e3         22     55373  254               1770       1180       Encrypted response packet len=68
......

# we initiated a file transfer 
# new tcp session is created, similar like in passive FTP mode
   1683 0.064012    100.0.0.1          200.0.0.1           TCP      0x02       66          Dell_75:49:e3         Cisco_ee:08:c0        55375  22     65535             0                     55375 > ssh [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=128 SACK_PERM=1
   1684 0.002196    200.0.0.1           100.0.0.1          TCP      0x12       66          Cisco_ee:08:c0        Dell_75:49:e3         22     55375  8192              0          1          ssh > 55375 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1
   1685 0.000035    100.0.0.1          200.0.0.1           TCP      0x10       54          Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32768             1          1          55375 > ssh [ACK] Seq=1 Ack=1 Win=4194304 Len=0
   1686 0.004089    200.0.0.1           100.0.0.1          SSHv2    0x18       87          Cisco_ee:08:c0        Dell_75:49:e3         22     55375  258               1          1          Server Protocol: SSH-2.0-1.82_sshlib GlobalSCAPE\r
   1687 0.000506    100.0.0.1          200.0.0.1           SSHv2    0x18       97          Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32767             1          34         Client Protocol: SSH-2.0-PuTTY_Local:_Jan__8_2012_14:40:35\r
   1688 0.000053    100.0.0.1          200.0.0.1           TCP      0x18       566         Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32767             44         34         [TCP segment of a reassembled PDU]
   1689 0.000015    100.0.0.1          200.0.0.1           SSHv2    0x18       182         Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32767             556        34         Client: Key Exchange Init
   1690 0.002266    200.0.0.1           100.0.0.1          SSHv2    0x18       542         Cisco_ee:08:c0        Dell_75:49:e3         22     55375  258               34         44         Server: Key Exchange Init
   1691 0.000001    200.0.0.1           100.0.0.1          TCP      0x10       60          Cisco_ee:08:c0        Dell_75:49:e3         22     55375  256               522        684        ssh > 55375 [ACK] Seq=522 Ack=684 Win=65536 Len=0
   1692 0.000109    100.0.0.1          200.0.0.1           SSHv2    0x18       70          Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32763             684        522        Client: Diffie-Hellman Key Exchange Init
   1693 0.002230    200.0.0.1           100.0.0.1          SSHv2    0x18       334         Cisco_ee:08:c0        Dell_75:49:e3         22     55375  256               522        700        Server: Diffie-Hellman Key Exchange Reply
   1697 0.000446    100.0.0.1          200.0.0.1           SSHv2    0x18       326         Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32761             700        802        Client: Diffie-Hellman GEX Init
   1698 0.023470    200.0.0.1           100.0.0.1          SSHv2    0x18       902         Cisco_ee:08:c0        Dell_75:49:e3         22     55375  254               802        972        Server: Diffie-Hellman GEX Reply
   1699 0.027365    100.0.0.1          200.0.0.1           SSHv2    0x18       70          Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32768             972        1650       Encrypted request packet len=16[Malformed Packet]
   1700 0.000111    100.0.0.1          200.0.0.1           SSHv2    0x18       142         Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32768             988        1650       Encrypted request packet len=88
   1701 0.001906    200.0.0.1           100.0.0.1          TCP      0x10       60          Cisco_ee:08:c0        Dell_75:49:e3         22     55375  254               1650       1076       ssh > 55375 [ACK] Seq=1650 Ack=1076 Win=65024 Len=0
   1702 0.000001    200.0.0.1           100.0.0.1          SSHv2    0x18       174         Cisco_ee:08:c0        Dell_75:49:e3         22     55375  254               1650       1076       Encrypted response packet len=120
   1703 0.000303    100.0.0.1          200.0.0.1           SSHv2    0x18       158         Dell_75:49:e3         Cisco_ee:08:c0        55375  22     32767             1076       1770       Encrypted request packet len=104
   ...

# once the tranfer is finished both sessions are closed

   2498 0.015696    100.0.0.1          200.0.0.1           TCP      0x14       54          Dell_75:49:e3         Cisco_ee:08:c0        55373  22     0                 3768       3158       55373 > ssh [RST, ACK] Seq=3768 Ack=3158 Win=0 Len=0

Example loggin session from FileZilla:
 
Status: Connecting to 200.0.0.1...
Response:   fzSftp started
Command:    open "user_name@200.0.0.1" 22
Command:    Trust new Hostkey: Once
Command:    Pass: **********
Status: Connected to 200.0.0.1
Status: Retrieving directory listing...
Command:    pwd
Response:   Current directory is: "/"
Command:    ls
Status: Listing directory /
Status: Calculating timezone offset of server...
Command:    mtime "Archive"
Response:   1365109008
Status: Timezone offsets: Server: 0 seconds. Local: 3600 seconds. Difference: 3600 seconds.
Status: Directory listing successful
Status: Connecting to 200.0.0.1...
Response:   fzSftp started
Command:    open "user_name@200.0.0.1" 22
Command:    Trust new Hostkey: Once
Command:    Pass: **********
Status: Connected to 200.0.0.1
Status: Starting download of /paht/to/the/file.txt
Command:    cd "/"
Response:   New directory is: "/"
Command:    get "/paht/to/the/file.txt" "c:\Users\file.txt"
Status: remote:/paht/to/the/file.txt => local:c:\Users\file.txt
Status: File transfer successful, transferred 360 bytes in 1 second
Status: Disconnected from server
Status: Disconnected from server

References
  1. http://blog.asmallorange.com/the-difference-between-ftp-sftp-and-ftps/
  2. https://wiki.filezilla-project.org/SFTP_specifications
  3. http://en.wikipedia.org/wiki/SFTP (some info about FTP alternatives)
  4. http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13

Sunday, June 9, 2013

SR-IOV technology enables low level network virtualization

In the virtualization space the SR-IOV technology was introduced in about 2008/2010 [1]. The technical details can be found under the links in the reference section but in plain English the technology allows to create many virtual devices base on single a physical device. For this to work the hardware (CPU, North chip) and operating system need to have SR-IOV support.

Below is a video demonstrating packet processing for an Intel Ethernet card that supports SR-IOV.


Interesting slides showing the concept from the video and  reference links:
  • After the frame enters the physical port on the NIC the low level driver/firmware (supporting the SR-IOV) distributes the packet (based on header classifications/hash value/etc) to separate virtual queues 
  • Each virtual queue is assigned directly to a virtual device 
  • Once the packet is in the queue it can be deliver to the VM DIRECTLY without the usual software hypervisor overhead
  • Packets don't have to be copied from physical port buffer(s) to OS RAM and than from OS RAM to VM OS buffers. The data can be sent directly from the physical port to VM OS buffers. That way the hypervisor processing overhead can be minimised.
  • Critical part for the technology is CPU and chipset virtualization support
  • As access to physical RAM need to be protected between hypervisor and VMs as well as VMs themselves the virtual memory address is translated to the physical location by the north chipset
  • For the DMA request to copy the packets the address translation between the hypervisor address space and the VM address space is transparent (north chip take care of it)
  • Another view how the packet is delivered from physical port to the VM
References
  1. http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf
  2. http://www.intel.com/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf
  3. http://communities.intel.com/community/wired/blog/2010/09/07/sr-iov-explained

Thursday, June 6, 2013

Network appliance architecture

Networking is a fascinating topic. There are thousands of books and RFC describing protocols from layer 1 to layer 7. In every big network we find variety of network devices that handle traffic and provide additional enhanced services. Example of such devices can be switches, routers, load balancers, traffic accelerators, firewalls, IDS, DDOS mitigation devices and others. Services could be QoS, security and traffic deduplication etc.

Looking at the network devices someone could ask one interesting questions: how to build a network device; how would you describe an internal device architecture?

Of course every company has its own patents, secrets, methods etc. how they built, manage and operate network appliances. Below are some information I found when working with them or reading about them.

Monday, May 27, 2013

How does the VXLAN protocol work

The VXLAN is one of the overlay network tunneling protocols that is used to built network infrastructure for cloud environment. Below are some details about the operation and specification.
  • Frame headers definition

  • The traffic between VMs is encapsulated in IP/UDP packets
  • Logical isolation is implemented in a form of logical overlay where the traffic is exchanged between encryption tunnels endpoints
  • VXLAN ID is used to identify the specify isolated L2 cloud network that belongs to a tenant
  • The tunnel endpoints represent the edge of the cloud network infrastructure
  • The tunnel endpoints perform encapsulation and decapsulation
  • It is there where all the logic is implemented to find out where to sent next a packet or to witch VM the packet should be delivered after decapsulation



  • A comprehensive summary and operational features can be found under the links in reference section, below are few of the main characteristics and benefits:
    • It operates over IP and used UDP to carry payload 
    • Multicast support is the only other requirement for switches and routers to support VXLAN 
    • Multicast is used to handle L2 broadcast traffic (like ARP requests)
    • Logical networks can be extended among virtual machines placed in different Layer 2 domains
References
  1. http://www.definethecloud.net/vxlan-deep-dive
  2. http://www.definethecloud.net/vxlan-deep-divepart-2
  3. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-685115.html
  4. http://www.emulex.com/artifacts/d658610a-d3b6-457c-bf2d-bf8d476c6a98/elx_wp_all_VXLAN.pdf
  5. http://blogs.cisco.com/datacenter/more-vxlan-qa/
  6. http://blog.scottlowe.org/2011/12/07/revisiting-vxlan-and-layer-3-connectivity/
  7. http://blog.scottlowe.org/2011/12/22/otv-and-vxlan-layer-3-connectivity-compared/

Sunday, April 28, 2013

How to capture HTTP traffic using tcpdump

Everyone knows tcpdump but not everyone knows how to use it in efficient way. Below is a nice trick how to capture the HTTP GET request and the server response in plain text.
 
root@server:~# tcpdump -c100 -A -l -tttt -s0 -qpnni any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

2013-04-28 13:57:16.725851 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`....(.@*..Hx...........*..P@   ...........}.P/.Hp......8@.     .........
..b.........
2013-04-28 13:57:16.728057 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`....(.7*..P@   ..........*..Hx............P.}}.[./.Hq..7..{.........
^.....b.....
2013-04-28 13:57:16.728093 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.Hq}.[............
..b.^...
2013-04-28 13:57:16.728445 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 166
`......@*..Hx...........*..P@   ...........}.P/.Hq}.[............
..b.^...GET / HTTP/1.1
User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
Host: www.google.com
Accept: */*


2013-04-28 13:57:16.729989 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@   ..........*..Hx............P.}}.[./.I.....^E.....
^.....b.
2013-04-28 13:57:16.742677 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 988
`......7*..P@   ..........*..Hx............P.}}.[./.I......h.....
^.....b.HTTP/1.1 302 Found
Location: http://www.google.co.uk/
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Set-Cookie: PREF=ID=0241f520b15aae6e:FF=0:TM=1367157332:LM=1367157332:S=5lVTK-ZqTfrki7HN; expires=Tue, 28-Apr-2015 13:55:32 GMT; path=/; domain=.google.com
Set-Cookie: NID=67=OyloOHElfW8AKOcsRJ4DeQnfMhqfmnqJgcpYXlsSrN2ouREV9KHjS9boJZTfBoFZvYtVg0ugcBa2lJQKX-WrQ_uMxoIvPg-4JehPfFEdyGl_oh0RS37x_V6a_ozMElzJ; expires=Mon, 28-Oct-2013 13:55:32 GMT; path=/; domain=.google.com; HttpOnly
P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
Date: Sun, 28 Apr 2013 13:55:32 GMT
Server: gws
Content-Length: 221
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.co.uk/">here</A>.
</BODY></HTML>

2013-04-28 13:57:16.742700 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.743753 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._w...........
..b.^...
2013-04-28 13:57:16.745725 IP6 2a00:1450:4009:808::1011.80 > 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909: tcp 0
`.... .7*..P@   ..........*..Hx............P.}}._w/.I.....ZS.....
^.....b.
2013-04-28 13:57:16.745743 IP6 2a00:1a48:7805:111:8cfc:cf10:1111:111.54909 > 2a00:1450:4009:808::1011.80: tcp 0
`.... .@*..Hx...........*..P@   ...........}.P/.I.}._x...........
..b.^...

10 packets captured
10 packets received by filter
0 packets dropped by kernel

Monday, April 15, 2013

Howto generate simple network diagrams

A picture is worth more than 1000 words one of the proverbs says. This is absolute truth when you  troubleshoot and need to understand a customer network environmental  Below is a very simple recipe (unfortunately) how to generate a network diagram with a tool called nwdiag.

Instalation
 
aptitude install python-dev
pip install nwdiag

# will generate a png file diag1.png
nwdiag  diag1.conf

Example 1: FW with one segment
 
{
  public [shape = cloud];
  public -- firewall;

  network inside {
      address = "192.168.100.x/24"

      firewall [address = "192.168.100.1"];
      web01 [address = ".11"];
      cachesrv [address = ".51"];
      db1 [address = ".101"];
  }
}

Example 2: FW with dmz and inside segment
 
{
  public [shape = cloud];
  public -- firewall;

  network dmz {
      address = "192.168.200.x/24";

      firewall [address = "192.168.200.1"];
      web01 [address = ".11"];
      cachesrv [address = ".51"];
  }
  network inside {
      address = "192.168.100.x/24"

      firewall [address = "192.168.100.1"];
      db1 [address = ".101"];
  }
}

Wednesday, April 10, 2013

Howto tune HTTP browser connection with Connection and Keep-Alive HTTP headers

I've come across this response today. This is diffident from all other I have seen so far. It uses the timeout and max options for the Keep-Alive header that should influences how the TCP connection should be handled.
 
HTTP/1.1 200 OK
Date: Wed, 10 Apr 2013 08:42:48 GMT
Server: Apache/2.2.3 (Red Hat)
Last-Modified: Wed, 10 Apr 2013 08:42:48 GMT
Cache-Control: no-store,no-cache,must-revalidate
Expires: -1
Content-Length: 1567
Keep-Alive: timeout=5, max=10
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8

Looking at the recent HTTP 1.1 RFC rfc2616 we can see that this options is addressed in more details here:

The original HTTP/1.0 form of persistent connections (the Connection: Keep-Alive and Keep-Alive header) is documented in RFC 2068.

Looking at RFC 2068 we see that it specifies in the paragraph  19.7.1.1 The Keep-Alive Header

The Keep-Alive header itself is optional, and is used only if a parameter is being sent. HTTP/1.1 does not define any parameters.

It doesn't specify anything about max or timeout parameters  With Google help I found this RFC draft Hypertext Transfer Protocol (HTTP) Keep-Alive Header. In this one we can see that:

The value of the timeout parameter is a single integer in seconds.
The value of the max parameter counts the number of requests since the connection was created.


Thursday, December 20, 2012

What do you need to implement virtual network and build hybrid cloud

To build an infrastructure that can be used to host a hybrid cloud environments or to benefit from the flexibility that a virtual networking provides we need software and hardware components. Below are couple of links I found when researching on this topic.

VMware ESXi/vSphere
Citrix XenServer
Opensource Linux alternatives
Microsoft Hyper-V
Vendors


Sunday, December 9, 2012

High availability pattern using anycast IP addresses for cloud and applications

Anycast architecture that helps to create and achieve HA

To run efficiently applications demands more and more resources. Even with the right amount of computational resources like servers, CPU, RAM, storage for them to be considered efficient and successfully on the market they have to meet many more requirements. It is impossible to list here all of them as they can depend on internal factors (for example driven by the application architecture itself ) or relay on external factors that may be specific and unique to a customer and an environment.

Although in this short blog post, I would like to discuss the importance of scalability factor and show one patters that can be used to build a highly available and efficient infrastructure systems.

There are two concept how we can try to implement a scalability: scale up vs scale out. For more information about scale up (or vertical scaling) these links provide further information [1]. We will concentrate here only on the scale out option. All the pictures below are base on this presentation that slides can be found here: OpenStack-Design-Summit-HA-Pairs-Are-Not-The-Only-Answer.
  1. To fully benefit from the HA pattern your application architecture should relay use share nothing paradigm

  2. That way if failure occurs only an isolated, small part of the computational resources will be impacted.


  3. Next we have to configure our routers and implementing necessary changes for a routing protocol.
  4. OSPF routing protocols is an examples and others can be used in similar way as well. For more info can be found here [4].


    The slides show only a fraction of the configuration. Another good example with con figs can be found here:  Anycast DNS - Part 4, Using OSPF

  5. As last you have to configure you servers to listen and accept traffic for our anycast IP.
A best practice is to configure the external IP on the loopback interface, disable ARP protocols for it and bind our application specifically to this IP.

References
  1. http://en.wikipedia.org/wiki/Scalability
  2. https://devcentral.f5.com/blogs/us/lots-of-little-virtual-web-applications-scale-out-better-than-scaling-up
  3. Seattle Conference on Scalability: Lessons In Building Scalable Systems
  4. What is “anycast” and how is it helpful?