Overlay technologies in data center

Everyone speaks about SDN an the benefits its brings when deploying cloud or enterprise infrastructures. But do we actually know or have any understanding what this all SDN is about? If you want be fluent in the language of virtual networking and network overlays in modern data centers you need to understand at least the following concepts:

• SDN controller (Northbound and Southbound API) (an example controller could be VMware NVP/NSX or OpenDaylighta; a list of vendor specific Northbound API)
• Openflow as Southbound API (example flow in Open vSwitch)
• Network Functions Virtualization
• Overlay and tunnels concept for cloud network inside data center (example protocols are VXLAN and STT)

In the remaining of the post we will concentrate solely on existing overlay technologies. These information was extracted from Cisco doc: Cisco Nexus 9000 Series Switches - Data Center Overlay Technologies).

Network-Based Overlay Networks

1. IEEE 802.1ad Provider Bridging or IEEE 802.1q Tunneling also known as IEEE 802.1QinQ or simply Q-in-Q
2. IEEE 802.1ah Provider Backbone Bridges (PBB) or Mac-in-Mac Tunnels
3. Cisco FabricPath allows multipath networking at Layer 2
4. TRILL - IETF Transparent Interconnection of Lots of Links is a Layer 2 multipathing technology
5. Shortest-Path Bridging (SPB) is defined in IEEE 802.1aq and is targeted as a replacement for Spanning Tree Protocol (example info based on Avaya documentation)
6. Cisco Overlay Transport Virtualization (OTV) is a Layer 2-over-Layer 3 encapsulation "MAC-in-IP" technology
7. The Cisco Location/Identifier Separation Protocol (LISP) is currently defined as a Layer 3 overlay scheme over a Layer 3 network
8. Multiprotocol Label Switching (MPLS)
9. Virtual Private LAN Service (VPLS) a Layer 2 tunneling protocols
10. Virtual Private Routed Network (VPRN) also known as BGP/MPLS or IP-VPN provides IP VPN services

Host-Based Overlay Networks

1. Virtual Extensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3 networ that uses IP/UDP encapsulation
2. Network Virtualization Using Generic Routing Encapsulation (NVGRE) allows creation of virtual Layer 2 topologies on top of a physical Layer 3 network
3. Stateless transport tunneling (STT) is an overlay encapsulation scheme over Layer 3 networks that use a TCP-like header

You can use bash shell instead of Cisco CLI on Nexus Switches

Every one who works on Linux and understand how to efficiently use Bash hates to work with the limited Cisco IOS CLI. The design objectives standing behind this CLI haven't changed for the last 20 years or so. It is obvious that this tools lacks plenty of features expected from a modern shell for many people.

But the evolution or even revolution that is happening in networking thanks to SDN is changing this terrible static network configuration landscape. The new generation of network devises like Cisco Nexus platform are going to support in the Cisco NX-OS :

• Bash shell
• Python shell 
• API access
• Linux containers for custom applications

For these who still don't believe you can read about this here:

• http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html

• http://www.cisco.com/c/en/us/products/switches/nexus-9516-switch/index.html

• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-729405.html

References

http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/white-paper-listing.html
http://rtomaszewski.blogspot.co.uk/search/label/sdn
http://rtomaszewski.blogspot.co.uk/2013/09/cisco-cheat-sheet.html

Description and demonstration of the Heartbleed bug in OpenSSL

There is a ton of posts on the Internet about the new bug in OpenSSL. I'm not going to repeat what others wrote  but rather give us a small demonstration.

Heartbeat packet description in SSL protocol suite

This is excellent blog posts we can take a look at the openssl code analysis and see where exactly the bug was hidden: Diagnosis of the OpenSSL Heartbleed Bug.

If you want to learn more how to build an potential exploid you can read and watch this: http://security.stackexchange.com/questions/55116/how-exactly-does-the-openssl-tls-heartbeat-heartbleed-exploit-work

A working code for a prof of concept can be found here:
http://www.garage4hackers.com/entry.php?b=2551
http://nakedsecurity.sophos.com/2014/04/08/anatomy-of-a-data-leak-bug-openssl-heartbleed/

Demonstration

How do I know if my site is vulnerable?

There are potentially many different ways how you can test if a site is vulnerable. As two extreme examples (a) we could write a simple SSL client and try to sent an hearbeat packet (not so trivial and requires some knowledge about the ssl protocol itself) or (b) search for a site on Internet that do the testing for us. I would definitively avoid (b). These sites can store the URL you provided and try to exploit you later.

A more simple and elegant solution can be built using openssl cli client tool instead. By running as single line script you can test if a server supports heartbeat or not. Next you have to find if the version of the OpenSSL you use is vulnerable.

 
$ openssl s_client -connect www.cloudflarechallenge.com:443 -tlsextdebug
CONNECTED(00000003)
TLS server extension "renegotiation info" (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension "EC point formats" (id=11), len=4
0000 - 03 00 01 02                                       ....
TLS server extension "session ticket" (id=35), len=0
TLS server extension "heartbeat" (id=15), len=1
0000 - 01                                                .
depth=4 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 4 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFLTCCBBWgAwIBAgIQSkGkHc+NJGGLqUs9YZlcxDANBgkqhkiG9w0BAQUFADBy
MQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDEYMBYGA1UE
AxMPRXNzZW50aWFsU1NMIENBMB4XDTE0MDQxMDAwMDAwMFoXDTE0MDcwOTIzNTk1
OVowWDEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRhdGVkMREwDwYDVQQL
EwhGcmVlIFNTTDEgMB4GA1UEAxMXY2xvdWRmbGFyZWNoYWxsZW5nZS5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCbQBaRWcPHl945y10L3tm2C+13
bm4oqaGMIekvJyYTF7VGJFKX+EYgvt/wWD+qJTO1Wbm5dknVQbt3PP7061M2H6/b
sG3M+xTfKK8d6/AAHWZMy0/ps+5cGPOzFFwL3JVwEFakoExGc3jT6S9RlhU5q4I+
q8Qd+jpHL7uKeklipCb8VIznRmtGKYI7H01kjyW8gwXYOrWKlKCHOIcR32LIxHfd
fv72QjT2kGupne3TmXAY+6cEL12ZqS2HCYpGBa8QQaZ7/dggc1X5OJL1yrQP8Le9
/faCOBHn0A4yzNp873BVMQ+7T+7k2PCSs7qAfB0TdvdfQFiPPFaTODDtPWClAgMB
AAGjggHXMIIB0zAfBgNVHSMEGDAWgBTay+qtWwhdzP/8JlTOSeVVxjj0+DAdBgNV
HQ4EFgQUbqyvF2sHtsjg5i82wBON35elvNQwDgYDVR0PAQH/BAQDAgWgMAwGA1Ud
EwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMCBgorBgEEAYI3
CgMDBglghkgBhvhCBAEwTwYDVR0gBEgwRjA6BgsrBgEEAbIxAQICBzArMCkGCCsG
AQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQUzAIBgZngQwBAgEw
OwYDVR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5jb21vZG9jYS5jb20vRXNzZW50
aWFsU1NMQ0EuY3JsMG4GCCsGAQUFBwEBBGIwYDA4BggrBgEFBQcwAoYsaHR0cDov
L2NydC5jb21vZG9jYS5jb20vRXNzZW50aWFsU1NMQ0FfMi5jcnQwJAYIKwYBBQUH
MAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTA/BgNVHREEODA2ghdjbG91ZGZs
YXJlY2hhbGxlbmdlLmNvbYIbd3d3LmNsb3VkZmxhcmVjaGFsbGVuZ2UuY29tMA0G
CSqGSIb3DQEBBQUAA4IBAQBlN1564xpz0f0EnCh5dKOjo6uk+kbLzEhkfaGd5Ydi
4diFQ9VYx3+Le1JCB/bDHMVUfwlqTpV0Eq8DZIWTO5wnP9BlRDiljVe7+y/jkQ/b
/B88kmBr2jjR9Aet1l8hOrqJycw6Ack6F+5hd/lYIvZ/0YH+h/qu9/Z6ii6rcUCd
UWERSKiTFsbM8PRmG/Cwb4Jm52N8ev6mcVYmxeBYIPmf51HBHEakN13oQcubCAjd
V9/8CugEMrl56lUpt7BYZMET2h4NyCDrfTlbFcDqQC+YBr5dLDOvLpe7T7Dv+r1P
wYJ+R0A4JC0F2RdUeIBWC5CycJcTx4h7ZSlNeWtFrZgJ
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
---
No client certificate CA names sent
---
SSL handshake has read 6784 bytes and written 376 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: EF16DB45C3D67F69A480645C5267C4FDC44F41FD4CF4911194E986FC21E72F62
    Session-ID-ctx:
    Master-Key: 9DF3223AAF1520D6437E643E83E4AD5B1A590776F375B7ED082E024F3EC9EB43617A0D1F7715DF299EA483F905095465
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - c5 00 41 79 f6 38 12 30-bf 5f 85 54 f7 93 09 1c   ..Ay.8.0._.T....
    0010 - c1 60 e2 23 ca 90 8f 17-0c 4a 9f db cc 40 0e ea   .`.#.....J...@..
    0020 - 55 b0 f8 49 f1 7e b0 4e-78 0f 36 4a 58 3a 60 e2   U..I.~.Nx.6JX:`.
    0030 - b4 2b 22 a2 49 e8 c5 42-d0 00 ad a6 ec 49 b3 4d   .+".I..B.....I.M
    0040 - 28 b1 c3 ad 03 c6 53 de-a3 e7 ec c8 aa ed 5e 97   (.....S.......^.
    0050 - 75 12 5e 9f 5f eb cf a9-4a ab b7 85 bf cd e0 12   u.^._...J.......
    0060 - 2c ec 0b 05 4f cf ac 16-e9 65 40 1b a8 60 dc 3a   ,...O....e@..`.:
    0070 - 99 a0 cf 7a 65 0b 4c 74-a5 fc a5 16 11 48 e2 94   ...ze.Lt.....H..
    0080 - 19 0e 17 a8 03 d0 d0 4b-a4 14 7e 49 05 75 36 65   .......K..~I.u6e
    0090 - d4 70 63 fa a7 92 5a 14-63 97 00 cf 6b 5b 45 36   .pc...Z.c...k[E6

    Start Time: 1397426832
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
GET /heartbleed HTTP/1.1
Host: www.cloudflarechallenge.com

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 13 Apr 2014 22:02:32 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=86400

f61
<!doctype html>
<html>
<head>
  <title>Heartbleed Challenge</title>

From the output we can see that:

• We connect to the server
• There are many packages exchange between the client (our openssl cli tool) and the web server; the packets types and formats are defined in the relevant RFC documents for SSL/TLS
• Option tlsextdebug instructs openssl to print out TLS extensions the server supports
• We can immediately see if the option is supported by our www server; what we have o do next is to check if the version of OpenSSL that we run is vulnerable or not 
• It is important to note that regardless if the www server supports the heartbeat extension or not you as a client can sent any legitimate HTTP requests; the whole problem is that if your client sent an heartbeat packet that was on purpose malicious the server in its response can reveal a lot more data that it should.

References

http://www.openssl.org/docs/apps/s_client.html
http://www.theregister.co.uk/2014/04/09/heartbleed_explained/
https://www.cloudflarechallenge.com/heartbleed

Can I use Shortest Path Bridging hardware to build my SDN network

Recently I've come across a document that compares a number of existing network overlays in SDN architecture: The 2013 Guide to Network Visualization and SDN.

What is new and interesting is the solution from Avaya. Instead of using VXLAN, STT and GRE like all other vendors they use SPB (we wrote about this here How does switch fabric network work) to build the SDN solution.

How does switch fabric network work

A network engineer can list a number of issues you can potentially run when using STP protocol  in your switch network. Over the years the network industry has created successor protocols like RSTP or MSTP. Both are improvements and offer much better convergence time and respond much quicker to switch topology changes. One of the major disadvantages for networks that relay on STP is the fact that they don't support multipathing. It means once network topology converges there will be blocked path between switches that are elected and managed by STP. This often redundant links can't be used because of a loop risk.

But there are better solutions today on the market to design better layer 2 Ethernet networks (more scalable, with higher throughput and with active link redundancy as an example). The 2 most popular are based on SPB and TRILL protocols. Both of them are used as a foundation in switch fabrics products. To better understand both of them the pictures below provide a side by side comparison. This was taken from Avaya document: Compare and Contrast SPB and TRILL.

Avaya is a SPB promoted so the comparison is a bit waited towards SPB but nevertheless it gives some inside view into both protocols.

References

http://cciethebeginning.wordpress.com/2008/11/20/differences-between-stp-and-rstp/
http://etherealmind.com/spb-attention/
http://en.wikipedia.org/wiki/IEEE_802.1aq
http://en.wikipedia.org/wiki/TRILL_(computing)
http://www.avaya.com/uk/resource/assets/whitepapers/SPB-TRILL_Compare_Contrast-DN4634.pdf
http://nanog.org/meetings/nanog50/presentations/Monday/NANOG50.Talk63.NANOG50_TRILL-SPB-Debate-Roisman.pdf
http://www.ebrahma.com/2012/06/trill-vs-spb-similarities-differences/
http://wikibon.org/wiki/v/Network_Fabrics,_L2_Multipath_and_L3