Overlay technologies in data center

Everyone speaks about SDN an the benefits its brings when deploying cloud or enterprise infrastructures. But do we actually know or have any understanding what this all SDN is about? If you want be fluent in the language of virtual networking and network overlays in modern data centers you need to understand at least the following concepts:

• SDN controller (Northbound and Southbound API) (an example controller could be VMware NVP/NSX or OpenDaylighta; a list of vendor specific Northbound API)
• Openflow as Southbound API (example flow in Open vSwitch)
• Network Functions Virtualization
• Overlay and tunnels concept for cloud network inside data center (example protocols are VXLAN and STT)

In the remaining of the post we will concentrate solely on existing overlay technologies. These information was extracted from Cisco doc: Cisco Nexus 9000 Series Switches - Data Center Overlay Technologies).

Network-Based Overlay Networks

1. IEEE 802.1ad Provider Bridging or IEEE 802.1q Tunneling also known as IEEE 802.1QinQ or simply Q-in-Q
2. IEEE 802.1ah Provider Backbone Bridges (PBB) or Mac-in-Mac Tunnels
3. Cisco FabricPath allows multipath networking at Layer 2
4. TRILL - IETF Transparent Interconnection of Lots of Links is a Layer 2 multipathing technology
5. Shortest-Path Bridging (SPB) is defined in IEEE 802.1aq and is targeted as a replacement for Spanning Tree Protocol (example info based on Avaya documentation)
6. Cisco Overlay Transport Virtualization (OTV) is a Layer 2-over-Layer 3 encapsulation "MAC-in-IP" technology
7. The Cisco Location/Identifier Separation Protocol (LISP) is currently defined as a Layer 3 overlay scheme over a Layer 3 network
8. Multiprotocol Label Switching (MPLS)
9. Virtual Private LAN Service (VPLS) a Layer 2 tunneling protocols
10. Virtual Private Routed Network (VPRN) also known as BGP/MPLS or IP-VPN provides IP VPN services

Host-Based Overlay Networks

1. Virtual Extensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3 networ that uses IP/UDP encapsulation
2. Network Virtualization Using Generic Routing Encapsulation (NVGRE) allows creation of virtual Layer 2 topologies on top of a physical Layer 3 network
3. Stateless transport tunneling (STT) is an overlay encapsulation scheme over Layer 3 networks that use a TCP-like header

Can I use Shortest Path Bridging hardware to build my SDN network

Recently I've come across a document that compares a number of existing network overlays in SDN architecture: The 2013 Guide to Network Visualization and SDN.

What is new and interesting is the solution from Avaya. Instead of using VXLAN, STT and GRE like all other vendors they use SPB (we wrote about this here How does switch fabric network work) to build the SDN solution.

How does the VXLAN protocol work

The VXLAN is one of the overlay network tunneling protocols that is used to built network infrastructure for cloud environment. Below are some details about the operation and specification.

• Frame headers definition

• The traffic between VMs is encapsulated in IP/UDP packets
• Logical isolation is implemented in a form of logical overlay where the traffic is exchanged between encryption tunnels endpoints
• VXLAN ID is used to identify the specify isolated L2 cloud network that belongs to a tenant
• The tunnel endpoints represent the edge of the cloud network infrastructure
• The tunnel endpoints perform encapsulation and decapsulation
• It is there where all the logic is implemented to find out where to sent next a packet or to witch VM the packet should be delivered after decapsulation

• A comprehensive summary and operational features can be found under the links in reference section, below are few of the main characteristics and benefits:
• It operates over IP and used UDP to carry payload 
• Multicast support is the only other requirement for switches and routers to support VXLAN 
• Multicast is used to handle L2 broadcast traffic (like ARP requests)
• Logical networks can be extended among virtual machines placed in different Layer 2 domains

References

1. http://www.definethecloud.net/vxlan-deep-dive
2. http://www.definethecloud.net/vxlan-deep-divepart-2
3. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-685115.html
4. http://www.emulex.com/artifacts/d658610a-d3b6-457c-bf2d-bf8d476c6a98/elx_wp_all_VXLAN.pdf
5. http://blogs.cisco.com/datacenter/more-vxlan-qa/
6. http://blog.scottlowe.org/2011/12/07/revisiting-vxlan-and-layer-3-connectivity/
7. http://blog.scottlowe.org/2011/12/22/otv-and-vxlan-layer-3-connectivity-compared/

Performance analysis of network tunnels in SDN cloud network

Additional network overlay is the foundation and building block for most modern cloud network architectures today. In practice it means that before we can even think how to architect and build network for the cloud we need to build a solid and reliable multi-tiered IP network topology to interconnect our hypervisors servers. Of course this is a big simplification and there are many vendors that provide hardware support for cloud network (SDN enabled network). Examples are Brocade VCS/MLX or Cisco Nexus platform/Random thoughts about Cisco nexus product line.

But what is important is that in essence what we are going to built will be a typical multi-tier network with access, distribution and core layers like this example below:

Once the network is built there is now time for the cloud network element to be added. This is again very simplistic view to avoid all the technical details.

The industry is still working to established a common ground and consensus how a cloud network should look like and what services it should provide but in practice (base on a few companies like Nicira or Midokura) it is tightly associated with  Software defined networking (SDN) concept and architecture. And the common practice today is to implement SDN network as an additional network overlay on top of IP fabric infrastructure.

Like every network, cloud network needs to provide IP connectivity for cloud resources (cloud servers for example). Often to achieve this al hypervisors are inter-connected using tunneling protocols. This model allow us to decouple the cloud network from the physical one and allow more flexibility. That way all VMs traffic is going to be routed within the tunnels. To solve the cloud network problem is to find a solution how to route between the hypervisors using the tunnels.

Data flows, connections and tunnels are manged by cloud controller (a distributed server cluster) that need to be deployed in our existing physical network. An example of Nicira NVP controller can be found here: Network Virtualization: a next generation modular platform for the data center virtual network.

As we agreed VMs data will be routed within tunnels. These are the most popular ones: NVGRE, STT and VXLAN (Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure).

As tunnels require additional resources there is an open question what overhead, resource consumption and performance implication will they represent. This post: The Overhead of Software Tunneling (*), do a comparison and tries to shed some more light on the topic.

	Throughput	Recv side cpu	Send side cpu
Linux Bridge:	9.3 Gbps	85%	75%
OVS Bridge:	9.4 Gbps	82%	70%
OVS-STT:	9.5 Gbps	70%	70%
OVS-GRE:	2.3 Gbps	75%	97%

This next table shows the aggregate throughput of two hypervisors with 4 VMs each.

	Throughput	CPU
OVS Bridge:	18.4 Gbps	150%
OVS-STT:	18.5 Gbps	120%
OVS-GRE:	2.3 Gbps	150%

We can see that not all tunnels are completely transparent when it comes to performance. The GRE tunnel shows a significant degradation in throughput. The TCP based STT tunnel works fine although  For a complete analysis, explanation and further discussion I recommend to read the blog above (*).

References

1. http://rtomaszewski.blogspot.co.uk/2012/12/what-do-you-need-to-implement-virtual.html
2. http://rtomaszewski.blogspot.co.uk/2012/11/what-network-topologies-can-i-build.html
3. http://rtomaszewski.blogspot.co.uk/2012/11/after-server-virtualization-there-is.html
4. http://rtomaszewski.blogspot.co.uk/2013/05/sdn-system-software-architecture.html
5. http://bradhedlund.com/2012/10/06/mind-blowing-l2-l4-network-virtualization-by-midokura-midonet/
6. http://blog.ioshints.info/2012/08/midokuras-midonet-layer-2-4-virtual.html

Introduction into tunneling protocols when deploying cloud network for your cloud infrastructure

Cloud network is a hot topic for cloud providers and hosting companies. In basic the concept should enable and allow tenants to create, manage and destroy network typologies on demand for the cloud servers by using cloud open API. That said, we want to allow a tenant to create an isolated virtual layer 2 network with its own IP subnet.

Before going into further details we have to realize that the problem isn't trivial to solve. The difficulty comes from a fact that the existing network that interconnects hypervizors hosts is not very flexible and adaptable for changes. That physical network architecture was build and tune to  allow to handle all traffic from all cloud VM across you cloud deployment. This represent its strength and limitation as it is not flexible enough when it comes to configure and create many isolated virtual layer 2 or layer 3 networks per single tenant. This is exactly the problem that the cloud network promises to resolve.

At the moment there isn't a single standard how to implement a cloud network. Instead, we have 3 different protocols that were proposed: VXLAN, NVGRE, STT [1].

All these protocols relay on the fact that the hypervisor hosts are interconnected  The implementation of the additional features is done by using tunneling mechanisms. All of them implement a L2 in L3 tunnels by using TCP, UDP or IP datagrams.

A short introduction and more explanation how this works can on the screenshots below that were taken from this video:  Video: Cloud Tunnels @ Cloud Mafia ( slies can be found here http://ifup.org/slides/cloud-tunnels/ )

References

1. VXLAN
2. NVGRE
3. STT