How to build a high performance network appliance like routers using commodity hardware and off the shelf components

You can assemble a server from off the shelf components that will be able to sent and receive traffic in multi Gigabit speed. Here is an example of an 10Gps net card from Intel.

But can we turn this server into a high performance network appliance? Do we still need a dedicated hardware like for example ASIC, FPGA, low latency RAM and TCAM RAM in network devise so they can efficiently switch and forward packets with maximum wire speed.

Router hardware design plan

Looking at this presentation from 2012 https://ripe64.ripe.net/presentations/18-ripe-64-router-architecture-challenges.pdf you would think that yes. These would be the obvious reasons (screenshots taken from the presentation):

Network processing unit (NPU) and new hardware design

The key points listed above still hold. But the next generation network appliances will be rather build with a help of a multicore generic NPU using the power of parallel processing than expensive and purposely design ASIC. With the right software (OS - often Linux, drivers, firmware, SDK, and API libraries) you will be able to turn a conventional x86 server with a modern PCIe data bus into a high performance, low latency and high speed network appliance.

Netronome Network Cards Accelerate SDN and NFV Designs
100Gps FlowNIC-6xxx network card
Hardware reference designs for FlowProcessor NPU chips

Openstack reference architecture using Cisco UCS hardware platform

There are many vendors to chose from when selecting your hardware for a complete Openstack deployment (example list of data center friendly networking vendors to look at ). And to make it even more difficult you need to think about all spectrum of vendors like networking, storage and compute.

In one of my previous posts (How to estimate hardware requirements for your private Openstack deployment) I tried to demonstrate an example hardware recommendation for an Openstack deployment. Today we are going to look at this topic once again but exploring the Cisco UCS product line instead.

The data below are taken from the Cisco PDF white paper Red Hat Openstack Architecture on Cisco UCS platform from the DesignZone for Cloud Automation Solution section on Cisco site (http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-cloud-computing/could_automation.html).

Openstack on Cisco UCS hardware platform

Cisco is no longer only a networking vendor. With the UCS they offer as well computer platform where you can put together a server with specific hard drive size,  mount of RAM or type of CPU, interconnection card etc. An example configuration taken from the Cisco document above:

That means if we put togheter the UCS servers and the Cisco Nexus switches and Openstack software we can build a simple POC like this one:

In the white paper we can actually find a full list of hardware if you would like to build it yourself.

Nested virtualization support on Rackspace public cloud

We have found the CPU hardware architecture that the public cloud is running on in this blog: Hypervisor hardware differences on Openstack Rackspace Cloud.

Problem

Does Rackspace public cloud support nested visualization?

Results discussion

• Public Cloud

Of course for the cloud to exists the physical sever where the hypervisor runs (Xen or KVM for example) needs to have in-hardware virtualization support (Intel VT-x or AMD-V). This is the only way to provide a high performance cloud servers.

But once the cloud server boots up the cloud virtual CPU no longer exports the hardware CPU virtualization capabilities. You can verify this with this little script below.

egrep -i 'vmx|svm|ept|vpid|npt|tpr_shadow|flexpriority|vnmi'

That means you can't use your cloud server to run another, a guest hypervisor ( called as well nested hypervisor).

• Private cloud 

The nested virtualization can be enabled. As an example this link describe some of the steps for Linux KVM:

•  Installing Rackspace Private Cloud - KVM.
• http://networkstatic.net/nested-kvm-hypervisor-support/

Another solution

If your cloud server doesn't offer nested virtualization support you can always use the emulation mode. Qemu supports running VM that way.

References

http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/
https://www.diigo.com/user/rtomaszewski/nested_virtualization?type=all&snapshot=no&sort=updated
http://en.wikipedia.org/wiki/X86_virtualization

Hypervisor hardware differences on Openstack Rackspace Cloud

You can spin up test cloud servers and extract CPU flags with the help of this little script using csplit.

 
cat /proc/cpuinfo | csplit -z  - '/processor/' '{*}'
diff xx0*
grep flags xx01 | cut -d ':' -f 2 | xargs -n1 echo | sort > flags.txt

By comparing the results we can definitely say that:

• Performance1 and performance2 cloud servers are running on the same hardware.
• New performance cloud servers are hosted on Intel CPU.
• The standard (next generation) series are being hosted on AMD CPU.

The CPU flags for comparison.

 
processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : Quad-Core AMD Opteron(tm) Processor 2374 HE
stepping        : 2
microcode       : 0x1000086
cpu MHz         : 2200.096
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good nopl pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch hw_pstate
bogomips        : 4400.19
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2600.068
cache size      : 20480 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 43
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes hypervisor lahf_lm arat pln pts dtherm
bogomips        : 5200.13
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

The successful architecture for a top-of-rack switch for data center

TOR switch architecture

We wrote before about Arista switches and about the Arista EOS architecture (network OS). A company with a name Pica8 is another example that follows a very similar technological model. They use the best out of the Linux and butter this up with some more hardware (ASIC) dependent software to achieve maximal performance.

What is interesting for Pica8 is that they take a very liberal approach to the hardware itself. They say they could use any plain switching chip or motherboard blades and turn it into a fully operational switch. The secret is once again a well design Pica8 network OS they built.

In comparison to Arista CLI (that is very alike the Cisco one) Pica8 uses a rather different syntax: http://pica8.org/blogs/?p=399. At first glance it has some similarities to what you typie on Juniper boxes :).

More info about them and products can be found here:
http://www.networkworld.com/news/2010/102810-pica8-opensource-switching.html?page=1
http://www.pica8.com/open-switching/1-gbe-10gbe-open-switches.php

How to estimate hardware requirements for your private Openstack deployment

I've found this little tool on the Mirantis web page: http://www.mirantis.com/openstack-services/bom-calculator/. You  can play with it to estimate how much hardware (and money as well :)) you need to build your own private Openstack cloud infrastructure. It gives as well as an overview about potential vendors and theirs hardware.

Simple example

Top 10 data center network vendor list

In previous post was saw how handful the Gartner magic quadrants can be when learning and researching particular technology or industry trend:

• New 2012 Gartner Magic Quadrant for Cloud Infrastructure as a Service (Iaas)
• Rackspace and Brocade pushing ADX platform

Below is an another comparison chart, this time targeting vendor network equipment in data centers.

By the way I was investigating this because of the networks vendor selection on this cloud hardware deployment calculator (How to estimate hardware requirements for your private Openstack deployment ) : Dell, Cisco, HP, Arista, Juniper, Brocade.

SR-IOV technology enables low level network virtualization

In the virtualization space the SR-IOV technology was introduced in about 2008/2010 [1]. The technical details can be found under the links in the reference section but in plain English the technology allows to create many virtual devices base on single a physical device. For this to work the hardware (CPU, North chip) and operating system need to have SR-IOV support.

Below is a video demonstrating packet processing for an Intel Ethernet card that supports SR-IOV.

Interesting slides showing the concept from the video and  reference links:

• After the frame enters the physical port on the NIC the low level driver/firmware (supporting the SR-IOV) distributes the packet (based on header classifications/hash value/etc) to separate virtual queues 

• Each virtual queue is assigned directly to a virtual device 
• Once the packet is in the queue it can be deliver to the VM DIRECTLY without the usual software hypervisor overhead
• Packets don't have to be copied from physical port buffer(s) to OS RAM and than from OS RAM to VM OS buffers. The data can be sent directly from the physical port to VM OS buffers. That way the hypervisor processing overhead can be minimised.

• Critical part for the technology is CPU and chipset virtualization support
• As access to physical RAM need to be protected between hypervisor and VMs as well as VMs themselves the virtual memory address is translated to the physical location by the north chipset
• For the DMA request to copy the packets the address translation between the hypervisor address space and the VM address space is transparent (north chip take care of it)

• Another view how the packet is delivered from physical port to the VM

References

1. http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf
2. http://www.intel.com/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf
3. http://communities.intel.com/community/wired/blog/2010/09/07/sr-iov-explained

What OS is used to manage data centre class network appliances from Dell

Dell acquired Force10 in 2011 and that way gained access to data center networking technologies like 10 and 40 Gigabit Ethernet switches.

Like many hardware devices a network appliance needs an (network) operating system (NOS) to control resources and to provide monitoring and management interface. Base on the publicly available information a simplified view of a hardware and software architecture for Force10 network devices can be described as:

• Packet processing is accelerated with the help of an modern ASIC chip(s)
• There is a low level microcode/firmware ( operating system driver) to manage the ASIC
• All of this is integrated within a Linux or *BSD (NetBSD) operating system
• Dell uses 2 different NOS: Dell Networking Operating System (DNOS) or Force10 Operating System (FTOS) depending on the hardware

References

1. http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Force10_FTOS_Spec_Sheet.pdf
2. http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Force10_Product_Quick_Reference_Guide.pdf

This is more like a bonus material in reference to [1] how to build a network using the latest Z9000 switches from Dell: http://bradhedlund.com/2012/01/25/construct-a-leaf-spine-design-with-40g-or-10g-an-observation-in-scaling-the-fabric/

Arista is recognized as one of the main data center networking vendor

As data centers architecture is transforms by the evolution driven by technologies like OpenFlow, SDN and cloud virtualization it is important who are the main players on the market. Below is a snapshot from the latest Magic Quadrant for Data Center Network Infrastructure showing the main vendors:

We can see that Arist is listed as one of the vendors along big market giants like Cisco, HP, Juniper, Dell, Brocade and others.

References

1. http://www.aristanetworks.com/media/system/pdf/AristaProductQuickReferenceGuide.pdf
2. http://www.theregister.co.uk/2012/09/19/arista_networks_7150s_switches/
3. https://eos.aristanetworks.com/home.php

Dual sim android phone from Samsung

According to this link Dual-SIM Samsung Galaxy S 4 launches in China with an Exynos 5 Octa inside there is going to be a model of the Samsung Galaxy S4 that can be used with 2 sim cards. How much I wish that they release this in Europe as well...

But looking at the hardware specification Model variants there is not much difference between the European/International version GT-I9500 or GT-I9505 and the model for the Chinese market GT-I9502. From a pure hardware specification point of view the GT-I9502 may work well in Europe with all networks up to 3G. But before deciding it is important to look at 4G (LTE) as this depends on the cpu type or hardware version.

The hard question you need to ask your self:

• What CPU model do I want to have
• Should the phone support 4G or not

I believe I will wait a little bit to see and read more about this Chines model or maybe there is going to be another interesting phone on the market in the next quarter ...

References

1. http://www.sammobile.com/2013/03/20/your-country-will-get-exynos-or-snapdragon-variant-of-the-galaxy-s-4-we-have-the-answer/
2. http://www.sammobile.com/2013/05/02/samsungs-octa-core-not-as-energy-efficient-as-we-thought/
3. http://forum.xda-developers.com/showthread.php?t=2191850
4. http://www.engadget.com/2013/05/06/samsung-galaxy-s4-octacore-review/
5. http://www.engadget.com/2013/04/26/dual-sim-samsung-galaxy-s-4-launches-in-china/
6. http://www.engadget.com/2013/03/29/samsung-exynos-octa-now-rocking-lte-destined-to-korean-market/
7. http://www.sammobile.com/2013/03/28/samsung-exynos-confimed-lte-exynos-5-octa/

How to build a data center

Working for a global hosting company can be a big fun and challenge every day. There is always something happening and changes are constant. But majority of us visit a data center only rarely or only at the beginning I believe. For these who would like to know a little more and understand why it may take a while to build a row in data center and fill it full with servers and network gear take a look at these example movies below. It is a good fun to watch ;).

Proprietary AMD graphics drivers for Linux

How do I start Catalyst Control Center (CCC) from bash

I created a demo user to test my X server config and ATI graphic driver. I run into a problem that my user didn't have relevant permissions to run commands with sudo. Every time I tried to lunch the Catalyst Control Centre I got a popup windows asking for password to perform administrative task.

Solution

1. The name of the program is displayed in the popup windows: amdcccle
2. If you didn't notice you can find this using these methods

$ dpkg -l | grep ati | egrep -iv 'configuration|application|automatic|compatible|compatibility|static|foomatic|ating|ation|ative' | grep ati
$ dpkg -l | grep amd

$ dpkg -L fglrx-amdcccle | grep bin
/usr/lib/fglrx/bin
/usr/lib/fglrx/bin/amdcccle
/usr/lib/fglrx/bin/amdxdg-su
/usr/lib/fglrx/bin/amdupdaterandrconfig

How do verify that my driver is loaded

More checks can be found in [1]. As a simple check you can run:

$ lsmod | grep fglrx
$ dmesg | grep fglrx

Execpt CCC what command can I used to list, print and change the graphic driver settings

You can use a command 'aticonfig'.  Example output can look like this:

$ aticonfig --lsa --odgc
* 0. 06:00.0 ATI Radeon HD 5700 Series

$ aticonfig --odgt
Default Adapter - ATI Radeon HD 5700 Series
                  Sensor 0: Temperature - 37.00 C

$ aticonfig --odgc
Default Adapter - ATI Radeon HD 5700 Series
                            Core (MHz)    Memory (MHz)
           Current Clocks :    850           1200
             Current Peak :    850           1200
  Configurable Peak Range : [600-960]     [1200-1445]
                 GPU load :    0%

References

1. Unofficial Wiki for the AMD Linux Driver
http://wiki.cchtml.com/index.php/Frequently_Asked_Questions

2. http://support.amd.com/us/gpudownload/linux/Pages/radeon_linux.aspx

Who owns the most servers

The cloud is expanding and changing the landscape in data centers. It would be interesting to know who has the most servers on the world, isn't it? The links below may not be fully up to date but they still revel pretty good overview about the industry. The most interesting part is the difference between the Google #1 and the rest of the world.

References

1. http://www.datacenterknowledge.com/archives/2009/05/14/whos-got-the-most-web-servers/
2. http://gizmodo.com/5517041/googles-insane-number-of-servers-visualized
3. http://news.netcraft.com/archives/2012/09/10/september-2012-web-server-survey.html