Akanda virtual router for virtual networks in Openstack

The Openstack virtual network, aka Neutron (old name Quantum) is responsible to deliver networking services for other cloud infrastructure projects. There is already quite a lot about it on the Internet and we reviewed couple of  possible deployment scenarios and use cases here (Emerging of the virtual network aka Quantum in Openstack) as well. Going further this route and discussing virtual networking opportunities it is a time to ask another question:

Does Openstack Virtual Network/Quantum/Neutron support virtual routers?

I don't have yet a very clear answer to this but found today this interesting project: https://github.com/dreamhost/akanda.

Looking at DreamHost wiki we can see that they use it to solve a problem of distributing and redirecting traffic in multi tenancy environment to secure, extend and control the networking traffic. As they claim it is being implemented with a help of virtual router, aka akanda.

http://wiki.dreamhost.com/DreamCompute_Overview

Network

Networking services for DreamCompute are delivered through OpenStack's Quantum service, coupled with Nicira's NVP (Network Virtualization Platform), and DreamHost's own Akanda project.

Nicira initiated and leads the Networking project team within the OpenStack community. Nicira's NVP, in concert with the OpenStack Quantum project, provides the ability to create private networks per tenant with high availability and network hardware independence. Nicira's NVP virtualizes the network; the software does for networks what hypervisors do for servers.

Another component to DreamCompute networking is the open source Akanda project developed by DreamHost. Akanda serves as a network traffic router (OSI Layer 3) for virtual networks created in DreamCompute. Akanda allows virtual networks to be configured to talk to each other, to LANs, to WANs or to the Internet.

The combination of OpenStack, Nicira and Akanda bring a level of network flexibility and security that sets DreamCompute apart from the competition.

How to build a data center

Working for a global hosting company can be a big fun and challenge every day. There is always something happening and changes are constant. But majority of us visit a data center only rarely or only at the beginning I believe. For these who would like to know a little more and understand why it may take a while to build a row in data center and fill it full with servers and network gear take a look at these example movies below. It is a good fun to watch ;).

Rackconnect load and performance tests

Rackconnect 

Rackconnect as a hybrid hosting solution that Rackspace has to offer that represents an excellent technology if you want to use the open cloud with your dedicated hardware.

When we think of dedicated hardware like servers performance statics are relatively easy to estimate or understand. This is different and be much more complicated for the IaaS cloud. Open cloud like IaaS is technology that is highly API driven and the performance results may differ base on the time of the day, physical location of the data center or the presence of the user.

It is to expected that the overall performance will be impacted as well as when we try to integrate with Open cloud. Rackconnect is a  hybrid technology that on one side uses the available cloud public API and on the other site manages the dedicated hardware. Cloud imposes a challenge for it because it can be inpredicatble, can have time spikes, timeouts or even local outages that can take from minutes to hours.

In this post I would like to summarize and list some of my experiments I did to better understand the cloud potential and as well as find areas of improvements for the Rackconnect itself.

Rackconnect load and performance tests articles:

1. Part1 - cloud build bursting performance from 1 to 5 cloud servers

A client uses Cisco ASA 5505 and core cloud account with a physical presence in London data center for Rackconnect testing.

1. Part2 - TO BE DONE
2. Extra - First post about this problem

References

http://www.rackspace.com/cloud/hybrid/
http://rtomaszewski.blogspot.co.uk/search/label/rackconnect

How long does it take to Rackconnect one to five cloud server on Cisco ASA 5505 (Part1)

Update:
There is a more comprehensive comparison and listing of available resources for the topic:
Rackconnect load and performance tests 

This post is a continuum of the previous one: How long does it take to rackconnect a newly built cloud server

The data below has been generated with the same script [1] that has been specifically modified to generate necessary statistics. The method used to measure a time for the Rackconnect build has been improved as well. With the new code we should get even better and more accurate results.

For the tests below the new measurement error should be around 7 seconds only. This has been mainly achieved with a modified script check_rackconnect.sh [2].

In basic the new code uses heuristic base on the bastion host local time stamps as well as the time stamps from the check_rackconnect.sh script. The final calculation of how long the Rackconenct build took is done later in log_status3 function [1].

All tests below are executed on the same Rackconnect environment: Cisco ASA 5505 and core cloud account. Every built cloud server has a configuration: flavor 1 (256MB), image type 112 (Ubuntu 10.04).

Test case #1

How long does it take to create 1 cloud sever and Rackconnect it.

A test below will simulate:

• Using cloud API create one cloud server 
• Monitor the cloud build 
• Once the cloud server is built start to monitor Rackconnect (RC) build 
• Once the RC build is done generate stats and delete the cloud server 
• Repeat the above cycle 10 times

The logs below show how to start the test and the resulting statistics.

$ python -u performance-single-cs.py -v -t 10 -s 1 -b pass@bastion -u user -k key  run | tee log.$(date +%s).txt
$ cat firstgen_rc_performance_report.1346017811.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 22:07:06.914565,     2012-08-26 22:11:30.591570, 263
     2,     2012-08-26 22:11:30.592652,     2012-08-26 22:16:00.184358, 269
     3,     2012-08-26 22:16:00.185239,     2012-08-26 22:20:29.710007, 269
     4,     2012-08-26 22:20:29.711945,     2012-08-26 22:24:53.598046, 263
     5,     2012-08-26 22:24:53.598958,     2012-08-26 22:29:41.288767, 287
     6,     2012-08-26 22:29:41.290309,     2012-08-26 22:33:40.750456, 239
     7,     2012-08-26 22:33:40.752308,     2012-08-26 22:37:40.368505, 239
     8,     2012-08-26 22:37:40.370067,     2012-08-26 22:41:39.680134, 239
     9,     2012-08-26 22:41:39.680827,     2012-08-26 22:46:03.081319, 263
    10,     2012-08-26 22:46:03.083230,     2012-08-26 22:50:11.637474, 248

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     213,     213,     214,     214,     244,     213,     213,     213,     214,     183

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      30,      42,      42,      29,      24,      23,      22,      22,      28,      48

A graphical representation of the above data.

 


Test case #2

 How long does it take to create 2 cloud severs and Rackconnect them.

As before the flowing test is going to simulate:

• Using cloud API create 2 cloud servers
• Monitor builds of the 2 cloud serves
• Once any of the cloud servers is built start to monitor the Rackconnect (RC) build
• Once the RC build is done save stats  and delete a cloud server
• Generate stats when all 2 cloud servers are deleted
• Repeat 10 times the above cycle

The logs below show how to start the test and the results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 2 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346020358.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 22:43:00.756231,     2012-08-26 22:48:06.580328, 305
     2,     2012-08-26 22:48:06.580876,     2012-08-26 22:53:06.387345, 299
     3,     2012-08-26 22:53:06.388921,     2012-08-26 22:57:45.312555, 278
     4,     2012-08-26 22:57:45.313452,     2012-08-26 23:02:42.359540, 297
     5,     2012-08-26 23:02:42.361089,     2012-08-26 23:08:06.284967, 323
     6,     2012-08-26 23:08:06.286676,     2012-08-26 23:13:06.233792, 299
     7,     2012-08-26 23:13:06.235701,     2012-08-26 23:18:06.165773, 299
     8,     2012-08-26 23:18:06.166642,     2012-08-26 23:23:02.864452, 296
     9,     2012-08-26 23:23:02.865955,     2012-08-26 23:27:35.655586, 272
    10,     2012-08-26 23:27:35.662204,     2012-08-26 23:32:38.403999, 302

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     216,     216,     216,     216,     186,     248,     247,     216,     216,     216
    2,     212,     213,     213,     212,     244,     213,     213,     213,     213,     213

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      62,      44,      19,      69,      32,      46,      46,      68,      19,      69
    2,      55,      62,      55,      29,      51,      47,      40,      28,      50,      35

A graphical representation of the test #2 data.
 



Test case #3

How long does it take to create 3 cloud severs and Rackconnect them.

The logs below show how to start the test and the resulting statistics

 
$ python -u firstgen_cs_performance.py -v -t 10 -s 3 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346023741.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 23:36:24.625215,     2012-08-26 23:41:45.844673, 321
     2,     2012-08-26 23:41:45.846358,     2012-08-26 23:47:07.165236, 321
     3,     2012-08-26 23:47:07.166380,     2012-08-26 23:51:55.442095, 288
     4,     2012-08-26 23:51:55.443153,     2012-08-26 23:57:13.466668, 318
     5,     2012-08-26 23:57:13.468356,     2012-08-27 00:02:58.579180, 345
     6,     2012-08-27 00:02:58.580118,     2012-08-27 00:08:19.777111, 321
     7,     2012-08-27 00:08:19.778890,     2012-08-27 00:13:37.742591, 317
     8,     2012-08-27 00:13:37.744408,     2012-08-27 00:18:55.713891, 317
     9,     2012-08-27 00:18:55.715903,     2012-08-27 00:23:46.616462, 290
    10,     2012-08-27 00:23:46.617623,     2012-08-27 00:29:01.689222, 315

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     220,     188,     219,     220,     219,     219,     219,     219,     220,     220
    2,     216,     215,     215,     216,     215,     215,     215,     247,     216,     216
    3,     212,     211,     212,     212,     211,     242,     242,     212,     212,     212

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      35,      98,      17,      44,      93,      36,      38,      42,      15,      79
    2,      64,      36,      55,      81,      67,      58,      32,      40,      60,      25
    3,      87,      60,      57,      47,      39,      57,      52,      36,      56,      63

Like before these are the data visualised on graphs
 


Test case #4

How long does it take to create four cloud severs and Rackconnect them.

The logs below show how to start the test and results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 4 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346071307.txt 
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-27 12:42:58.464643,     2012-08-27 12:48:51.826041, 353
     2,     2012-08-27 12:48:51.827840,     2012-08-27 12:54:51.083906, 359
     3,     2012-08-27 12:54:51.085615,     2012-08-27 13:00:59.075276, 367
     4,     2012-08-27 13:00:59.076493,     2012-08-27 13:06:37.176187, 338
     5,     2012-08-27 13:06:37.177456,     2012-08-27 13:12:24.448441, 347
     6,     2012-08-27 13:12:24.449740,     2012-08-27 13:18:02.517695, 338
     7,     2012-08-27 13:18:02.519349,     2012-08-27 13:23:52.897831, 350
     8,     2012-08-27 13:23:52.898722,     2012-08-27 13:29:58.276944, 365
     9,     2012-08-27 13:29:58.278082,     2012-08-27 13:35:42.351643, 344
    10,     2012-08-27 13:35:42.352320,     2012-08-27 13:41:47.521873, 365

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     224,     224,     224,     224,     224,     224,     192,     257,     195,     225
    2,     157,     222,     222,     221,     221,     222,     189,     222,     225,     254
    3,     249,     219,     250,     219,     218,     219,     217,     219,     221,     219
    4,     215,     216,     216,     216,     215,     216,     245,     248,     218,     216

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      58,      62,      90,      91,      42,      48,      33,     100,      43,      82
    2,      59,      23,      71,      37,      93,      87,      63,      12,      89,      83
    3,      70,      41,      90,      40,      89,      81,     112,      44,      49,      21
    4,      61,     107,      45,      71,      84,      41,      55,      62,      94,      51

Like before these are the data visualised in a form of a graphs
 


Test case #5

How long does it take to create five cloud severs and Rackconnect them.

The logs below show how to start the test and results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 5 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
cat firstgen_rc_performance_report.1346075296.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-27 13:46:48.810109,     2012-08-27 13:53:00.032992, 371
     2,     2012-08-27 13:53:00.034355,     2012-08-27 13:59:41.839530, 401
     3,     2012-08-27 13:59:41.840434,     2012-08-27 14:05:53.244177, 371
     4,     2012-08-27 14:05:53.245105,     2012-08-27 14:11:55.703266, 362
     5,     2012-08-27 14:11:55.704239,     2012-08-27 14:18:19.149206, 383
     6,     2012-08-27 14:18:19.150468,     2012-08-27 14:24:33.617616, 374
     7,     2012-08-27 14:24:33.618816,     2012-08-27 14:30:51.220322, 377
     8,     2012-08-27 14:30:51.221913,     2012-08-27 14:36:53.563330, 362
     9,     2012-08-27 14:36:53.564169,     2012-08-27 14:42:31.914619, 338
    10,     2012-08-27 14:42:31.915871,     2012-08-27 14:48:16.481674, 344

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     228,     228,     227,     194,     195,     228,     228,     227,     259,     228
    2,     225,     192,     225,     224,     256,     225,     226,     192,     192,     225
    3,     222,     222,     222,     222,     221,     223,     254,     222,     189,     223
    4,     220,     220,     250,     219,     218,     251,     188,     250,     217,     220
    5,     217,     217,     248,     246,     247,     215,     217,     216,     214,     217

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      36,      54,      30,      44,      64,      30,      61,      48,      43,      44
    2,      52,      56,      54,     103,      97,      80,      50,      43,      51,      67
    3,      97,     166,      58,      42,      93,      68,      90,      52,      58,      69
    4,     114,      42,      71,     101,      53,      89,      55,      65,      49,     100
    5,      67,      74,      88,      73,      60,      80,      93,     113,     101,      67

Like before these are the data visualised in a form of a graph
 



Summary and results description

Through all the 5 test cases above we have been creating cloud servers and repeating this process 10 times. The graphs show that the numbers are changing and the visible trend is that the times are increasing.

We take a look at each of the 5 test cases again and summarize the 10 repetitions to a 3 number result: min, max and average Rackconnect build time. Next we can summarize all the 5 tests/50 cloud build tests and represent all the data in a single graph. The graph below compares all above tests.


As an example, for the test 1 we have the min, max and average times. It has been calculated base on the results from the Test Case #1. We did the some for the other Test Cases.

We see that every time we increase a number of cloud server to build by one the rackconnect build time increases as well. It means that if we burst 1, 2 and up to 5 cloud servers in one single test the time needed for the cloud infrastructure to provision and than to finish up and RackConnect a single cloud server increases as well.

References

1. https://github.com/rtomaszewski/cloud-performance/tree/nextgenv1.0
2. https://github.com/rtomaszewski/cloud-performance/blob/nextgenv1.0/check_rackconnect.sh
3. http://www.rackspace.com/cloud/hybrid/

How to check or monitor build status of a cloud server that belong to a RackConnect cloud account

There is a difference when you use cloud account that is linked with Rackspace RackConnect (RC) product.

Every cloud server from a cloud account that is rackconnected is gong to be reconfigured. All tasks that the RackConnect system will execute can be seen and followed on MyRackspace portal. In short these tasks will change the initial IP settings, route configuration and firewall settings on the original cloud server.

• RC tasks from MyRackspace portal

Cloud Server Created: Add "rackconnect" user 
Cloud Server Created: Validate existence of gateway interface on dedicated network device 
Cloud Server Created: Retrieve metadata 
Cloud Server Created: Provision public IP address 
Cloud Server Created: Update access on dedicated network devices 
Cloud Server Created: Configure network stack 
Cloud Server Created: Configure software firewall 
Cloud Server Created: Update software firewall on other Cloud Servers

• Status and monitoring

At the moment the only way to know that the RC is done is to monitor the network settings on the cloud server manually. This is a known limitation and there are going to be new changes deployed to address this in a close future. For now to know that the RC is done we can for example monitor the last possible task: Cloud Server Created: Configure software firewall.

As soon as we know that the firewall config has changed the RC is done (almost done because there is one last task that can still be something that affects our cloud server). A simple example how the settings change is below.

• Before the RC changes

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

• After the RC changes

# iptables -nL
Chain INPUT (policy DROP)
target     prot opt source               destination         
RS-RackConnect-INBOUND  all  --  0.0.0.0/0            0.0.0.0/0           /* RackConnectChain-INBOUND */ 

Chain FORWARD (policy DROP)
target     prot opt source               destination         
RS-RackConnect-INBOUND  all  --  0.0.0.0/0            0.0.0.0/0           /* RackConnectChain-FORWARD */ 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RS-RackConnect-INBOUND (2 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED /* RackConnectChain-INBOUND-RE */ 
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           /* Local-Loopback */ 
...

To know that the RC is done you need some simple script to check this. An example bash script is listed below.


That means the you can use the example script above and run it as many times as you want (in some loop with delays between the executions). As soon as the cloud server will be rackconnected the script output will turn into 'yes'.

Working for the best in the industry

Technology leaders

I have recently start doing more analysis and  research of the market trends. It is extraordinary good feeling to know that you can volunteer for a work at the best companies worldwide ;). First it was F5 Networks and now with Rackspace. The blend of Load balancing, Cloud and Linux technologies is going to be the future foundation for next generation data center solution architectures.

References

www.f5.com
www.rackspace.com
www.openstack.org
http://www.diigo.com/user/rtomaszewski/magic_quadrants%20f5