Nested virtualization support on Rackspace public cloud

We have found the CPU hardware architecture that the public cloud is running on in this blog: Hypervisor hardware differences on Openstack Rackspace Cloud.

Problem

Does Rackspace public cloud support nested visualization?

Results discussion

• Public Cloud

Of course for the cloud to exists the physical sever where the hypervisor runs (Xen or KVM for example) needs to have in-hardware virtualization support (Intel VT-x or AMD-V). This is the only way to provide a high performance cloud servers.

But once the cloud server boots up the cloud virtual CPU no longer exports the hardware CPU virtualization capabilities. You can verify this with this little script below.

egrep -i 'vmx|svm|ept|vpid|npt|tpr_shadow|flexpriority|vnmi'

That means you can't use your cloud server to run another, a guest hypervisor ( called as well nested hypervisor).

• Private cloud 

The nested virtualization can be enabled. As an example this link describe some of the steps for Linux KVM:

•  Installing Rackspace Private Cloud - KVM.
• http://networkstatic.net/nested-kvm-hypervisor-support/

Another solution

If your cloud server doesn't offer nested virtualization support you can always use the emulation mode. Qemu supports running VM that way.

References

http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/
https://www.diigo.com/user/rtomaszewski/nested_virtualization?type=all&snapshot=no&sort=updated
http://en.wikipedia.org/wiki/X86_virtualization

What do you need to implement virtual network and build hybrid cloud

To build an infrastructure that can be used to host a hybrid cloud environments or to benefit from the flexibility that a virtual networking provides we need software and hardware components. Below are couple of links I found when researching on this topic.

VMware ESXi/vSphere

• vSphere Distributed Switch
• Overview of vNetwork Distributed Switch concepts

Citrix XenServer

• Overview of XenServer Distributed Virtual Switch/Controller and Troubleshooting Network Issues

Opensource Linux alternatives

• Open vSwitch (IBM introduction and good overview of the product Virtual networking in Linux )
• Virtual Square / vde switch - Example how to use it with Rackspace Private Cloud Installing Rackspace Private Cloud - KVM or with  Virtualbox
• Linux bridge

Microsoft Hyper-V

• Hyper-V Virtual Switch Overview in Windows 2012

Vendors

• bigswitch networks
• nicira 
• 10 Next-Gen Networking Startups Ripe For The Plucking

Rackconnect load and performance tests

Rackconnect 

Rackconnect as a hybrid hosting solution that Rackspace has to offer that represents an excellent technology if you want to use the open cloud with your dedicated hardware.

When we think of dedicated hardware like servers performance statics are relatively easy to estimate or understand. This is different and be much more complicated for the IaaS cloud. Open cloud like IaaS is technology that is highly API driven and the performance results may differ base on the time of the day, physical location of the data center or the presence of the user.

It is to expected that the overall performance will be impacted as well as when we try to integrate with Open cloud. Rackconnect is a  hybrid technology that on one side uses the available cloud public API and on the other site manages the dedicated hardware. Cloud imposes a challenge for it because it can be inpredicatble, can have time spikes, timeouts or even local outages that can take from minutes to hours.

In this post I would like to summarize and list some of my experiments I did to better understand the cloud potential and as well as find areas of improvements for the Rackconnect itself.

Rackconnect load and performance tests articles:

1. Part1 - cloud build bursting performance from 1 to 5 cloud servers

A client uses Cisco ASA 5505 and core cloud account with a physical presence in London data center for Rackconnect testing.

1. Part2 - TO BE DONE
2. Extra - First post about this problem

References

http://www.rackspace.com/cloud/hybrid/
http://rtomaszewski.blogspot.co.uk/search/label/rackconnect

How long does it take to Rackconnect one to five cloud server on Cisco ASA 5505 (Part1)

Update:
There is a more comprehensive comparison and listing of available resources for the topic:
Rackconnect load and performance tests 

This post is a continuum of the previous one: How long does it take to rackconnect a newly built cloud server

The data below has been generated with the same script [1] that has been specifically modified to generate necessary statistics. The method used to measure a time for the Rackconnect build has been improved as well. With the new code we should get even better and more accurate results.

For the tests below the new measurement error should be around 7 seconds only. This has been mainly achieved with a modified script check_rackconnect.sh [2].

In basic the new code uses heuristic base on the bastion host local time stamps as well as the time stamps from the check_rackconnect.sh script. The final calculation of how long the Rackconenct build took is done later in log_status3 function [1].

All tests below are executed on the same Rackconnect environment: Cisco ASA 5505 and core cloud account. Every built cloud server has a configuration: flavor 1 (256MB), image type 112 (Ubuntu 10.04).

Test case #1

How long does it take to create 1 cloud sever and Rackconnect it.

A test below will simulate:

• Using cloud API create one cloud server 
• Monitor the cloud build 
• Once the cloud server is built start to monitor Rackconnect (RC) build 
• Once the RC build is done generate stats and delete the cloud server 
• Repeat the above cycle 10 times

The logs below show how to start the test and the resulting statistics.

$ python -u performance-single-cs.py -v -t 10 -s 1 -b pass@bastion -u user -k key  run | tee log.$(date +%s).txt
$ cat firstgen_rc_performance_report.1346017811.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 22:07:06.914565,     2012-08-26 22:11:30.591570, 263
     2,     2012-08-26 22:11:30.592652,     2012-08-26 22:16:00.184358, 269
     3,     2012-08-26 22:16:00.185239,     2012-08-26 22:20:29.710007, 269
     4,     2012-08-26 22:20:29.711945,     2012-08-26 22:24:53.598046, 263
     5,     2012-08-26 22:24:53.598958,     2012-08-26 22:29:41.288767, 287
     6,     2012-08-26 22:29:41.290309,     2012-08-26 22:33:40.750456, 239
     7,     2012-08-26 22:33:40.752308,     2012-08-26 22:37:40.368505, 239
     8,     2012-08-26 22:37:40.370067,     2012-08-26 22:41:39.680134, 239
     9,     2012-08-26 22:41:39.680827,     2012-08-26 22:46:03.081319, 263
    10,     2012-08-26 22:46:03.083230,     2012-08-26 22:50:11.637474, 248

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     213,     213,     214,     214,     244,     213,     213,     213,     214,     183

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      30,      42,      42,      29,      24,      23,      22,      22,      28,      48

A graphical representation of the above data.

 


Test case #2

 How long does it take to create 2 cloud severs and Rackconnect them.

As before the flowing test is going to simulate:

• Using cloud API create 2 cloud servers
• Monitor builds of the 2 cloud serves
• Once any of the cloud servers is built start to monitor the Rackconnect (RC) build
• Once the RC build is done save stats  and delete a cloud server
• Generate stats when all 2 cloud servers are deleted
• Repeat 10 times the above cycle

The logs below show how to start the test and the results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 2 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346020358.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 22:43:00.756231,     2012-08-26 22:48:06.580328, 305
     2,     2012-08-26 22:48:06.580876,     2012-08-26 22:53:06.387345, 299
     3,     2012-08-26 22:53:06.388921,     2012-08-26 22:57:45.312555, 278
     4,     2012-08-26 22:57:45.313452,     2012-08-26 23:02:42.359540, 297
     5,     2012-08-26 23:02:42.361089,     2012-08-26 23:08:06.284967, 323
     6,     2012-08-26 23:08:06.286676,     2012-08-26 23:13:06.233792, 299
     7,     2012-08-26 23:13:06.235701,     2012-08-26 23:18:06.165773, 299
     8,     2012-08-26 23:18:06.166642,     2012-08-26 23:23:02.864452, 296
     9,     2012-08-26 23:23:02.865955,     2012-08-26 23:27:35.655586, 272
    10,     2012-08-26 23:27:35.662204,     2012-08-26 23:32:38.403999, 302

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     216,     216,     216,     216,     186,     248,     247,     216,     216,     216
    2,     212,     213,     213,     212,     244,     213,     213,     213,     213,     213

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      62,      44,      19,      69,      32,      46,      46,      68,      19,      69
    2,      55,      62,      55,      29,      51,      47,      40,      28,      50,      35

A graphical representation of the test #2 data.
 



Test case #3

How long does it take to create 3 cloud severs and Rackconnect them.

The logs below show how to start the test and the resulting statistics

 
$ python -u firstgen_cs_performance.py -v -t 10 -s 3 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346023741.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-26 23:36:24.625215,     2012-08-26 23:41:45.844673, 321
     2,     2012-08-26 23:41:45.846358,     2012-08-26 23:47:07.165236, 321
     3,     2012-08-26 23:47:07.166380,     2012-08-26 23:51:55.442095, 288
     4,     2012-08-26 23:51:55.443153,     2012-08-26 23:57:13.466668, 318
     5,     2012-08-26 23:57:13.468356,     2012-08-27 00:02:58.579180, 345
     6,     2012-08-27 00:02:58.580118,     2012-08-27 00:08:19.777111, 321
     7,     2012-08-27 00:08:19.778890,     2012-08-27 00:13:37.742591, 317
     8,     2012-08-27 00:13:37.744408,     2012-08-27 00:18:55.713891, 317
     9,     2012-08-27 00:18:55.715903,     2012-08-27 00:23:46.616462, 290
    10,     2012-08-27 00:23:46.617623,     2012-08-27 00:29:01.689222, 315

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     220,     188,     219,     220,     219,     219,     219,     219,     220,     220
    2,     216,     215,     215,     216,     215,     215,     215,     247,     216,     216
    3,     212,     211,     212,     212,     211,     242,     242,     212,     212,     212

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      35,      98,      17,      44,      93,      36,      38,      42,      15,      79
    2,      64,      36,      55,      81,      67,      58,      32,      40,      60,      25
    3,      87,      60,      57,      47,      39,      57,      52,      36,      56,      63

Like before these are the data visualised on graphs
 


Test case #4

How long does it take to create four cloud severs and Rackconnect them.

The logs below show how to start the test and results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 4 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
$ cat firstgen_rc_performance_report.1346071307.txt 
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-27 12:42:58.464643,     2012-08-27 12:48:51.826041, 353
     2,     2012-08-27 12:48:51.827840,     2012-08-27 12:54:51.083906, 359
     3,     2012-08-27 12:54:51.085615,     2012-08-27 13:00:59.075276, 367
     4,     2012-08-27 13:00:59.076493,     2012-08-27 13:06:37.176187, 338
     5,     2012-08-27 13:06:37.177456,     2012-08-27 13:12:24.448441, 347
     6,     2012-08-27 13:12:24.449740,     2012-08-27 13:18:02.517695, 338
     7,     2012-08-27 13:18:02.519349,     2012-08-27 13:23:52.897831, 350
     8,     2012-08-27 13:23:52.898722,     2012-08-27 13:29:58.276944, 365
     9,     2012-08-27 13:29:58.278082,     2012-08-27 13:35:42.351643, 344
    10,     2012-08-27 13:35:42.352320,     2012-08-27 13:41:47.521873, 365

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     224,     224,     224,     224,     224,     224,     192,     257,     195,     225
    2,     157,     222,     222,     221,     221,     222,     189,     222,     225,     254
    3,     249,     219,     250,     219,     218,     219,     217,     219,     221,     219
    4,     215,     216,     216,     216,     215,     216,     245,     248,     218,     216

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      58,      62,      90,      91,      42,      48,      33,     100,      43,      82
    2,      59,      23,      71,      37,      93,      87,      63,      12,      89,      83
    3,      70,      41,      90,      40,      89,      81,     112,      44,      49,      21
    4,      61,     107,      45,      71,      84,      41,      55,      62,      94,      51

Like before these are the data visualised in a form of a graphs
 


Test case #5

How long does it take to create five cloud severs and Rackconnect them.

The logs below show how to start the test and results.

$ python -u firstgen_cs_performance.py -v -t 10 -s 5 -b pass@bastion -u user -k key  run 2>&1 | tee log.$(date +%s).txt 
cat firstgen_rc_performance_report.1346075296.txt
Overall tests duration and statistics
test #,                          start,                            end, duration [s]
     1,     2012-08-27 13:46:48.810109,     2012-08-27 13:53:00.032992, 371
     2,     2012-08-27 13:53:00.034355,     2012-08-27 13:59:41.839530, 401
     3,     2012-08-27 13:59:41.840434,     2012-08-27 14:05:53.244177, 371
     4,     2012-08-27 14:05:53.245105,     2012-08-27 14:11:55.703266, 362
     5,     2012-08-27 14:11:55.704239,     2012-08-27 14:18:19.149206, 383
     6,     2012-08-27 14:18:19.150468,     2012-08-27 14:24:33.617616, 374
     7,     2012-08-27 14:24:33.618816,     2012-08-27 14:30:51.220322, 377
     8,     2012-08-27 14:30:51.221913,     2012-08-27 14:36:53.563330, 362
     9,     2012-08-27 14:36:53.564169,     2012-08-27 14:42:31.914619, 338
    10,     2012-08-27 14:42:31.915871,     2012-08-27 14:48:16.481674, 344

cloud building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,     228,     228,     227,     194,     195,     228,     228,     227,     259,     228
    2,     225,     192,     225,     224,     256,     225,     226,     192,     192,     225
    3,     222,     222,     222,     222,     221,     223,     254,     222,     189,     223
    4,     220,     220,     250,     219,     218,     251,     188,     250,     217,     220
    5,     217,     217,     248,     246,     247,     215,     217,     216,     214,     217

rackconnect building statistics
 cs #,   test1,   test2,   test3,   test4,   test5,   test6,   test7,   test8,   test9,  test10
    1,      36,      54,      30,      44,      64,      30,      61,      48,      43,      44
    2,      52,      56,      54,     103,      97,      80,      50,      43,      51,      67
    3,      97,     166,      58,      42,      93,      68,      90,      52,      58,      69
    4,     114,      42,      71,     101,      53,      89,      55,      65,      49,     100
    5,      67,      74,      88,      73,      60,      80,      93,     113,     101,      67

Like before these are the data visualised in a form of a graph
 



Summary and results description

Through all the 5 test cases above we have been creating cloud servers and repeating this process 10 times. The graphs show that the numbers are changing and the visible trend is that the times are increasing.

We take a look at each of the 5 test cases again and summarize the 10 repetitions to a 3 number result: min, max and average Rackconnect build time. Next we can summarize all the 5 tests/50 cloud build tests and represent all the data in a single graph. The graph below compares all above tests.


As an example, for the test 1 we have the min, max and average times. It has been calculated base on the results from the Test Case #1. We did the some for the other Test Cases.

We see that every time we increase a number of cloud server to build by one the rackconnect build time increases as well. It means that if we burst 1, 2 and up to 5 cloud servers in one single test the time needed for the cloud infrastructure to provision and than to finish up and RackConnect a single cloud server increases as well.

References

1. https://github.com/rtomaszewski/cloud-performance/tree/nextgenv1.0
2. https://github.com/rtomaszewski/cloud-performance/blob/nextgenv1.0/check_rackconnect.sh
3. http://www.rackspace.com/cloud/hybrid/

How to check or monitor build status of a cloud server that belong to a RackConnect cloud account

There is a difference when you use cloud account that is linked with Rackspace RackConnect (RC) product.

Every cloud server from a cloud account that is rackconnected is gong to be reconfigured. All tasks that the RackConnect system will execute can be seen and followed on MyRackspace portal. In short these tasks will change the initial IP settings, route configuration and firewall settings on the original cloud server.

• RC tasks from MyRackspace portal

Cloud Server Created: Add "rackconnect" user 
Cloud Server Created: Validate existence of gateway interface on dedicated network device 
Cloud Server Created: Retrieve metadata 
Cloud Server Created: Provision public IP address 
Cloud Server Created: Update access on dedicated network devices 
Cloud Server Created: Configure network stack 
Cloud Server Created: Configure software firewall 
Cloud Server Created: Update software firewall on other Cloud Servers

• Status and monitoring

At the moment the only way to know that the RC is done is to monitor the network settings on the cloud server manually. This is a known limitation and there are going to be new changes deployed to address this in a close future. For now to know that the RC is done we can for example monitor the last possible task: Cloud Server Created: Configure software firewall.

As soon as we know that the firewall config has changed the RC is done (almost done because there is one last task that can still be something that affects our cloud server). A simple example how the settings change is below.

• Before the RC changes

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

• After the RC changes

# iptables -nL
Chain INPUT (policy DROP)
target     prot opt source               destination         
RS-RackConnect-INBOUND  all  --  0.0.0.0/0            0.0.0.0/0           /* RackConnectChain-INBOUND */ 

Chain FORWARD (policy DROP)
target     prot opt source               destination         
RS-RackConnect-INBOUND  all  --  0.0.0.0/0            0.0.0.0/0           /* RackConnectChain-FORWARD */ 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain RS-RackConnect-INBOUND (2 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED /* RackConnectChain-INBOUND-RE */ 
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           /* Local-Loopback */ 
...

To know that the RC is done you need some simple script to check this. An example bash script is listed below.


That means the you can use the example script above and run it as many times as you want (in some loop with delays between the executions). As soon as the cloud server will be rackconnected the script output will turn into 'yes'.

I'm using Rackconnect and after my cloud server is built I can't ping anything

Rackconnect is Rackspace product that allows you to use a dedicated as well as the cloud products (cloud servers, cloud files and others). It provides cloud servers a secure way to connect to the dedicated and other way around. The majority of implementations are done by using F5 load balancers and Cisco ASA firewalls but company has plans to support other vendors as well. To have a better understanding of RC an implementation can be done for example on:

• standalone ASA
• standalone F5
• HA ASA 
• HA F5 
• single ASA and single F5 
• single ASA and HA f5 
• HA ASA and single F5 
• HA ASA and HA F5

There are sometimes situations when the things don't work as expected. It can be because of a mis-configuration. In a very rare cases because of bugs. Below is an example of a problem I was ask to help to troubleshoot.

Problem 

My newly built cloud server is isolated. After logging over console I can't ping any other servers. Example:

• I can't ping my default gateway from the cloud
• I can't ping any other internal or external address (example Google DNS 8.8.8.8)

Troubleshooting and Analysis

The customer is using Rackconnect. That makes all the network and routing configuration a little bit different comparing to how we set up a standard cloud server without Rackconnect.

Example of a route output from a cloud servers that is rackconnected:

$ ip r | sort | column -t
10.176.0.0/12    via  10.176.0.1     dev    eth1
10.176.0.0/18    dev  eth1           proto  kernel  scope   link  src  10.176.4.37
10.191.192.0/18  via  10.176.0.1     dev    eth1
default          via  10.176.11.111  dev    eth1

For a comparison a routing table from a cloud server that belongs to a cloud account that is not linked to Rackconnect:

# ip r | sort | column -t
10.176.0.0/12     via  10.177.128.1   dev    eth1
10.177.128.0/18   dev  eth1           proto  kernel  scope   link  src  10.177.132.15
10.191.192.0/18   via  10.177.128.1   dev    eth1
164.177.146.0/24  dev  eth0           proto  kernel  scope   link  src  164.177.146.87
default           via  164.177.146.1  dev    eth0    metric  100

Starting troubleshooting I was able to reproduce the issue and confirm that you can't ping the default gateway or anything internal or external IP:

$ ping 10.176.11.111
PING 10.176.11.111 (10.176.11.111) 56(84) bytes of data.
^C
--- 10.176.11.111 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 1293ms

Looking further I saw that the host fails to resolve the IP 10.176.11.111 (default gateway) to its MAC address. The following output from the cloud server confirmed this.

$ arp -an
? (10.176.11.111) at incomplete on eth1

As the customer was using RC with F5 load balancer i went there to check what traffic was hitting the LB. I confirmed that the cloud default gateway is defined on F5 as a self ip object so F5 should be responding to the pings.

# tmsh list /net self
net self 10.176.11.111/18 {
    vlan hybridServiceNet-100
}

When pinging from cloud server I saw that F5 see the ARP request but never replays.

[lb:Active]~ # tcpdump -s0 -l -nn -i 0.0:nnn arp or icmp or host 10.176.4.37 | grep --color=always 10.176.4.37
15:14:26.465108 arp who-has 10.176.11.111 tell 10.176.4.37

I tried to ping the cloud server from F5 and to my surprise it worked fine.

Even at some point shortly after I pinged the cloud server from F5 successfully I was able to see ICMP requests from the cloud server but still F5 was not responding back. That means once the cloud server learned the MAC address of its default gateway it stopped sending the ARP requests and proceed to sending ICMP requests next, as expected.

The cloud server learned the MAC of its default gateway for a while but as the entry timeouted there were only ARP request on the wire again.

[lb:Active] ~ # tcpdump -s0 -l -nn -i 0.0:nnn arp or icmp or host 10.176.4.37 | grep --color=always 10.176.4.37
15:20:35.309694 IP 10.176.4.37 > 10.176.11.111: ICMP echo request, id 36453, seq 1, length 64 in slot1/tmm0 lis= flowtype=0 flowid=0 =00000000:00000000:00000000:00000000 remoteport=0 localport=0 proto=0 vlan=0

By looking further at the F5 configuration I found that there was a missing filter to allow this traffic. It is important because the default RC implementation is dropping any traffic sent from the cloud network to the Rackconnect device (F5 for my customer). Below is a example of a filter that was needed for the ping to start to work.

Example of a missing filter

net packet-filter RCAuto-ID_8-NP_28780-CS_57663-GW_62497 {
    action accept
    order 8
    rule "( src host 10.176.4.37 ) and ( dst host 10.176.11.111 )"
}

The easy way to create it is to go to MyRackspace portal and add all the basic network polices under the Rackconnect. By adding these network policies the Rackconnect system generated number of rules for the F5 and one of them was the missing filter described above:

Basic Access Configuration Policy 1 CLOUD SERVERS CLOUD SERVERS ANY ALL  
Basic Access Configuration Policy 2 CLOUD SERVERS DEDICATED ANY ALL
Basic Access Configuration Policy 3 CLOUD SERVERS INTERNET ANY ALL    
Basic Access Configuration Policy 4 DEDICATED CLOUD SERVERS ANY ALL

This resolved the issue with the default gateway but I was still unable to ping 8.8.8.8. Once again  looking at the tcpdumps on F5 I saw traffic hitting F5. The loadbalancer never forwarded it further. The traffic was simply dropped on the incoming VLAN.

To understand why this is happening it is important to know that by default F5 will drop any traffic unless there is a configuration object to handle it. In my case there were no specific virtual servers, NAT, SNAT that could handle the traffic. We had only a forwarding VS. As the VS was enabled only on specific VLANS it was not used to handle the cloud traffic so all traffic was dropped.

ltm virtual VS-FORWARDING {
    destination any:any
    ip-forward
    mask any
    profiles {
        PROF-FASTL4-FORWARDING { }
    }
    translate-address disabled
    translate-port disabled
 vlans {   
        internal
        external
    }
    vlans-enabled
}

Once we resolved this and enable the VS on necessary vlans all issues were resolved ;).