Search This Blog

Tuesday, October 30, 2012

How to install parallel Linux tool on Ubuntu or Debian

Parallel is a quite new tool created under the GNU foundation  As its name says it helps to execute jobs in parallel on one or many computers.

Problem

How to install parallel tool on Ubuntu 12.04 Precise [2]

Solution

As the package hasn't been debianised in Ubuntu yet we have to install it using the old school methods.

Follow link [3] and download the appropriate package. Once downloaded install it using dpkg.

wget http://download.opensuse.org/repositories/home:/tange/xUbuntu_12.04/all/parallel_20110422-1_all.deb
dpkg -i parallel_20110422-1_all.deb
type  -a parallel                                                                       22:51:35
parallel is /usr/bin/parallel

References
  1. http://www.gnu.org/software/parallel/
  2. https://wiki.ubuntu.com/DevelopmentCodeNames
  3. https://build.opensuse.org/package/show?package=parallel&project=home%3Atange

Thursday, October 25, 2012

How to extract a duration of a tcp session from tcpdump file


I took a tcpdump to capture all my application connections to data base server. I can filter the tcpudmp data and extract the session that are relevant by using standard tcpdump filters.

Problem

How to find a duration of a tcp session without manually checking packets and calculating the elapsed time.

Solution

There are many tools that can read and understand a tcpudmp file. One of them is tcptrace. An  example of how to use it to find the time is demonstrated below.

root@db1:~# tcptrace -n -l -o1 
1 arg remaining, starting with 'google.pcap'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov  4, 2004

12 packets seen, 12 TCP packets traced
elapsed wallclock time: 0:00:00.001738, 6904 pkts/sec analyzed
trace file elapsed time: 0:00:07.092266
TCP connection info:
1 TCP connection traced:
TCP connection 1:
        host a:        2a00:1a48:7805:0111:8cfc:cf10:ff08:0a2f:55939
        host b:        2a00:1450:400c:0c05::0063:80
        complete conn: yes
        first packet:  Wed Oct 24 22:49:59.166611 2012
        last packet:   Wed Oct 24 22:50:06.258878 2012
        elapsed time:  0:00:07.092266
        total packets: 12
        filename:      google.pcap
   a->b:                              b->a:
     total packets:             6           total packets:             6
     ack pkts sent:             5           ack pkts sent:             6
     ...

References
  1. http://www.tcptrace.org/manual/node11_tf.html
  2. http://docstore.mik.ua/orelly/networking_2ndEd/tshoot/ch05_05.htm
  3. http://www.noah.org/wiki/Packet_sniffing
  4. http://www.darknet.org.uk/2007/11/tcpflow-tcp-flow-recorder-for-protocol-analysis-and-debugging/
  5. http://danielmiessler.com/study/tcpdump/

Wednesday, October 24, 2012

How to check all metadata on your cloud servers

You can create a cloud server and attach a metadata to it. You can modify the settings through out the cloud server life.

Problem

How to check all metadata on all your cloud servers in a simple way.

root@server:~# nova list | tail -n +4 | tac | tail -n +2 | cut  -d'|' -f2 | while read i; do  nova show $i | egrep -i 'name|metadata'; done | sed 's/  */ /g'
| metadata | {} |
| name | test4 |

| metadata | {u'mykey2': u'value2 and 3', u'myke1': u'value1'} |
| name | test4 |

| metadata | {u'rado': u'my value2', u'rade2': u'value3'} |
| name | test3 |

| metadata | {u'rado': u'my value2'} |
| name | test2 |

| metadata | {} |
| name | test1 |

| metadata | {} |
| name | manage2 |

References
  1. http://stackoverflow.com/questions/8017456/output-file-lines-from-last-to-first-in-bash
  2. http://stackoverflow.com/questions/1411713/how-to-split-a-file-and-keep-the-first-line-in-each-of-the-pieces

Example of nova boot command

Problem

There is a lot of written about how the Cloud API works. What the request has to look like and what the correct successes is. But there aren't many practical tutorials or howtos on how to use the tools in practice.

Below is an example of how to create a cloud server using the nova tool.

Solution

This is a link that describe how to install the nova tool [1]. I'm not going to cover this here and I assume you already have it.

  1. Creaet a simple cloud server

  2. root@server:~# nova help | grep image
    root@server:~# nova help | grep flavor
    root@server:~# nova help | grep meta
    root@server:~# nova boot --flavor 2 --image 5cebb13a-f783-4f8c-8058-c4182c724ccd test4 
    +-------------------------+--------------------------------------+
    | Property                | Value                                |
    +-------------------------+--------------------------------------+
    | OS-DCF:diskConfig       | AUTO                                 |
    | OS-EXT-STS:power_state  | 0                                    |
    | OS-EXT-STS:task_state   | scheduling                           |
    | OS-EXT-STS:vm_state     | building                             |
    | accessIPv4              |                                      |
    | accessIPv6              |                                      |
    | adminPass               | 7777                                 |
    | created                 | 2012-10-23T23:11:46Z                 |
    | flavor                  | 512MB Standard Instance              |
    | hostId                  |                                      |
    | id                      | db16f2bc-0f75-4215-927b-c8a34e165dce |
    | image                   | Ubuntu 12.04 LTS (Precise Pangolin)  |
    | metadata                | {}                                   |
    | name                    | test4                                |
    | progress                | 0                                    |
    | rax-bandwidth:bandwidth | []                                   |
    | status                  | BUILD                                |
    | tenant_id               | 7777                                 |
    | updated                 | 2012-10-23T23:11:47Z                 |
    | user_id                 | 7777                                 |
    +-------------------------+--------------------------------------+
    


  3. Creaet a simple cloud server with metadata
root@server:~# nova boot --flavor 2 --image 5cebb13a-f783-4f8c-8058-c4182c724ccd test4 --meta myke1='value1' --meta mykey2='value2 and 3'
+-------------------------+---------------------------------------------------+
| Property                | Value                                             |
+-------------------------+---------------------------------------------------+
| OS-DCF:diskConfig       | AUTO                                              |
| OS-EXT-STS:power_state  | 0                                                 |
| OS-EXT-STS:task_state   | scheduling                                        |
| OS-EXT-STS:vm_state     | building                                          |
| accessIPv4              |                                                   |
| accessIPv6              |                                                   |
| adminPass               | 777                                               |
| created                 | 2012-10-23T23:15:18Z                              |
| flavor                  | 512MB Standard Instance                           |
| hostId                  |                                                   |
| id                      | 3f6be5ae-9c4e-4b0f-880f-c4928e426864              |
| image                   | Ubuntu 12.04 LTS (Precise Pangolin)               |
| metadata                | {u'mykey2': u'value2 and 3', u'myke1': u'value1'} |
| name                    | test4                                             |
| progress                | 0                                                 |
| rax-bandwidth:bandwidth | []                                                |
| status                  | BUILD                                             |
| tenant_id               | 7777                                              |
| updated                 | 2012-10-23T23:15:19Z                              |
| user_id                 | 777                                               |
+-------------------------+---------------------------------------------------+

References
  1. http://docs.rackspace.com/servers/api/v2/cs-gettingstarted/content/section_gs_install_nova.html
  2. http://docs.openstack.org/api/quick-start/content/
  3. http://devops.rackspace.com/13/getting-started-using-python-novaclient-to-manage-cloud-servers
  4. http://vexxhost.com/blog/2012/09/10/installing-cpanel-on-a-cloud-server/

Example of nova credential command



If your credential settings are correct you are going to see this output

root@server:~# set | grep OS_
OS_AUTH_SYSTEM=rackspace_uk
OS_AUTH_URL=https://lon.identity.api.rackspacecloud.com/v2.0/
OS_NO_CACHE=1
OS_PASSWORD=777
OS_REGION_NAME=LON
OS_TENANT_NAME=777
OS_USERNAME=777

root@server:~# nova credentials
+------------------------+---------------------------------------------------------------------------------------+       22:51:35
| User Credentials       | Value                                                                                 |
+------------------------+---------------------------------------------------------------------------------------+
| RAX-AUTH:defaultRegion |                                                                                       |
| id                     | 777                                                                                 |
| name                   | name                                                                       |
| roles                  | [{u'description': u'User Admin Role.', u'id': u'3', u'name': u'identity:user-admin'}] |
+------------------------+---------------------------------------------------------------------------------------+
+---------+--------------------------------------------+
| Token   | Value                                      |
+---------+--------------------------------------------+
| expires | 2012-10-24T13:21:47.000+01:00              |
| id      | f6e9c367-7777-7777-7777-777777777777       |
| tenant  | {u'id': u'777', u'name': u'777'} |
+---------+--------------------------------------------+


If your credentials are wrong you will see this one instead

root@server:~# nova credentials
ERROR: Invalid OpenStack Nova credentials.

root@server:~# nova --debug credentials

REQ: curl -i https://lon.identity.api.rackspacecloud.com/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "777", "apiKey": "777", "tenantName": "777"}}}'

connect: (lon.identity.api.rackspacecloud.com, 443)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: lon.identity.api.rackspacecloud.com\r\nContent-Length: 154\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: python-novaclient\r\n\r\n{"auth": {"RAX-KSKEY:apiKeyCredentials": {"username": "777", "apiKey": "777", "tenantName": "777"}}}'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Server: nginx/0.8.55
header: Date: Tue, 23 Oct 2012 22:55:15 GMT
header: Content-Type: application/json
header: Transfer-Encoding: chunked
header: Connection: keep-alive
header: Content-Encoding: gzip
header: response-source: cloud-auth
header: vary: Accept, Accept-Encoding, X-Auth-Token
header: VIA: 1.0 Repose (Repose/2.3.5)
RESP:{'status': '401', 'content-length': '73', 'via': '1.0 Repose (Repose/2.3.5)', 'response-source': 'cloud-auth', 'transfer-encoding': 'chunked', 'vary': 'Accept, Accept-Encoding, X-Auth-Token', 'server': 'nginx/0.8.55', 'connection': 'keep-alive', '-content-encoding': 'gzip', 'date': 'Tue, 23 Oct 2012 22:55:15 GMT', 'content-type': 'application/json'} {"unauthorized":{"code":401,"message":"Username or api key is invalid"}}

DEBUG (shell:534) Invalid OpenStack Nova credentials.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/novaclient/shell.py", line 531, in main
    OpenStackComputeShell().main(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/novaclient/shell.py", line 463, in main
    raise exc.CommandError("Invalid OpenStack Nova credentials.")
CommandError: Invalid OpenStack Nova credentials.
ERROR: Invalid OpenStack Nova credentials.

Monday, October 22, 2012

How many Linux distributions support Openstack

Ubuntu is one of the first distributions that started to show a support for Openstack. They have been very active from the beginning and continue to expand in the cloud. As an example this video can summarize how Ubuntu and Canonical has been involved in the project:



But is Canonical with Ubuntu the only Linux distribution that is interested in supporting and popularizing a wide adoption of Openstack [2] and open cloud standards? Likely they are not alone. There are already two other main distribution like Red Hat  SUSE that joined the Openstack Platinum Membership club as well ;) [1].



With all these efforts and livid community this is only a matter of time to see this to be come a next Linux of the Internet Cloud in the Open software world ;)

References
  1. http://www.openstack.org/foundation/companies/
  2. http://blog.canonical.com/2012/09/19/why-were-number-one-for-openstack/

Sunday, October 21, 2012

What is new in Folsom Openstack release

On the latest Openstack design summit there was a lot of talking about the future roadmap of what the next Openstack release (code name Grizzly) is going to be. It is nice to know where the project is heading to but more importantly it is good to know what new features and improvements we made and coded in Folsom.

These 3 links should give you enough information to start.
  1. Mirantis Webcast with a recorded sound and video What’s new in OpenStack Folsom
  2. Slides to the video http://www.slideshare.net/mirantis/slideshare-folsom-qawebinar
  3. FAQ after presentation OpenStack Cloud Folsom Webcast: Followup on your questions

Why programmers work at night

Do you find yourself less productive during a day than at night. This article Why programmers work at night tries to provide an extensive explanations for this. The discovery although is not very surprising.


Usual everyone during a day have to battle with constant distractions at work like urgent emails, calls, conferences, important stuff,... etc. This is different at night when majority prepare for a bed to feel asleep. From a logical point of view this is the best time for any creative work that needs our concentration and full brain power.

Happy night coding ;)

What do these all Cloud products do

The Cloud helps to evolve what w call today as Web Hosting. The changes directly impact the image and position of Rackspace on today's SMB marker. Since Openstack lunch in 2010 the company is undergoing a transformation to create new better products, to empower the customers and give the a freedom of choice what they can use to build their web application and platforms.



To answer our question this presentation gives you an excellent summary of what cloud products you can find in Rackspace portfolio and what you can build using them: How the Open Cloud Can Benefit Your Business - Rackspace UK



Thursday, October 18, 2012

Funny picture from one of the Google's Data Center

There has been a new footage and video about a Google data center this year. To make it more interesting they created a mini Google map version where you can interactively walk through the data center building and look around ;) During my virtual journey this is what found:



References
  1. http://www.engadget.com/2012/10/17/google-inside-data-centers/
  2. http://www.engadget.com/photos/inside-google-s-data-centers/#5366823

Who owns the most servers

The cloud is expanding and changing the landscape in data centers. It would be interesting to know who has the most servers on the world, isn't it? The links below may not be fully up to date but they still revel pretty good overview about the industry. The most interesting part is the difference between the Google #1 and the rest of the world.

References
  1. http://www.datacenterknowledge.com/archives/2009/05/14/whos-got-the-most-web-servers/
  2. http://gizmodo.com/5517041/googles-insane-number-of-servers-visualized
  3. http://news.netcraft.com/archives/2012/09/10/september-2012-web-server-survey.html

GNS3 doesn't list new VirtualBox Network Devices

Problem

GNS3 doesn't see any new Host-Only Ethernet Adapter created in VirtualBox.

Troubleshooting

We can verify and see that the new interfaces exists on the system.

C:\Users\user>netsh interface ipv4 show interfaces
Idx     Met         MTU          State                Name
---  ----------  ----------  ------------  ---------------------------
  1          50  4294967295  connected     Loopback Pseudo-Interface 1
 37          20        1500  connected     VirtualBox Host-Only Network #2
 21          20        1500  connected     VMware Network Adapter VMnet1
 22          20        1500  connected     VMware Network Adapter VMnet8
 24          20        1500  connected     VirtualBox Host-Only Network
 28          10        1500  connected     Local Area Connection 2
 45          20        1500  connected     VirtualBox Host-Only Network #3

The two above interfaces are missing and don't show in the GNS3 when trying add new VM. Because you don't see them on the GUI can't select them when trying to add a cloud VM to your network topology.

c:\Program Files\GNS3>"Network device list.cmd"

Network adapters on this machine:

NIO_gen_eth:\Device\NPF_{BAB3B416-5025-4142-A041-290EB05E7EA2}
 Name       : VMware Network Adapter VMnet8
 IP Address : 192.168.33.1
 Description: VMware Virtual Ethernet Adapter

NIO_gen_eth:\Device\NPF_{AA540984-A0AD-45BC-905F-C39E4A15010C}
 Name       : VirtualBox Host-Only Network
 IP Address : 192.168.56.1
 Description: Sun

NIO_gen_eth:\Device\NPF_{939368AC-34FE-47CB-9D4B-DC26D284D91E}
 Name       : VMware Network Adapter VMnet1
 IP Address : 192.168.91.1
 Description: VMware Virtual Ethernet Adapter

NIO_gen_eth:\Device\NPF_{991068D7-DDE1-45AB-B68F-24970ECA951B}
 Name       : Local Area Connection 2
 IP Address : 192.168.0.2
 Description: Marvell Yukon Ethernet Controller

Use as follows:
 F0/0 = NIO_gen_eth:\Device\NPF_{...}

Solution

This is Windows problem that doesn't propagate and advertise new virtual interfaces once created. The solution is to restart the Windows Service called 'NetGroup Packet Filter Driver'.

C:\Windows\system32>net stop npf
The NetGroup Packet Filter Driver service was stopped successfully.

C:\Windows\system32>net start npf
The NetGroup Packet Filter Driver service was started successfully.

References
  1. http://blog.ngocpham.info/blog/2010/04/13/howto-gns3-network-device-list-problem/
  2. http://forum.gns3.net/topic4878.html
  3. http://blog.stephendolphin.co.uk/quick-notes/quick-helpful-list-of-netsh-commands-for-your-windows-server-core-2008r2-installation/
  4. http://www.computerhope.com/netsh.htm

Monday, October 15, 2012

Best practices on how to learn multi vendor networking hardware with a help of modern virtualization

When I read my fist book to CCNA back in at the university I didn't know much about network simulators or emulators. I learn the stuff from the books and by practicing on our uni lab. But since my recent come back to networking again I needed today something that helps me to simulate or even reproduce some network config scenarios. As I'm not interested in buying any second hand Cisco gear (or any other vendor) or renting remote labs or setting complex repros in our datacenter I have found a simple solution for me: GNS3.

I didn't have a big chance yet to play with it extensively but at the first glimpse it has a huge potential. It can boot and emulate many vendor devices like [2]:

  • Cisco switches and routers
  • Cisco PIX and ASA firewalls
  • Vyatta firewalls
  • Juniper
  • Linux Open vSwitch  
  • generic VM

These are some further features described on the main page [1]:

Features overview

Design of high quality and complex network topologies.
Emulation of many Cisco IOS router platforms, IPS, PIX and ASA firewalls, JunOS.
Simulation of simple Ethernet, ATM and Frame Relay switches.
Connection of the simulated network to the real world!
Packet capture using Wireshark.
 
References
  1. http://www.gns3.net/
  2. http://www.gns3.net/appliances/ http://www.gns3.net/hardware-emulated/
  3. http://www.gns3.net/screenshots/

Redhat application deployment automaton with Puppet module and F5

I have found this youtube video [1] on the the Puppet channel. It is an interesting presentation about the day to day issues the folks from Redhat run into when developing, managing and deploying new code for customer facing sites they host.

On of the things they mention that was difficult for them and why the started this project were:

  • Lack of a cross functional team who could understand the whole architecture.
  • People tend to have a limited understanding of other areas that they don't work in; Examples are SysAdmin about NetAdmin, NetAdmin about Develpers etc.
  • On new depoyment a difficulties to find a team/person who can take an ownership of an issue as the problem may be above the area of their own expertise.
  • Diverse and inconsistencies dev, staging and production systems.

To solve the problem they decided to refactor the architecture and automate as much as it was possible. The video [1] shows what issues they run into and how they solved it with a help of Puppet module they created and the F5 load balancer.

In short a message they convey in the presentation is: automate, standardize, and automate once again.

References
  1. Managing F5 LTM with Puppet - Matthew Carpenter and Bret McMillan of Red Hat
  2. http://www.youtube.com/user/PuppetLabsInc

Tuesday, October 9, 2012

Google does use SDN in its data centers

There is a lot of rumors that Google already uses SDN inside its data centers as well as to route and control the external traffic between them.

According to this article Going With the Flow: Google’s Secret Switch to the Next Wave of Networking Google is operating an SDN enabled network. But can you really believe in everything what you read? Researching more on the the topic of SDN plus Google I have found this more  skeptical article. It says that this may not be a whole truth Openflow @ Google: Brilliant, but not revolutionary .

Despite this to contradicting articles It would be nice to know that SDN can scale and has already proved its usefulness on a large scale. But without this there is still the open question you have to ask your self.

Is SDN really going to transform and shape the traffic within and outside of small or big data centers. Will it really provide a more sophisticated ways to control and direct the external traffic over wan links as well as over the 10 GiB enabled internal networks? Without any prove of concept (POC) this all is only one idea more that may or may not get bigger traction in networking.

To finis this up I would like to refer refer to this article Balloons, Bags, and SDNs . There is one very useful and I particularly smart conclusion about the whole SDN future:

Before you say, “it won’t scale,” ask, “compared to what?”

With it in mind let's wait how the market is going to move and what expectation will or will not be fulfilled :).

Monday, October 8, 2012

Software defined network (SDN) as an example of new evolution in networking

 In my research about SDN I have read recently a very interesting blog post about technologies in IT. The post has a strong emphasis on merging of new technologies and its later on evolution on the market. I higly recomed to take a look at the full blog that can be found here: SDN, Career choices and magic graphs

The author, Ivan Pepelnjak in his article brought together couple of terms from a field of marketing, psychology and consulting. These are the terms:


In general, in the world we know today where innovations are happening every day these concepts help to develop a little bit of wider understanding of new technologies in IT. Working as an individual it can be sometimes very overwhelming to face so many new technologies day by day. But knowing that this all belongs to a bigger picture helps to develop a health approved to it.

That said unfortunately I still didn't find the answer for my question about a future of SDN in networking but I'm definitely one step closer to it.