Dirty trick how to analysis ASA performance based on interface overruns and underruns

There are number of firewall vendors on the market you can chose from (other links to Gartner magic quadrant for firewalls here and here). Every vendor has a product line ranging from the low to high end firewalls. An example product list for Cisco ASA can be seen here: http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html#~tab-b

Problem

I can see in my firewalls interface stats underruns and overrruns and the counters increase.

Solution

This is rather a dirty trick and your monitoring system should be able to graph the interface stats. But if you are in a position like me where you have no visibility to interface statistics like you could have in Zenoss, Cacti, Zabbix or other monitoring system you may need to manually check this...

• We need to first start collecting data so we can look at it later.

Run at least one a day the command and save in a file 1.txt, 2.txt, etc.

sh clock
sh int

• After some time you should have a collection of files 

$ ls -1 *.txt
1.txt
2.txt
3.txt
4.1.txt
4.2.txt
5.1.txt
6.1.txt
8.txt

An example outpout form the file can be seen here: https://github.com/rtomaszewski/experiments/blob/master/asa-interfaces-example.txt

• Download the script asa-interfaces.sh and run it 

bash asa-interfaces.sh

Base on the files you collected it will generate stats for every interface (time stamp is in the last column). Example output:

Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        734 input errors, 0 CRC, 0 frame, 734 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        1502 input errors, 0 CRC, 0 frame, 1502 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        1794 input errors, 0 CRC, 0 frame, 1794 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        1881 input errors, 0 CRC, 0 frame, 1881 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1921 input errors, 0 CRC, 0 frame, 1921 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1971 input errors, 0 CRC, 0 frame, 1971 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        364 input errors, 0 CRC, 0 frame, 364 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        382 input errors, 0 CRC, 0 frame, 382 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        392 input errors, 0 CRC, 0 frame, 392 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        468 input errors, 0 CRC, 0 frame, 468 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        707 input errors, 0 CRC, 0 frame, 707 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        756 input errors, 0 CRC, 0 frame, 756 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        640 input errors, 0 CRC, 0 frame, 640 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        658 input errors, 0 CRC, 0 frame, 658 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        683 input errors, 0 CRC, 0 frame, 683 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        797 input errors, 0 CRC, 0 frame, 797 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        811 input errors, 0 CRC, 0 frame, 811 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        863 input errors, 0 CRC, 0 frame, 863 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        984 input errors, 0 CRC, 0 frame, 984 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1052 input errors, 0 CRC, 0 frame, 1052 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014



Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        646983182 packets output, 473597063148 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        700155558 packets output, 509814505730 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        753341661 packets output, 546026853810 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        1025937535 packets output, 734304301602 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        1054530605 packets output, 761409276094 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        1105491616 packets output, 798565630885 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        1264871240 packets output, 907739984962 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        1315876113 packets output, 943680519398 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        985243431 packets output, 309823858329 bytes, 459 underruns 07:48:45.631 cst Wed Feb 12 2014
        1070533450 packets output, 336205856058 bytes, 459 underruns 05:38:08.573 cst Thu Feb 13 2014
        1159894277 packets output, 366047579951 bytes, 483 underruns 03:37:26.853 cst Fri Feb 14 2014
        1596471490 packets output, 500836893219 bytes, 483 underruns 03:52:52.523 cst Wed Feb 19 2014
        1635530484 packets output, 511489408071 bytes, 483 underruns 10:03:08.799 cst Wed Feb 19 2014
        1722164227 packets output, 536769375853 bytes, 483 underruns 06:11:22.244 cst Thu Feb 20 2014
        2032554621 packets output, 636075162304 bytes, 2831 underruns 08:35:21.315 cst Sun Feb 23 2014
        2174454722 packets output, 688313076839 bytes, 2831 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        1968362 packets output, 524301440 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1987058 packets output, 526612914 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2005883 packets output, 528940672 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        4036852 packets output, 3167775775 bytes, 2831 underruns 03:52:52.523 cst Wed Feb 19 2014
        13676338 packets output, 15853359823 bytes, 2831 underruns 10:03:08.799 cst Wed Feb 19 2014
        23861856 packets output, 16649050850 bytes, 3052 underruns 06:11:22.244 cst Thu Feb 20 2014
        66743830 packets output, 20187731129 bytes, 5941 underruns 08:35:21.315 cst Sun Feb 23 2014
        80290600 packets output, 21286340673 bytes, 6860 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        16582048 packets output, 17699836232 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        17971640 packets output, 19234649922 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        19380417 packets output, 20791969660 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        26970739 packets output, 29162172960 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        27259841 packets output, 29471830004 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        29612954 packets output, 32141440890 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        39077736 packets output, 42912691094 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        42074827 packets output, 46322035220 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        1863787 packets output, 265441230 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1977505 packets output, 281732398 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2091970 packets output, 298145244 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        2718733 packets output, 387975068 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        2750523 packets output, 392530668 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        2855417 packets output, 407567752 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        3242848 packets output, 463113612 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        3366749 packets output, 480878922 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        229534738 packets output, 55157234973 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        249682890 packets output, 59948227086 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        272763726 packets output, 66350185657 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        378020447 packets output, 91307448807 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        384704374 packets output, 93165635304 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        402556578 packets output, 97469565455 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        492798902 packets output, 119550853698 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        564346002 packets output, 137603523999 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        142287604 packets output, 56966204294 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        154809474 packets output, 62049309926 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        167733332 packets output, 67152657884 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        231962627 packets output, 93689642614 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        235640974 packets output, 95384548398 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        249769631 packets output, 103735197461 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        290301462 packets output, 119748550482 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        303003248 packets output, 125098088305 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 packets output, 0 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        0 packets output, 0 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        0 packets output, 0 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        0 packets output, 0 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        0 packets output, 0 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        0 packets output, 0 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014

Once you have the data in-front of you can easily see how the stats were changing over time, over longer period of time like a week.

In my case we suspected that the Firewall hit the capacity limit but further investigation confirmed that the device is doing well and no upgrade is necessary.

References

http://www.gossamer-threads.com/lists/cisco/nsp/152428
http://ccna2ccnp.blogspot.co.uk/2012/12/ciscoasa-oversubcription-maximizing.html
http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1904.html
http://en.wikipedia.org/wiki/Buffer_underrun
http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.html

How to design a network for Openstack or cloud deployment

Designing a network is a topic within itself and there is no way we can cover all of it in this single post. Cisco has its own certification path path CCDE for these who want to know more.

Cisco design and implementation guide - old best practices

In a very simplistic view back in the old days a network used to be design similar to the picture below (although it is hard to say when the new era started ;)). Every big network had to have a core, distribution(sometimes called aggregation) and access layer. The network was engineered mainly to help with North to South traffic in the data center or in another words to help get the data out and into the data center.

New cloud friendly data center design best practice

With the advent and popularization of new networking devises that support layer 2 routing, commonly know as networks fabrics (more info about TRILL and fabric) the network design has shifted in data centers. The way we design the networks today is to maximize the East to West traffic instead of the North to South ( old design above). The purpose of the new network is to allow more efficiently exchange data between the servers within the rack or data center.

They say that a picture is worth more than a thousand words. To help us to visualize how a new data center network/cloud network is designed these demonstration pictures (taken from Cisco document: Cisco Massively Salable Data Center) will shed some more light on it. Please note that we no longer use the core, distribution or access keyword but instead: spine, leaf or superspine to describe the different network layers :).

Openstack reference architecture using Cisco UCS hardware platform

There are many vendors to chose from when selecting your hardware for a complete Openstack deployment (example list of data center friendly networking vendors to look at ). And to make it even more difficult you need to think about all spectrum of vendors like networking, storage and compute.

In one of my previous posts (How to estimate hardware requirements for your private Openstack deployment) I tried to demonstrate an example hardware recommendation for an Openstack deployment. Today we are going to look at this topic once again but exploring the Cisco UCS product line instead.

The data below are taken from the Cisco PDF white paper Red Hat Openstack Architecture on Cisco UCS platform from the DesignZone for Cloud Automation Solution section on Cisco site (http://www.cisco.com/c/en/us/solutions/enterprise/data-center-designs-cloud-computing/could_automation.html).

Openstack on Cisco UCS hardware platform

Cisco is no longer only a networking vendor. With the UCS they offer as well computer platform where you can put together a server with specific hard drive size,  mount of RAM or type of CPU, interconnection card etc. An example configuration taken from the Cisco document above:

That means if we put togheter the UCS servers and the Cisco Nexus switches and Openstack software we can build a simple POC like this one:

In the white paper we can actually find a full list of hardware if you would like to build it yourself.

SSL certificate chain order matters

The certs we trust are usually stored in the CApath on Linux systems. The file is a simple text file with all the certs concatenated one after another.

Problem  

Does the order of certificated stored in the CAfile chain file matter for the client or server?

Analysis and verification  

The simple answer is it depends. As the certs from the CApath/CAfile are used by the client it is independent of the SSL/TLS server we are connecting to. The implementation details of the servers should matter.

That means that the certificate order is important only to the local client itself. In the SSL handshaking the content of this file is never sent to the server. An example handshaking can be found: here: http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_1-1/ssl.html.

To verify of the order of the certs matters for the openssl client we can run the following test. Both files ca1 and ca2 have the same certs but in different order. Example output.

$ openssl s_client -connect 1.1.1.1:443 -state -msg -CAfile ca1
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 2.0 [length 0077], CLIENT-HELLO
    01 03 01 00 4e 00 00 00 20 00 00 39 00 00 38 00
    ...
    ab 3b be 51 9d fa 43
SSL_connect:SSLv2/v3 write client hello A
<<< TLS 1.0 Handshake [length 002a], ServerHello
    02 00 00 26 03 01 2b ae 63 1e ec a0 82 a4 dc 25
    a9 4b 71 14 0a 54 2a ce 3d 6f 38 f5 26 e4 dd 8b
    7e e7 94 d5 02 b7 00 00 04 00
SSL_connect:SSLv3 read server hello A
<<< TLS 1.0 Handshake [length 0e16], Certificate
    11 11 0e 12 00 0e 0f 00 05 69 30 82 05 65 30 82
    22 22 a0 03 02 01 02 02 07 2b 86 02 70 e7 be 22
    ...
    09 0c 4d f6 a7 6b b4 99 84 65 ca 7a 88 e2 e2 44
    be 5c f7 ea 1c f5
depth=2 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 /OU=Domain Control Validated/CN=mydomain.mysite.com
verify return:1
SSL_connect:SSLv3 read server certificate A
<<< TLS 1.0 Handshake [length 0004], ServerHelloDone
    0e 00 00 00
SSL_connect:SSLv3 read server done A
>>> TLS 1.0 Handshake [length 0106], ClientKeyExchange
    11 10 21 32 11 10 13 11 13 11 10 7b 1c c1 d1 10
    ...
    81 1f 71 f1 10 12
SSL_connect:SSLv3 write client key exchange A
>>> TLS 1.0 ChangeCipherSpec [length 0001]
    01
SSL_connect:SSLv3 write change cipher spec A
>>> TLS 1.0 Handshake [length 0010], Finished
    11 11 11 1c 1f 13 6f 1d 11 12 1a 19 ed 64 e8 4b
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
<<< TLS 1.0 ChangeCipherSpec [length 0001]
    01
<<< TLS 1.0 Handshake [length 0010], Finished
    14 00 11 1c ed 9d fd 1f ab db ee ef 29 9a 1c 32
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=mydomain.mysite.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBCCCCCCCCCCCCCCDDDDDDDDEEEEEEFFFFF
...
111111111111111111111111111111111111111111111111111ah6I=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=mydomain.mysite.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
---
SSL handshake has read 3710 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID:
    Session-ID-ctx:
    Master-Key: 11861BE5828519468B6C59B0F01D3FF3126EA2B59DFB985E1C7D88B68E63BF399BCDEF7451D68421C2CE344765CDE572
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1392721077
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

References

http://blog.edgecloud.com/post/19519955133/ssl-certificate-chain-order-matters
http://stackoverflow.com/questions/8431528/nginx-ssl-certificate-authentication-signed-by-intermediate-ca-chain
http://rtomaszewski.blogspot.co.uk/search/label/openssl
http://jw35.blogspot.co.uk/2010/05/doing-certificate-verification-in.html

Howto deploy cowsay to all your cloud servers using ansible configuration management

There are many devops tools that you can use. An example tool chain list can be found http://www.rackspace.com/devops/ and if you are more interested in devops in general make a habit to regularly checking our blog at http://developer.rackspace.com/blog.

Demonstration

In this post we are going to create number of fresh cloud servers: mybastion and 7 serverX
We will install ansible on the mybastion.
Once ansible is installed we will configure our remaining serversX.

Howto and results description

• Spin up cloud servers

We are going to use our helper program http://rtomaszewski.blogspot.co.uk/2013/12/how-to-automatically-deploy-your-public.html to spin up the cloud servers first.

 
auxnova --image 80fbcb55-b206-41f9-9bc2-2dd7aac6c061 --flavor 2 mybastion &> mybastion.log
for i in $(seq 1 7); do echo $i; auxnova --image 80fbcb55-b206-41f9-9bc2-2dd7aac6c061 --flavor 2 server$i &> server${i}.log; done

• Install ansible on mybastion host

auxssh bastion-ip  
auxnova --image 80fbcb55-b206-41f9-9bc2-2dd7aac6c061 --flavor 2 mybastion &> mybastion.log
for i in $(seq 1 7); do echo $i; auxnova --image 80fbcb55-b206-41f9-9bc2-2dd7aac6c061 --flavor 2 server$i &> server${i}.log; done

aptitude update
aptitude upgrade

aptitude install libc6-dev  

aptitude install python-dev
aptitude install python-pip

# http://docs.ansible.com/intro_installation.html
git clone git://github.com/ansible/ansible.git
pip install paramiko PyYAML jinja2 httplib2

cd ansible 
make 
make install

export PATH=$PATH:/usr/local/bin/

• Configure ansible on mybastion

cp -vb /root/ansible/examples/ansible.cfg ~/
sed 's/#host_key_checking = False/host_key_checking = False/g' ansible.cfg > ansible.cfg.old && mv ansible.cfg.old ansible.cfg
sed 's/\(transport *= \)\(.*\)/\1 paramiko/' ansible.cfg  > ansible.cfg.new && mv -v ansible.cfg{.new,}

root@mybastion:~# egrep 'host_key_checking|transport' ansible.cfg | column -t
transport          =  paramiko
host_key_checking  =  False

• Update the host file for static DNS names

for i in $(seq 1 7); do nova show server$i | egrep 'accessIPv4| name' | cut -d '|' -f3 | xargs -n2 | awk '{ print $2, $1}'; done
162.13.9.170 server1
162.13.9.102 server2
162.13.11.232 server3
162.13.9.215 server4
162.13.9.59 server5
162.13.11.227 server6
162.13.11.222 server7

cat >> /etc/hosts <<END
162.13.9.170 server1
162.13.9.102 server2
162.13.11.232 server3
162.13.9.215 server4
162.13.9.59 server5
162.13.11.227 server6
162.13.11.222 server7
END

• Configure ansible inventory file

cp -b ansible_hosts{,.old}
rm ansible_hosts
for i in $(seq 1 7); do 
  cat server${i}.log | egrep ' name|adminPass' | cut -d '|' -f 3 | xargs -n2 | sed 's/\([^ ]*\) *\(.*\)/\1 ansible_ssh_user=root ansible_ssh_pass=\2/' >> ansible_hosts;
done

root@mybastion:~# cat  ansible_hosts
server1 ansible_ssh_user=root ansible_ssh_pass=JaD2KJYswZ92
server2 ansible_ssh_user=root ansible_ssh_pass=keic343S3uJt
server3 ansible_ssh_user=root ansible_ssh_pass=2SNvD4cLN36m
server4 ansible_ssh_user=root ansible_ssh_pass=T7VKkm3GLjKm
server5 ansible_ssh_user=root ansible_ssh_pass=ghdWJeKRRt2f
server6 ansible_ssh_user=root ansible_ssh_pass=N2qVsxt93Lfb
server7 ansible_ssh_user=root ansible_ssh_pass=yMgS5Jx4zexM

• Run a test to verify ansible and inventory is setup correctly

root@mybastion:~# ansible all -m ping -i ansible_hosts
server1 | success >> {
    "changed": false,
    "ping": "pong"
}

server3 | success >> {
    "changed": false,
    "ping": "pong"
}

server4 | success >> {
    "changed": false,
    "ping": "pong"
}

server2 | success >> {
    "changed": false,
    "ping": "pong"
}

server5 | success >> {
    "changed": false,
    "ping": "pong"
}

server7 | success >> {
    "changed": false,
    "ping": "pong"
}

server6 | success >> {
    "changed": false,
    "ping": "pong"
}

• Run a simple command

ansible all -i ansible_hosts -a date
ansible all -i ansible_hosts -m command -a /bin/date

server1 | success | rc=0 >>
Mon Feb 10 01:43:49 UTC 2014

server3 | success | rc=0 >>
Mon Feb 10 01:43:49 UTC 2014

server2 | success | rc=0 >>
Mon Feb 10 01:43:49 UTC 2014

server4 | success | rc=0 >>
Mon Feb 10 01:43:49 UTC 2014

server5 | success | rc=0 >>
Mon Feb 10 01:43:49 UTC 2014

server7 | success | rc=0 >>
Mon Feb 10 01:43:50 UTC 2014

server6 | success | rc=0 >>
Mon Feb 10 01:43:50 UTC 2014

• Use ansible modules to install the packages on the serverX

ansible all -i ansible_hosts -m apt -a 'name=cowsay state=installed'
server1 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (166 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 50438 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server4 | success >> { 
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (861 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server3 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (1075 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server5 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (1116 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server2 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (108 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server6 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (1292 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

server7 | success >> {
    "changed": true,
    "stderr": "",
    "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n  filters\nThe following NEW packages will be installed:\n  cowsay\n0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 19.9 kB of archives.\nAfter this operation, 287 kB of additional disk space will be used.\nGet:1 http://mirror.rackspace.com/ubuntu/ precise/universe cowsay all 3.03+dfsg1-3 [19.9 kB]\nFetched 19.9 kB in 0s (1056 kB/s)\nSelecting previously unselected package cowsay.\n(Reading database ... 49847 files and directories currently installed.)\nUnpacking cowsay (from .../cowsay_3.03+dfsg1-3_all.deb) ...\nProcessing triggers for man-db ...\nSetting up cowsay (3.03+dfsg1-3) ...\n"
}

• Run the cow program on all hosts and collect results

ansible all -i ansible_hosts -m shell -a 'executable=/bin/bash cowsay lucky number on $HOSTNAME is  $RANDOM'
server1 | success | rc=0 >>
 _________________________________
< lucky number on server1 is 9420 >
 ---------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server2 | success | rc=0 >>
 __________________________________
< lucky number on server2 is 12466 >
 ----------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server4 | success | rc=0 >>
 __________________________________
< lucky number on server4 is 27268 >
 ----------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server5 | success | rc=0 >>
 __________________________________
< lucky number on server5 is 26818 >
 ----------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server3 | success | rc=0 >>
 __________________________________
< lucky number on server3 is 17164 >
 ----------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server6 | success | rc=0 >>
 __________________________________
< lucky number on server6 is 28732 >
 ----------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

server7 | success | rc=0 >>
 _________________________________
< lucky number on server7 is 6655 >
 ---------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

References

http://www.ansible.com/home
http://docs.ansible.com/intro_getting_started.html
http://docs.ansible.com/intro_adhoc.html
http://docs.ansible.com/list_of_packaging_modules.html

https://github.com/rtomaszewski/dotfiles/blob/master/.bashrc_rado_aux

Concurrency and parallelism in python

Difference between concurrency and parallelism

The GIL problem is a well know limitation in CPython. Below is on of the video from Heroku conference that shows why this is important (as a bonus you get as well a demo of how to write code in Go language if you want)

The further consequences of this design limitations can be seen in this excellent Mirantis blog post that analyses the performance of an python program: Edge of the Stack: Improve Performance of Python Programs by Restricting Them to a Single CPU.

So what can you do about it? Well until there is GIL in Cpython (and you want or need to stick with this version of python) you may want to chose another library/module for better concurrency support. A long list of available options can be found here: https://wiki.python.org/moin/Concurrency/

At the end to finish up our discussion I can refer you to an practical benchmark that shows a code and do performance analyzes with dealing with concurrency in python: Gevent, Threads, and Benchmarks