On Cisco ASA firewall how to find the real Interface MAC address

Normally the output from 'sh interface' shows interfaces MAC addresses. This is truth when you have a single ASA.

Problem

How to find a real interface MAC address on HA ASA cluster node.

Resolution

There are no floating IPs in ASA cluster design. Instead there active IP will be moved between the ASA nodes when a failover occurs. For guys who work on different cluster implementation it may be very confusing.

When a firewall is part of an HA active/standby cluster the physical interface MAC address (showed in the output form sh interface) and the IP assigned to it has always a value of the primary unit.

When a failover happens both ASAs swaps IP and MAC during.

For example, if we have assigned an IP 1.1.1.1 to the primary unit on our ASA cluster this IP will be once held by the unit A once by the unit B. That means, when you try to connect to this IP you never know to what physical ASA unit you are actually connecting.

To find out the real MAC of an interface you need to look at the sh version output.

fw-1092388-553262/pri/stby# sh ver | i Gig
 0: Ext: GigabitEthernet0/0  : address is 1111.aaaa.deea, irq 9   ------  real address on the ASA 
 1: Ext: GigabitEthernet0/1  : address is 1111.aaaa.deeb, irq 9
 2: Ext: GigabitEthernet0/2  : address is 1111.aaaa.deec, irq 9
 3: Ext: GigabitEthernet0/3  : address is 1111.aaaa.deed, irq 9
 
fw-1092388-553262/pri/stby# sh int | i MAC|Int
Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        MAC address 2222.fd52.ac28, MTU 1500                     ------ active MAC that migrates every time a failover happens

Best practice

If you want to know to what physical ASA unit you connected take a look at the output from sh version. Never relay on the values in sh int output.

References

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/ef.html#wp1929064
http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/ha_overview.html

Google Compute Engine Console

In one of our previous posts (Links to Cloud provider web management consoles) we took a look at some of the major cloud providers out there. In this post we take a look how Cloud Google Compute platform looks like.

• Google Cloud Platform

https://cloud.google.com/

Top Openstack contributors

There is a new site on the main Openstack portal: http://activity.openstack.org/. When looking around to see what information you can find there I stumble upon this post below.

What came as a surprise:

• High position for IBM. 
• Not Rackspace or Canonical or Mirantis but Red Hat as a number one contributor.

Top 10 Openstack organizations activities in 2013/2014

During TLS handshaking server can ignore a cipher from the preference list in ClientHello message

I've noticed recently that when my client tries to negotiated a TLS session its ciphers preference list is being ignored by the server.

• Test 1

$ openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA

• Test 2

$ openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA

• Test 3

# openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA:AES128-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: C7E7F65B4E927AFDE568E52FDAE52495CE815DAC2958B82C854146CD383FC00BB47ED691840C713CE762B6430FCB3230
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568347
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

• Test 4

# openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA:DES-CBC-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: 1F0FB31410AAFFC234DC5598FFA2A676C63119A6367453D05578F9B1652DD085DB1D5C016694B9D293F3A2DFF769EF51
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568389
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

We can see in (1) and (2) that the TLS servers supports 2 different ciphers suits.
In (3) and (4) we see that even though the preference list was directly specified by the client the TLS server ignored the settings in (3).

Example ssldump output from (3).

 
# ssldump -A -n -i eth0 port 443 and host my_server_ip
New TCP connection #1: 162.13.0.27(51869) <-> my_server_ip(443)
1 1  0.1013 (0.1013)  C>SV3.1(61)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 23 b5 62 f5 08 38 14 53 d9 ac a4 51 c2 28
          0d e2 0e b6 41 89 2d af a1 50 55 bd a9 35 05 75
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.1984 (0.0970)  S>CV3.1(49)  Handshake
      ServerHello
        Version 3.1
        random[32]=
          2b ae 63 1e 9c da f3 38 b6 f9 dd c6 c0 4f ad 6c
          2c f0 f3 52 67 6e 28 29 8a 24 95 95 a5 67 f2 6f
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_AES_128_CBC_SHA
        compressionMethod                   NULL
1 3  0.1984 (0.0000)  S>CV3.1(674)  Handshake
      Certificate
        certificate[664]=
          50 82 02 94 50 82 01 fd 02 02 03 79 50 0d 06 09
          2a 86 48 86 f7 0d 01 01 05 05 00 50 81 b6 31 0b
          50 11 06 03 55 04 06 13 02 55 53 31 0e 50 0c 06
          03 55 04 08 13 05 54 65 78 61 73 31 14 50 12 06
          03 55 04 07 13 0b 53 61 6e 20 41 6e 74 6f 6e 69
          6f 31 12 50 10 06 03 55 04 0a 13 11 52 61 63 6b
          73 70 61 63 65 31 1e 50 1c 06 03 55 04 0b 13 15
          53 79 73 74 65 6d 20 41 64 6d 69 6e 69 73 74 72
          61 74 69 6f 6e 31 23 50 21 06 03 55 04 03 13 1a
...
          b3 79 0a 37 cd 27 93 af
1 4  0.1984 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
1 5  0.2126 (0.0141)  C>SV3.1(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          33 25 53 17 c2 45 b0 32 8a ca c1 66 39 6e f0 31
          98 96 4c 34 f2 e4 fd b7 0c e6 15 af c7 d3 fc e4
          0d 15 c9 c9 d4 e0 78 5a a1 13 dc 55 8b 5a bc 69
          68 24 f5 d1 50 6a 19 2e 71 9a 66 ee 3a 64 bc 1e
          d8 9a da d4 e0 44 96 b3 43 20 f1 a0 b6 4c 49 8e
          b2 ae 2b a6 12 68 78 19 eb 61 06 1c 34 8a 03 22
          ab 7e ff f8 88 44 89 97 cd 53 06 b0 b6 66 7b 77
          2a c0 0a 15 a3 54 2d 8c 5b 74 bc fe 31 3f 1f 8f
1 6  0.2126 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
1 7  0.2126 (0.0000)  C>SV3.1(48)  Handshake
1 8  0.3100 (0.0974)  S>CV3.1(1)  ChangeCipherSpec
1 9  0.3100 (0.0000)  S>CV3.1(48)  Handshake
1    1.4782 (1.1681)  C>S  TCP FIN
1    1.5740 (0.0958)  S>C  TCP FIN

Problem

Can TLS server ignore the cipher suite list passed from the client during TLS handshaking?

Analysis

The specification how the ClientHello message can be handled by the server during TLS handshaking has changed from TLS v1.0 to v1.2. The extract from the RFC comparing both relevant sections.

• rfc2246

7.4.1.2. Client hello

   The CipherSuite list, passed from the client to the server in the
   client hello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first). Each CipherSuite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length) and a MAC algorithm.

   The server will select a cipher suite or, if no acceptable choices 
   are presented, return a handshake failure alert and close the 
   connection.

• rfc5246

7.4.1.2.  Client Hello

   The cipher suite list, passed from the client to the server in the
   ClientHello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first).  Each cipher suite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length), a MAC algorithm, and a PRF.  The server will select a cipher
   suite or, if no acceptable choices are presented, return a handshake
   failure alert and close the connection.

   If the list contains cipher suites the server does not recognize, 
   support, or wish to use, the server MUST ignore those cipher suites,
   and process the remaining ones as usual.

In the yellow parts we can see that a server implementing TLS v1.2 has the ability to ignore ciphers specified by the client in its list ClientHello message. The criteria are very wide. The TLS server can ignore a cipher only because it doesn't wish to use it.

References

http://www.openssl.org/docs/apps/ciphers.html
DES and IDEA ciphers are deprecated in the latest TLS protocol
http://www.ietf.org/rfc/rfc2246.txt - The TLS Protocol, Version 1.0
http://tools.ietf.org/html/rfc5246.txt - The Transport Layer Security (TLS) Protocol, Version 1.2
http://support.f5.com/kb/en-us/solutions/public/10000/200/sol10209.html - ssldump

A definitely interesting reading about ciphers in the web browser to show some parasitical view at the cipher list topic.
https://groups.google.com/forum/#!topic/mozilla.dev.tech.crypto/36na1B2brGU

How to run my TLS server

Problem

How to quickly deploy and run SSL/TLS server for testing?

Solution

• You need first key and certificate. There are many guides on Internet how to generate these with the help of openssl (example http://www.akadia.com/services/ssh_test_certificate.html).
• Use openssl s_server to run start the server and customize the supported ciphers 

openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -tls1
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -ssl3
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -cipher DES-CBC-SHA:AES256-SHA

DES and IDEA ciphers are deprecated in the latest TLS protocol

When incorporating security into your solution and applications it is important to maintain a high level view and follow security best practices. That means you need a FW. The FW should have a DMZ and Inside segments. To actively protect your web applications you can deploy WAF or another kind of IPS. To passively monitor traffic you can implement IDS additionally.

But as our solution is being extend by new and more sophisticated network devices it is still important to understand and maintain the low level security parameters for the network protocols. When I mean low level I mean the low level details of the TLS/SSL network protocols that are being used when using HTTPS for example.

Problem

Is that secure or recommended to enable and support DES or IDEA ciphers in application or SSL-offloading load balancers?

Analysis and results discussion

According to RFC 5469 IDEA and DES should not be used any more. The reasons are listed in the RFC.

To verify if your server responds to clients using these ciphers you can try:

 
# (1)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -ssl3
# or
# (2)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -tls1
# or 
# (3)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA 
CONNECTED(00000003)
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
 0 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICATCCAWoCCQCxkFtlc6Bd0TANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB
VTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
cyBQdHkgTHRkMB4XDTE0MDExMjIxMzMwN1oXDTE1MDExMjIxMzMwN1owRTELMAkG
A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0
IFdpZGdpdHMgUHR5IEx0ZDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAtVtD
BfrmHU/T9m4xvvlP7+J4zJ2BYFY8QfSvQ1tyQw+BwvPyh9zyzgd0Zw4iOa6ThlQ3
GTr7e3FMQooMWpK0XXTYKbbWGqyVfnkcwmWjapJxOv8OaXlDS5TIc7MursFXp16e
oOjvpyuddX2gilQLiO6n1b6vyKsFfPW0eoPPmf8CAwEAATANBgkqhkiG9w0BAQUF
AAOBgQBGd8xD6ZINxy8Vf1jFrX+4EyPEL3+DkAU4lInd83kIuDd8i2fzia4YOfKh
JB3/ML8kLGLMh6R0WpHbaoGQvNM5qn7GdFL+DDBvXqlyZtIrfKamx+s5GxUiP0SV
5miO9Oh1mkxhXUqaVHaJR0DeTYEAuA0dc1lMoJlPoSMedlgJBg==
-----END CERTIFICATE-----
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
No client certificate CA names sent
---
SSL handshake has read 710 bytes and written 273 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC-SHA
    Session-ID: A5568C18EFB2DA77B729A247EA8E605BEBC4DF478129357D002C26DFA89F96C7
    Session-ID-ctx:
    Master-Key: F9CDF6CD91F3E4F5117758104906C779E18493062397EFFE7E4C518F0894398A01D969D5EE07804ED436A24444CD92FA
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Compression: 1 (zlib compression)
    Start Time: 1389565902
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

An example output from the (3) showing that the server supports the legacy and depreciated ciphers.

 
# ssldump -A -n -i lo port 443
New TCP connection #1: 127.0.0.1(50211) <-> 127.0.0.1(443)
1 1  0.0007 (0.0007)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 18 25 a9 86 1c 58 ff f0 90 ca fe ba f8 eb
          c8 23 46 fd 5b 7a 4a aa 51 c2 37 40 6a 8b dc 01
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0010 (0.0003)  S>CV3.2(58)  Handshake
      ServerHello
        Version 3.2
        random[32]=
          52 d3 18 25 95 9c 3e 34 80 d8 00 3d fe 02 8f bf
          3c 1a 72 5d d1 4f 30 8c 6c 3b fa 64 0e 82 1c 6c
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_DES_CBC_SHA
        compressionMethod                 unknown value
1 3  0.0021 (0.0011)  S>CV3.2(527)  Handshake
      Certificate
        certificate[517]=
          30 82 02 01 30 82 01 6a 02 09 00 b1 90 5b 65 73
          a0 5d d1 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05
          05 00 30 45 31 0b 30 09 06 03 55 04 06 13 02 41
          55 31 13 30 11 06 03 55 04 08 0c 0a 53 6f 6d 65
          2d 53 74 61 74 65 31 21 30 1f 06 03 55 04 0a 0c
          18 49 6e 74 65 72 6e 65 74 20 57 69 64 67 69 74
          73 20 50 74 79 20 4c 74 64 30 1e 17 0d 31 34 30
          31 31 32 32 31 33 33 30 37 5a 17 0d 31 35 30 31
          31 32 32 31 33 33 30 37 5a 30 45 31 0b 30 09 06
          03 55 04 06 13 02 41 55 31 13 30 11 06 03 55 04
          08 0c 0a 53 6f 6d 65 2d 53 74 61 74 65 31 21 30
          1f 06 03 55 04 0a 0c 18 49 6e 74 65 72 6e 65 74
          20 57 69 64 67 69 74 73 20 50 74 79 20 4c 74 64
          30 81 9f 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01
          05 00 03 81 8d 00 30 81 89 02 81 81 00 b5 5b 43
          05 fa e6 1d 4f d3 f6 6e 31 be f9 4f ef e2 78 cc
          9d 81 60 56 3c 41 f4 af 43 5b 72 43 0f 81 c2 f3
          f2 87 dc f2 ce 07 74 67 0e 22 39 ae 93 86 54 37
          19 3a fb 7b 71 4c 42 8a 0c 5a 92 b4 5d 74 d8 29
          b6 d6 1a ac 95 7e 79 1c c2 65 a3 6a 92 71 3a ff
          0e 69 79 43 4b 94 c8 73 b3 2e ae c1 57 a7 5e 9e
          a0 e8 ef a7 2b 9d 75 7d a0 8a 54 0b 88 ee a7 d5
          be af c8 ab 05 7c f5 b4 7a 83 cf 99 ff 02 03 01
          00 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05
          00 03 81 81 00 46 77 cc 43 e9 92 0d c7 2f 15 7f
          58 c5 ad 7f b8 13 23 c4 2f 7f 83 90 05 38 94 89
          dd f3 79 08 b8 37 7c 8b 67 f3 89 ae 18 39 f2 a1
          24 1d ff 30 bf 24 2c 62 cc 87 a4 74 5a 91 db 6a
          81 90 bc d3 39 aa 7e c6 74 52 fe 0c 30 6f 5e a9
          72 66 d2 2b 7c a6 a6 c7 eb 39 1b 15 22 3f 44 95
          e6 68 8e f4 e8 75 9a 4c 61 5d 4a 9a 54 76 89 47
          40 de 4d 81 00 b8 0d 1d 73 59 4c a0 99 4f a1 23
          1e 76 58 09 06
1 4  0.0021 (0.0000)  S>CV3.2(4)  Handshake
      ServerHelloDone
1 5  0.0085 (0.0063)  C>SV3.2(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          71 83 c8 f4 af ab be 5e a6 e0 ec 06 ab 14 be e3
          41 25 5f f9 9e b3 29 a1 a5 1a a9 25 8d c8 1e 3d
          f2 06 3b 50 68 58 ca 1b bf 9b 1a e5 3f 4d c7 f5
          43 67 93 a1 fc f8 16 9e 35 24 7f a6 4c ad 9b 0f
          c4 db 6e a8 3d 97 5e 5f 96 0f 40 7b a3 42 62 e4
          7c 07 f9 65 97 a4 52 1a 30 cc 11 d6 43 06 7d 85
          4b e9 d5 1e 2e af 9a bd 90 cd 4d 6e aa 9e 00 29
          07 12 cd 96 bd 59 ca 5c dc a3 88 00 53 6e 8f ec
1 6  0.0085 (0.0000)  C>SV3.2(1)  ChangeCipherSpec
1 7  0.0085 (0.0000)  C>SV3.2(56)  Handshake
1 8  0.0099 (0.0014)  S>CV3.2(170)  Handshake
      TLS_RSA_WITH_RC4_128_MD51 9  0.0476 (0.0376)  S>CV3.2(1)  ChangeCipherSpec
1 10 0.0476 (0.0000)  S>CV3.2(56)  Handshake
1    0.7913 (0.7436)  C>S  TCP FIN
1    0.7917 (0.0004)  S>C  TCP FIN

Output proving the ciphers are not supported.

 
# ssldump -A -n -i eth0 port 443 and host 31.222.129.61
New TCP connection #1: 162.13.0.27(34228) <-> 31.222.129.61(443)
1 1  0.0017 (0.0017)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 19 53 c5 78 4c 06 8c e7 fc 47 a1 92 ec a4
          90 63 ca a2 6e a5 7e 58 bb 72 9b a1 be c1 84 3a
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0021 (0.0003)  S>CV3.1(2)  Alert
    level           fatal
    value           handshake_failure
1    0.0021 (0.0000)  S>C  TCP FIN
1    0.0044 (0.0022)  C>S  TCP FIN

 
# openssl s_client -connect 31.222.129.61:443 -state -msg -cipher DES-CBC-SHA
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> TLS 1.1  [length 003b]
    01 00 00 37 03 02 52 d3 19 53 c5 78 4c 06 8c e7
    fc 47 a1 92 ec a4 90 63 ca a2 6e a5 7e 58 bb 72
    9b a1 be c1 84 3a 00 00 04 00 09 00 ff 02 01 00
    00 09 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state
SSL3 alert read:fatal:handshake failure
<<< TLS 1.0 Alert [length 0002], fatal handshake_failure
    02 28
SSL_connect:error in unknown state
139646822749888:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:741:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 64 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

References

DES and IDEA Cipher Suites for Transport Layer Security (TLS)
http://www.ietf.org/rfc/rfc5469.txt

The TLS Protocol, Version 1.0
http://www.ietf.org/rfc/rfc2246.txt

The Transport Layer Security (TLS) Protocol, Version 1.2
http://tools.ietf.org/html/rfc5246.txt

ASA performance troubleshooting tips

This is more a work in progress. Below are couple of tips and ideas how to deal with high traffic performance issues.

Limit connection per IP

Often a load can be generated from unique single (or a group of IPs). To limit the number of connection.

access-list http_conn_limit extended permit tcp any any eq www 
! access-list http_conn_limit extended permit tcp any any eq https
! you can add any other ACL to catch the intresting traffic 

class-map http_conn_limit_class
 match access-list http_conn_limit

policy-map http_conn_limit_map
 class http_conn_limit_class
  set connection per-client-max 100 

service-policy global_policy global
service-policy http_conn_limit_map interface outside

Reference:
http://rtomaszewski.blogspot.co.uk/2013/12/cisco-asa-connection-table-state.html
http://www.itlibrary.net/index.php/cisco-asa/8-limiting-connections-rate-for-traffic-destined-on-port-80
http://www.cisco.com/en/US/docs/security/asa/asa72/configuration/guide/mpc.html
http://blog.ine.com/2009/04/19/understanding-modular-policy-framework/

Kick off a client sessions

If you identify a client that you want to deny traffic and close all its connections.

access-list 101 extended deny ip host [ip] any
shun [ip]
no shun [ip]

Using Qemu on cloud server to run emulated virtual machines

We know that there is not support for nested hypervisors on cloud instances: Nested virtualization support on Rackspace public cloud.

Problem

How to use Qemu on cloud server and start virtual machine to overcome the nested virtualization limitation.

Demonstration and results description

To overcome this limitation we will use Qemu in its emulated mode. Qemu in this mode doesn't require any specials virtualization support in CPU (HVM - Hardware-assisted virtualization).

The VM image was downloaded from here: http://people.debian.org/~aurel32/qemu/i386/. The default u/p is root.

Alternatively we could use Virtualbox. Although I'm not quite sure what would work better yet and give more options to customizing the VMs.

References

https://wiki.debian.org/QEMU
http://www.linux-kvm.org/page/FAQ - this is more to show what can be missing as the CS don't support KVM
http://en.wikipedia.org/wiki/QEMU
http://www.linuxforu.com/2012/05/virtualisation-faceoff-qemu-virtualbox-vmware-player-parallels-workstation/

Nested virtualization support on Rackspace public cloud

We have found the CPU hardware architecture that the public cloud is running on in this blog: Hypervisor hardware differences on Openstack Rackspace Cloud.

Problem

Does Rackspace public cloud support nested visualization?

Results discussion

• Public Cloud

Of course for the cloud to exists the physical sever where the hypervisor runs (Xen or KVM for example) needs to have in-hardware virtualization support (Intel VT-x or AMD-V). This is the only way to provide a high performance cloud servers.

But once the cloud server boots up the cloud virtual CPU no longer exports the hardware CPU virtualization capabilities. You can verify this with this little script below.

egrep -i 'vmx|svm|ept|vpid|npt|tpr_shadow|flexpriority|vnmi'

That means you can't use your cloud server to run another, a guest hypervisor ( called as well nested hypervisor).

• Private cloud 

The nested virtualization can be enabled. As an example this link describe some of the steps for Linux KVM:

•  Installing Rackspace Private Cloud - KVM.
• http://networkstatic.net/nested-kvm-hypervisor-support/

Another solution

If your cloud server doesn't offer nested virtualization support you can always use the emulation mode. Qemu supports running VM that way.

References

http://www.ibm.com/developerworks/cloud/library/cl-nestedvirtualization/
https://www.diigo.com/user/rtomaszewski/nested_virtualization?type=all&snapshot=no&sort=updated
http://en.wikipedia.org/wiki/X86_virtualization

Hypervisor hardware differences on Openstack Rackspace Cloud

You can spin up test cloud servers and extract CPU flags with the help of this little script using csplit.

 
cat /proc/cpuinfo | csplit -z  - '/processor/' '{*}'
diff xx0*
grep flags xx01 | cut -d ':' -f 2 | xargs -n1 echo | sort > flags.txt

By comparing the results we can definitely say that:

• Performance1 and performance2 cloud servers are running on the same hardware.
• New performance cloud servers are hosted on Intel CPU.
• The standard (next generation) series are being hosted on AMD CPU.

The CPU flags for comparison.

 
processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : Quad-Core AMD Opteron(tm) Processor 2374 HE
stepping        : 2
microcode       : 0x1000086
cpu MHz         : 2200.096
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good nopl pni cx16 popcnt hypervisor lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch hw_pstate
bogomips        : 4400.19
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
stepping        : 7
microcode       : 0x70d
cpu MHz         : 2600.068
cache size      : 20480 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 43
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes hypervisor lahf_lm arat pln pts dtherm
bogomips        : 5200.13
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

How to divide and split file a part based on regex

We have a following file.

 
# cat text-to-split.txt
       1  aaa
       2  b1
       3  bb2
       4  bbb3
       5  c
       6  c1
       7  cc2
       8  aaa
       9  b1
      10  bb2
      11  c
      12  c1
      13  cc2
      14  aaa
      15  b1
      16  c
      17  cc2
      18  aaa
      19  b1
      20  aaa
      21  c1
      22  ccc

Problem

How to split and divide file based on its content?

Analisis and results description

Example 1 : Single split line in whole file

The file will be divided on each line matching a single patter.

 
# csplit -k text-to-split.txt '%aaa%' '/aaa/' '{*}'
74
62
41
22
35
root@perf1:~/split# for i in xx0*; do echo $i; cat -n $i; done
xx00
     1       1  aaa
     2       2  b1
     3       3  bb2
     4       4  bbb3
     5       5  c
     6       6  c1
     7       7  cc2
xx01
     1       8  aaa
     2       9  b1
     3      10  bb2
     4      11  c
     5      12  c1
     6      13  cc2
xx02
     1      14  aaa
     2      15  b1
     3      16  c
     4      17  cc2
xx03
     1      18  aaa
     2      19  b1
xx04
     1      20  aaa
     2      21  c1
     3      22  ccc

Example 2 : multiple split line

The csplit takes a variable number of regular expressions.
It scans the file and once a line matches the regex it splits the file at this point.
It evaluates then the next regular expression and continue to scan remaining file data.
When a match is found the file is split at this point again.
The last regex is used to split the remaining file until we read EOF.

In this example we:

• Jump to line containing b1 (don't copy and save the data - %)
• Continue searching for aaa and split when found.
• Continue searching for c1 and split when found.
• Use the last regex (c1) if file still have data.

csplit -k text-to-split.txt '%b1%' '/aaa/' '/c1/' '{*}'
63
41
96
23
root@perf1:~/split# for i in xx0*; do echo $i; cat -n $i; done
xx00
     1       2  b1
     2       3  bb2
     3       4  bbb3
     4       5  c
     5       6  c1
     6       7  cc2
xx01
     1       8  aaa
     2       9  b1
     3      10  bb2
     4      11  c
xx02
     1      12  c1
     2      13  cc2
     3      14  aaa
     4      15  b1
     5      16  c
     6      17  cc2
     7      18  aaa
     8      19  b1
     9      20  aaa
xx03
     1      21  c1
     2      22  ccc

References

http://rtomaszewski.blogspot.co.uk/2013/05/openssl-cheat-sheet.html

Practical online Neutron OVS Lab recording

This is worth re-posting. We learned in the previous post (Openstack Neutron architecture explained based on OVS and VMware NVP plugin comparison (*)) how does the Neutron architecture changes depending what plugins do we use.

In (*) the video #3 walks you through a live lab. Here you can find all the commands run.
https://github.com/rtomaszewski/experiments/blob/master/openstack_neutron_lab

Windows cheat sheet

msconfig - is a system utility to troubleshoot the Microsoft Windows startup process.

Openstack Neutron architecture explained based on OVS and VMware NVP plugin comparison

There are 3 recorded meetup videos that were organized by the onlinemeetup Openstack group (http://www.meetup.com/OpenStack-Online-Meetup/). These are an excellent source of information into network virtualization, nova-network, Neutron OVS and NVP plugins.

The recorded sessions can be found on YouTube here:

OpenStack Networking - Theory Session, Part 1
OpenStack Networking - Theory Session, Part 2
OpenStack Networking - Hands-On Lab, Part 3

Network virtualization basic

Nova Networking

The first version of network implemented in Openstack is called nova-networking and can be still used. Some of the advantages and limitations can be seen below.

The most complex deployment architecture used VLANs to implemented tenant and isolation. This scenario has a lot ideas that are then later shared in Neutron plugins.

OVS plugin

As you can see the architecture looks very similar. There are some subtle differences although like: instead of VLAN we use GRE tunnels, instead of Linux bridge we use the OpenVswitch (OVS). The important thing to note is that we don't use OpenFlow protocol to control the OVS switches. The switch will be pre-programmed by the agent running on the hypervisor.

NVP plugin

To describe and explain how NVP works it is good to compare its architecture to OVS plugin above. The first slide shows what component are not being used.

The network communication model with NVP provides new component.

The main differences are:

• OVS switches will be programmed by the NVP cluster using OpenFlow protocol
• Instead of GRE we use STT tunneling
• Security groups will be natively implemented in OVS (no need for iptables)
• The virtual router is highly available and is implemented on external nodes 

Debugging networking issues in Neutron

Overview of cloud networking and Neutron in Openstack

We saw what challenges a successful Neutron deployment in Openstack would have to overcome in this post: Status and maturity of the Neutron in Openstack Havana release. But the journey into cloud network is worth the trouble and you can find many discussions that show the potential business and technological advantage you gain (an example What your Dev, Engineering and sales teams can achieve by using Openstack cloud and Quantum network).

But as the Neutron continues to bring new and advance networking features with every consecutive  Openstack release ( Havana Neutron features) there is always a fear that it may become very complex and difficult to troubleshoot. To help to fill the gap the video from Dave Neary (slides are here Networking in OpenStack for non-networking people) is taking us on the journey what cloud networking and Neutron is as well as showing the basic troubleshooting every one should be aware of.

Neutron troubleshooting

These couple of slides from the video summarized very well what troubleshooting actions you can do when dealing with network connectivity problems from and to your VM. For more info watch the video yourself.

References

http://openstack.redhat.com/Main_Page
http://openstack.redhat.com/Networking
http://openstack.redhat.com/Networking_in_too_much_detail

OpenShift PaaS platform from Redhat

We've looked before at AppFog PaaS platform that is leveraging CloudFoundry and Openstack. Today we will investigate the Redhat PaaS alternative, namely OpenShift.

What is OpenShift

OpenShift is a Platform as a Service solution according the the cloud systems taxonomy. It offers a solution, a software development platform that facilitates easy and rapid application development in the cloud.

It can provide a preintegrated software stack environment and can leverage IaaS providers for the virtual infrastructure management (How to Deploy OpenShift Enterprise on Red Hat OpenStack).

A comprehensive presentation that provide further details can be found here: http://rhsummit.files.wordpress.com/2013/06/noceda_t_0120_consumepaasinthecloudwithopenshift.pdf

Demo

• Sign in for free account on https://openshift.redhat.com
• Login and from "My App" tab create new application by selecting "Add Application ..." button.
• I've used as simple Python app based on CherryPy framework. 
• Once your gear (a Openshift name for your app container) is created you can take a look at the hello world code 
• To modify the source code clone the git repo. You will need to first register your public key.
• Generate a new ssh key pair.

$  ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rado/.ssh/id_rsa): /tmp/tmp.key

$ ssh-agent | tee -a agent.sh
SSH_AUTH_SOCK=/tmp/ssh-pMmGGPP15635/agent.15635; export SSH_AUTH_SOCK;
SSH_AGENT_PID=15636; export SSH_AGENT_PID;
echo Agent pid 15636;

$source agent.sh
$ssh-add tmp.key
Identity added: tmp.key (tmp.key)

• Clone your source repo to modify your app source code.

$ git clone ssh://52c499a1e0b8cdda0f000016@rado-rado1stapp.rhcloud.com/~/git/rado.git/
Cloning into 'rado'...
remote: Counting objects: 62, done.
remote: Compressing objects: 100% (41/41), done.
remote: Total 62 (delta 16), reused 62 (delta 16)
Receiving objects: 100% (62/62), 18.93 KiB, done.
Resolving deltas: 100% (16/16), done.

• Modify the code to personalize it.

$cd rado/

$ find  | grep -v git
./.openshift
./.openshift/markers
./.openshift/action_hooks
./.openshift/action_hooks/README.md
./.openshift/cron
./.openshift/cron/minutely
./.openshift/cron/monthly
./.openshift/cron/weekly
./.openshift/cron/weekly/chrono.dat
./.openshift/cron/weekly/chronograph
./.openshift/cron/weekly/jobs.deny
./.openshift/cron/weekly/README
./.openshift/cron/weekly/jobs.allow
./.openshift/cron/README.cron
./.openshift/cron/daily
./.openshift/cron/hourly
./README.md
./LICENSE
./data
./libs
./wsgi
./wsgi/static
./wsgi/static/README
./wsgi/application
./app.py.disabled
./setup.py

$vim ./wsgi/application
$ cat wsgi/application
import sys
sys.stdout = sys.stderr

import atexit
import threading
import cherrypy

cherrypy.config.update({'environment': 'embedded'})

if cherrypy.__version__.startswith('3.0') and cherrypy.engine.state == 0:
    cherrypy.engine.start(blocking=False)
    atexit.register(cherrypy.engine.stop)

class Root(object):
    def index(self):
        return 'Hello from rado 1st app on openshift ;)!'
    index.exposed = True

application = cherrypy.Application(Root(), script_name=None, config=None)

• Once you are happy with t he code it is time to push changes back to git and redeploy it inside OpenShift.

$ git commit ./wsgi/application
[master 0eccd4d] init
 1 file changed, 1 insertion(+), 1 deletion(-)

$ git push origin master
Counting objects: 7, done.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 393 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Stopping PYTHON cart
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable SHELL was undefined
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable USER was undefined
remote: [Wed Jan 01 17:59:08 2014] [warn] PassEnv variable LOGNAME was undefined
remote: Waiting for stop to finish
remote: Building git ref 'master', commit 0eccd4d
remote: running develop
remote: running egg_info
remote: creating Example_CherryPy.egg-info
remote: writing requirements to Example_CherryPy.egg-info/requires.txt
remote: writing Example_CherryPy.egg-info/PKG-INFO
remote: writing top-level names to Example_CherryPy.egg-info/top_level.txt
remote: writing dependency_links to Example_CherryPy.egg-info/dependency_links.txt
remote: writing requirements to Example_CherryPy.egg-info/requires.txt
remote: writing Example_CherryPy.egg-info/PKG-INFO
remote: writing top-level names to Example_CherryPy.egg-info/top_level.txt
remote: writing dependency_links to Example_CherryPy.egg-info/dependency_links.txt
remote: writing manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: reading manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: writing manifest file 'Example_CherryPy.egg-info/SOURCES.txt'
remote: running build_ext
remote: Creating /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/Example-CherryPy.egg-link (link to .)
remote: Example-CherryPy 1.0 is already the active version in easy-install.pth
remote:
remote: Installed /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/repo
remote: Processing dependencies for Example-CherryPy==1.0
remote: Searching for CherryPy==3.2.4
remote: Best match: CherryPy 3.2.4
remote: Processing CherryPy-3.2.4-py2.7.egg
remote: CherryPy 3.2.4 is already the active version in easy-install.pth
remote: Installing cherryd script to /var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin
remote:
remote: Using /var/lib/openshift/52c499a1e0b8cdda0f000016/app-root/runtime/dependencies/python/virtenv/lib/python2.7/site-packages/CherryPy-3.2.4-py2.7.egg
remote: Finished processing dependencies for Example-CherryPy==1.0
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Preparing build for deployment
remote: Deployment id is f17888e7
remote: Activating deployment
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.fish cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Script /var/lib/openshift/52c499a1e0b8cdda0f000016/python//virtenv/bin/activate.csh cannot be made relative (it's not a normal script that starts with #!/var/lib/openshift/52c499a1e0b8cdda0f000016/python/virtenv/bin/python)
remote: Starting PYTHON cart
remote: Result: success
remote: Activation status: success
remote: Deployment completed with status: success

• Final test that all worked fine ;)

$ curl -v -s http://rado-rado1stapp.rhcloud.com ;echo
* About to connect() to rado-rado1stapp.rhcloud.com port 80 (#0)
*   Trying 54.221.76.112... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: rado-rado1stapp.rhcloud.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 01 Jan 2014 23:33:20 GMT
< Server: Apache/2.2.22 (Red Hat Enterprise Web Server)
< Content-Length: 40
< Content-Type: text/html;charset=utf-8
< Vary: Accept-Encoding
<
* Connection #0 to host rado-rado1stapp.rhcloud.com left intact
* Closing connection #0

Hello from rado 1st app on openshift ;)!

Redhat Cloudforms product explained

As the cloud technology continues to evolve we see even more new cool product names that vendors advertise. One of these is Cloudforms from Redhat. But what does Redhat Cloudforms do and where it can be used? The answer can be found in one of the slides from Redhat Summit 2013.

From Introduction to Red Hat Openstack slides we can learn that Cloudforms:

• Is a management platform for heterogeneous clouds.
• It does support RHVE, Openstack, Redhat RDO and proprietary virtualization solutions.

Bootstrapping a VM in Openstack environment

In this Openstack Architecture, by Russell Bryant presentation from Redhat 2013 summit we can see description of all Openstack components together with best practices how to deploy them to achieve maximum scalability. If you are interested in more details this link will help you to take the next level: Openstack software architecture.

The amount and verbosity of available information can be overwhelming and it may become difficult after reading all of it to answer a single question:

How does a single VM is created in Openstack and how does the Openstack systems interact together to achieve it.

The slide below shows the six steps and hides the necessary complexity (at least at the beginning, for another view take a look at Event flow when a cloud instance is provisioned in Openstack).