SSL certificate chain order matters

The certs we trust are usually stored in the CApath on Linux systems. The file is a simple text file with all the certs concatenated one after another.

Problem  

Does the order of certificated stored in the CAfile chain file matter for the client or server?

Analysis and verification  

The simple answer is it depends. As the certs from the CApath/CAfile are used by the client it is independent of the SSL/TLS server we are connecting to. The implementation details of the servers should matter.

That means that the certificate order is important only to the local client itself. In the SSL handshaking the content of this file is never sent to the server. An example handshaking can be found: here: http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_1-1/ssl.html.

To verify of the order of the certs matters for the openssl client we can run the following test. Both files ca1 and ca2 have the same certs but in different order. Example output.

$ openssl s_client -connect 1.1.1.1:443 -state -msg -CAfile ca1
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 2.0 [length 0077], CLIENT-HELLO
    01 03 01 00 4e 00 00 00 20 00 00 39 00 00 38 00
    ...
    ab 3b be 51 9d fa 43
SSL_connect:SSLv2/v3 write client hello A
<<< TLS 1.0 Handshake [length 002a], ServerHello
    02 00 00 26 03 01 2b ae 63 1e ec a0 82 a4 dc 25
    a9 4b 71 14 0a 54 2a ce 3d 6f 38 f5 26 e4 dd 8b
    7e e7 94 d5 02 b7 00 00 04 00
SSL_connect:SSLv3 read server hello A
<<< TLS 1.0 Handshake [length 0e16], Certificate
    11 11 0e 12 00 0e 0f 00 05 69 30 82 05 65 30 82
    22 22 a0 03 02 01 02 02 07 2b 86 02 70 e7 be 22
    ...
    09 0c 4d f6 a7 6b b4 99 84 65 ca 7a 88 e2 e2 44
    be 5c f7 ea 1c f5
depth=2 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 /OU=Domain Control Validated/CN=mydomain.mysite.com
verify return:1
SSL_connect:SSLv3 read server certificate A
<<< TLS 1.0 Handshake [length 0004], ServerHelloDone
    0e 00 00 00
SSL_connect:SSLv3 read server done A
>>> TLS 1.0 Handshake [length 0106], ClientKeyExchange
    11 10 21 32 11 10 13 11 13 11 10 7b 1c c1 d1 10
    ...
    81 1f 71 f1 10 12
SSL_connect:SSLv3 write client key exchange A
>>> TLS 1.0 ChangeCipherSpec [length 0001]
    01
SSL_connect:SSLv3 write change cipher spec A
>>> TLS 1.0 Handshake [length 0010], Finished
    11 11 11 1c 1f 13 6f 1d 11 12 1a 19 ed 64 e8 4b
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
<<< TLS 1.0 ChangeCipherSpec [length 0001]
    01
<<< TLS 1.0 Handshake [length 0010], Finished
    14 00 11 1c ed 9d fd 1f ab db ee ef 29 9a 1c 32
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=mydomain.mysite.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBCCCCCCCCCCCCCCDDDDDDDDEEEEEEFFFFF
...
111111111111111111111111111111111111111111111111111ah6I=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=mydomain.mysite.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
---
SSL handshake has read 3710 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID:
    Session-ID-ctx:
    Master-Key: 11861BE5828519468B6C59B0F01D3FF3126EA2B59DFB985E1C7D88B68E63BF399BCDEF7451D68421C2CE344765CDE572
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1392721077
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

References

http://blog.edgecloud.com/post/19519955133/ssl-certificate-chain-order-matters
http://stackoverflow.com/questions/8431528/nginx-ssl-certificate-authentication-signed-by-intermediate-ca-chain
http://rtomaszewski.blogspot.co.uk/search/label/openssl
http://jw35.blogspot.co.uk/2010/05/doing-certificate-verification-in.html

How to run my TLS server

Problem

How to quickly deploy and run SSL/TLS server for testing?

Solution

• You need first key and certificate. There are many guides on Internet how to generate these with the help of openssl (example http://www.akadia.com/services/ssh_test_certificate.html).
• Use openssl s_server to run start the server and customize the supported ciphers 

openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -tls1
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -ssl3
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -cipher DES-CBC-SHA:AES256-SHA

DES and IDEA ciphers are deprecated in the latest TLS protocol

When incorporating security into your solution and applications it is important to maintain a high level view and follow security best practices. That means you need a FW. The FW should have a DMZ and Inside segments. To actively protect your web applications you can deploy WAF or another kind of IPS. To passively monitor traffic you can implement IDS additionally.

But as our solution is being extend by new and more sophisticated network devices it is still important to understand and maintain the low level security parameters for the network protocols. When I mean low level I mean the low level details of the TLS/SSL network protocols that are being used when using HTTPS for example.

Problem

Is that secure or recommended to enable and support DES or IDEA ciphers in application or SSL-offloading load balancers?

Analysis and results discussion

According to RFC 5469 IDEA and DES should not be used any more. The reasons are listed in the RFC.

To verify if your server responds to clients using these ciphers you can try:

 
# (1)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -ssl3
# or
# (2)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -tls1
# or 
# (3)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA 
CONNECTED(00000003)
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
 0 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICATCCAWoCCQCxkFtlc6Bd0TANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB
VTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
cyBQdHkgTHRkMB4XDTE0MDExMjIxMzMwN1oXDTE1MDExMjIxMzMwN1owRTELMAkG
A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0
IFdpZGdpdHMgUHR5IEx0ZDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAtVtD
BfrmHU/T9m4xvvlP7+J4zJ2BYFY8QfSvQ1tyQw+BwvPyh9zyzgd0Zw4iOa6ThlQ3
GTr7e3FMQooMWpK0XXTYKbbWGqyVfnkcwmWjapJxOv8OaXlDS5TIc7MursFXp16e
oOjvpyuddX2gilQLiO6n1b6vyKsFfPW0eoPPmf8CAwEAATANBgkqhkiG9w0BAQUF
AAOBgQBGd8xD6ZINxy8Vf1jFrX+4EyPEL3+DkAU4lInd83kIuDd8i2fzia4YOfKh
JB3/ML8kLGLMh6R0WpHbaoGQvNM5qn7GdFL+DDBvXqlyZtIrfKamx+s5GxUiP0SV
5miO9Oh1mkxhXUqaVHaJR0DeTYEAuA0dc1lMoJlPoSMedlgJBg==
-----END CERTIFICATE-----
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
No client certificate CA names sent
---
SSL handshake has read 710 bytes and written 273 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC-SHA
    Session-ID: A5568C18EFB2DA77B729A247EA8E605BEBC4DF478129357D002C26DFA89F96C7
    Session-ID-ctx:
    Master-Key: F9CDF6CD91F3E4F5117758104906C779E18493062397EFFE7E4C518F0894398A01D969D5EE07804ED436A24444CD92FA
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Compression: 1 (zlib compression)
    Start Time: 1389565902
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

An example output from the (3) showing that the server supports the legacy and depreciated ciphers.

 
# ssldump -A -n -i lo port 443
New TCP connection #1: 127.0.0.1(50211) <-> 127.0.0.1(443)
1 1  0.0007 (0.0007)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 18 25 a9 86 1c 58 ff f0 90 ca fe ba f8 eb
          c8 23 46 fd 5b 7a 4a aa 51 c2 37 40 6a 8b dc 01
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0010 (0.0003)  S>CV3.2(58)  Handshake
      ServerHello
        Version 3.2
        random[32]=
          52 d3 18 25 95 9c 3e 34 80 d8 00 3d fe 02 8f bf
          3c 1a 72 5d d1 4f 30 8c 6c 3b fa 64 0e 82 1c 6c
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_DES_CBC_SHA
        compressionMethod                 unknown value
1 3  0.0021 (0.0011)  S>CV3.2(527)  Handshake
      Certificate
        certificate[517]=
          30 82 02 01 30 82 01 6a 02 09 00 b1 90 5b 65 73
          a0 5d d1 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05
          05 00 30 45 31 0b 30 09 06 03 55 04 06 13 02 41
          55 31 13 30 11 06 03 55 04 08 0c 0a 53 6f 6d 65
          2d 53 74 61 74 65 31 21 30 1f 06 03 55 04 0a 0c
          18 49 6e 74 65 72 6e 65 74 20 57 69 64 67 69 74
          73 20 50 74 79 20 4c 74 64 30 1e 17 0d 31 34 30
          31 31 32 32 31 33 33 30 37 5a 17 0d 31 35 30 31
          31 32 32 31 33 33 30 37 5a 30 45 31 0b 30 09 06
          03 55 04 06 13 02 41 55 31 13 30 11 06 03 55 04
          08 0c 0a 53 6f 6d 65 2d 53 74 61 74 65 31 21 30
          1f 06 03 55 04 0a 0c 18 49 6e 74 65 72 6e 65 74
          20 57 69 64 67 69 74 73 20 50 74 79 20 4c 74 64
          30 81 9f 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01
          05 00 03 81 8d 00 30 81 89 02 81 81 00 b5 5b 43
          05 fa e6 1d 4f d3 f6 6e 31 be f9 4f ef e2 78 cc
          9d 81 60 56 3c 41 f4 af 43 5b 72 43 0f 81 c2 f3
          f2 87 dc f2 ce 07 74 67 0e 22 39 ae 93 86 54 37
          19 3a fb 7b 71 4c 42 8a 0c 5a 92 b4 5d 74 d8 29
          b6 d6 1a ac 95 7e 79 1c c2 65 a3 6a 92 71 3a ff
          0e 69 79 43 4b 94 c8 73 b3 2e ae c1 57 a7 5e 9e
          a0 e8 ef a7 2b 9d 75 7d a0 8a 54 0b 88 ee a7 d5
          be af c8 ab 05 7c f5 b4 7a 83 cf 99 ff 02 03 01
          00 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05
          00 03 81 81 00 46 77 cc 43 e9 92 0d c7 2f 15 7f
          58 c5 ad 7f b8 13 23 c4 2f 7f 83 90 05 38 94 89
          dd f3 79 08 b8 37 7c 8b 67 f3 89 ae 18 39 f2 a1
          24 1d ff 30 bf 24 2c 62 cc 87 a4 74 5a 91 db 6a
          81 90 bc d3 39 aa 7e c6 74 52 fe 0c 30 6f 5e a9
          72 66 d2 2b 7c a6 a6 c7 eb 39 1b 15 22 3f 44 95
          e6 68 8e f4 e8 75 9a 4c 61 5d 4a 9a 54 76 89 47
          40 de 4d 81 00 b8 0d 1d 73 59 4c a0 99 4f a1 23
          1e 76 58 09 06
1 4  0.0021 (0.0000)  S>CV3.2(4)  Handshake
      ServerHelloDone
1 5  0.0085 (0.0063)  C>SV3.2(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          71 83 c8 f4 af ab be 5e a6 e0 ec 06 ab 14 be e3
          41 25 5f f9 9e b3 29 a1 a5 1a a9 25 8d c8 1e 3d
          f2 06 3b 50 68 58 ca 1b bf 9b 1a e5 3f 4d c7 f5
          43 67 93 a1 fc f8 16 9e 35 24 7f a6 4c ad 9b 0f
          c4 db 6e a8 3d 97 5e 5f 96 0f 40 7b a3 42 62 e4
          7c 07 f9 65 97 a4 52 1a 30 cc 11 d6 43 06 7d 85
          4b e9 d5 1e 2e af 9a bd 90 cd 4d 6e aa 9e 00 29
          07 12 cd 96 bd 59 ca 5c dc a3 88 00 53 6e 8f ec
1 6  0.0085 (0.0000)  C>SV3.2(1)  ChangeCipherSpec
1 7  0.0085 (0.0000)  C>SV3.2(56)  Handshake
1 8  0.0099 (0.0014)  S>CV3.2(170)  Handshake
      TLS_RSA_WITH_RC4_128_MD51 9  0.0476 (0.0376)  S>CV3.2(1)  ChangeCipherSpec
1 10 0.0476 (0.0000)  S>CV3.2(56)  Handshake
1    0.7913 (0.7436)  C>S  TCP FIN
1    0.7917 (0.0004)  S>C  TCP FIN

Output proving the ciphers are not supported.

 
# ssldump -A -n -i eth0 port 443 and host 31.222.129.61
New TCP connection #1: 162.13.0.27(34228) <-> 31.222.129.61(443)
1 1  0.0017 (0.0017)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 19 53 c5 78 4c 06 8c e7 fc 47 a1 92 ec a4
          90 63 ca a2 6e a5 7e 58 bb 72 9b a1 be c1 84 3a
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0021 (0.0003)  S>CV3.1(2)  Alert
    level           fatal
    value           handshake_failure
1    0.0021 (0.0000)  S>C  TCP FIN
1    0.0044 (0.0022)  C>S  TCP FIN

 
# openssl s_client -connect 31.222.129.61:443 -state -msg -cipher DES-CBC-SHA
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> TLS 1.1  [length 003b]
    01 00 00 37 03 02 52 d3 19 53 c5 78 4c 06 8c e7
    fc 47 a1 92 ec a4 90 63 ca a2 6e a5 7e 58 bb 72
    9b a1 be c1 84 3a 00 00 04 00 09 00 ff 02 01 00
    00 09 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state
SSL3 alert read:fatal:handshake failure
<<< TLS 1.0 Alert [length 0002], fatal handshake_failure
    02 28
SSL_connect:error in unknown state
139646822749888:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:741:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 64 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

References

DES and IDEA Cipher Suites for Transport Layer Security (TLS)
http://www.ietf.org/rfc/rfc5469.txt

The TLS Protocol, Version 1.0
http://www.ietf.org/rfc/rfc2246.txt

The Transport Layer Security (TLS) Protocol, Version 1.2
http://tools.ietf.org/html/rfc5246.txt

How to use wildcard certificate with alternative name extensions or server name indication (SNI) certificates

There are number of ways how you can incorporate security into your web site. One and the most common method is to use SSL/TLS protocol to create and maintain a secure channel between the client and server.

Normally, by default for every site (for example for every home page URLlike ww.example.com) that you want to protect you need to set up separate SSL/TLS configuration. The most important part of the configuration is the private key and certificate. In standard SSL deployments this leads to a situation that for every new site you have a new public IP that is tight through DNS to URL name that is used as a CN(common name) in the new certificate.

The security of the TLS/SSL protocol heavily depends on the method how the client verifies and confirms and  the identity of your site. The most common and the most important part of the client check is to evaluate and compare the site URL with the CN value embedded in the certificate.

From high level point of view to grantee your sites security you need to protect and mange all your private keys and certificates on all devices like web servers, load balancers etc.

Problem

How to use a single certificate to protect multiple different sites (domains).
How to use a single public IP to host multiple SSL sites.

Solution 1: wildcard plus alternative names

• Wildcard 

To use a single certificate for multiple sites we can use wildcard certificate. This certificate can be used for all domains with a shared name, like for example *.rado.com. There is a limitation that the wildcard can only be used to mask one single domain level name. That means:

subdomain1.rado.com - ok
sub2.subdomain1.rado.com - bad

• Alternative names

A certificate can be used to protect 2 and more different domains. For example: ww.rado.com and www.radoninja.com. All what you need is to provide one or more alternative names when registering and buying a certificate.

• Combine alternative names and wildcard in a single certificate

You can combine these to options. You can have a wildcard certificate with multiple alternative names using wildcard domains, example:

*.rado.com - CN

*.subdomain1.rado.com - alternative name to overcome the wildcard limitation

*.radoninja.com - alternative name for 2th domain

*.subdomain.radoninja.com - another alternative name, etc...

Solution 2

Alternatively to use a single certificate with multiple domains uou can use the newer TLS extension called SNI.

The disadvantage is that SNI is relatively new. There are some older web clients, for example Win XP or some mobile browsers that don't support it yet. That means that  your site may not be available for these clients if you supports only SNI.

Example

http://www.ssltools.com/certificate_lookup/www.wikipedia.org

SSL Certificate

Common Name : *.wikipedia.org 
Subject Alternative Names : *.wikipedia.org, wikipedia.org, m.wikipedia.org, *.m.wikipedia.org, wikibooks.org, m.wikibooks.org, *.wikibooks.org, *.m.wikibooks.org, wikidata.org, m.wikidata.org, *.wikidata.org, *.m.wikidata.org, wikimedia.org, m.wikimedia.org, *.wikimedia.org, *.m.wikimedia.org, wikimediafoundation.org, m.wikimediafoundation.org, *.wikimediafoundation.org, *.m.wikimediafoundation.org, wikinews.org, m.wikinews.org, *.wikinews.org, *.m.wikinews.org, wikiquote.org, m.wikiquote.org, *.wikiquote.org, *.m.wikiquote.org, wikisource.org, m.wikisource.org, *.wikisource.org, *.m.wikisource.org, wikiversity.org, m.wikiversity.org, *.wikiversity.org, *.m.wikiversity.org, wikivoyage.org, m.wikivoyage.org, *.wikivoyage.org, *.m.wikivoyage.org, wiktionary.org, m.wiktionary.org, *.wiktionary.org, *.m.wiktionary.org, mediawiki.org, *.mediawiki.org, m.mediawiki.org, *.m.mediawiki.org 
Issuer Name : DigiCert High Assurance CA-3 
Serial Number : 07:24:ee:a9:7c:55:f2:57:5e:28:8b:a4:cc:f2:0e:8e 
SHA1 Thumbprint : DA:AA:A4:9B:AD:0C:1F:A3:29:71:D8:CC:62:BA:72:D1:A4:DB:94:9F 
Key Length : 2048 bit 
Signature Algorithm : sha1WithRSAEncryption 
Secure Renegotiation: Supported

References

http://en.wikipedia.org/wiki/Server_Name_Indication
http://en.wikipedia.org/wiki/Subject_Alternative_Name
http://stackoverflow.com/questions/2115611/wildcard-ssl-on-sub-subdomain

http://en.wikipedia.org/wiki/Server_Name_Indication
http://www.delantek.com/san.html
https://devcentral.f5.com/articles/multiple-certs-one-vip-tls-server-name-indication-via-irules#.Un832_lpmYI

3.1.  Server Identity
http://www.ietf.org/rfc/rfc2818.txt

http://www.networksorcery.com/enp/protocol/tls.htm

How to recognise encrypted private SSL key

Problem

How to recognise an encrypted private key.

Solution

You can recognise and see if a key is encrypted by looking at the first lines of your private SSL key file. Below is an example:

• Key is encrypted  

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,123456ABCD719580

diYueU...
-----END RSA PRIVATE KEY-----

• Key is not encrypted 

-----BEGIN RSA PRIVATE KEY-----
MIIEogI...
-----END RSA PRIVATE KEY-----

How to host 2 ssl sites on a single public IP on F5 load balancer

As the pool of free IPv4 is getting lower it is important to efficiently mange our existing assigned public IPs. One thing that can be used is to use SSL offloading and hosting multiple HTTPS sites on single public IP using wild cart certificates.

The core of this solution is the configuration of the lb and the vhost setting on your server. The server reads the HTTP Host header and base on it decide what site the user try to access.

Problem

How to configure servers and load balancer to host 2 ssl sites on a single public IP.

Analisis and example configuration

• Example F5 configuration

For SSL to work we need a wild card certificate.
We have to import the certificate and key to F5 and create client site ssl profile.

Example wild cart certificate:

 
# openssl x509 -in /var/tmp/wildcard.rado.net.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:9c:4a:4b:11:11:bc
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Arizona, L=Scottsdale, O=GoDaddy.com, Inc., OU=http://certificates.godaddy.com/repository, CN=Go Daddy Secure Certification Authority/serialNumber=111111
        Validity
            Not Before: Nov 21 11:57:16 2012 GMT
            Not After : Nov 20 09:31:37 2013 GMT
        Subject: O=*.rado.net, OU=Domain Control Validated, CN=*.rado.net
        ....

SSL profile:

 
# tmsh list ltm profile client-ssl  wildcard-client-ssl-profile
ltm profile client-ssl wildcard-client-ssl-profile {
    cert wildcard-rado.net.crt
    chain CA.crt
    defaults-from clientssl
    key wildcard-rado.net.key
}

Virtual server to terminate and load balance the traffic:

 
# tmsh list ltm virtual VS-1.1.1.1-443
ltm virtual VS-1.1.1.1-443 {
    destination 192.168.99.68:https
    ip-protocol tcp
    mask 255.255.255.255a
    pool POOL-192.168.99.68-80
    profiles {
        http { }
        tcp { }
        wildcard-client-ssl-profile {
            context clientside
        }
    }
}

Servers where the traffic is going to be load balanced:

 
# tmsh list ltm pool POOL-192.168.99.68-80
ltm pool POOL-192.168.99.68-80 {
    load-balancing-mode least-connections-member
    members {
        10.177.1.1:http {
            session monitor-enabled
        }
        10.177.1.2:http {
            session monitor-enabled
        }
    }
    monitor http
}

• Apache configuration on one of the servers


This is a simple example configuration for the http server.


 
# cat vhost1.conf
<VirtualHost *:80>
        ServerName vhost1.rado.net
        DocumentRoot /var/www/vhost1
</VirtualHost>

# cat vhost2.conf
<VirtualHost *:80>
        ServerName vhost2.rado.net
        DocumentRoot /var/www/vhost2
</VirtualHost>

# cat /var/www/vhost1/index.html
<html><body><h1>It works!</h1>
<p>
This is VHOST 1
</p>
</body></html>

# a2ensite vhost1.conf 
# a2ensite vhost2.conf 

# service apache2 reload


A quick local test on the server will confirm if the config is correct.


 
# curl -v -H "Host: vhost1.rado.net" http://5.1.1.1
* About to connect() to 5.1.1.1 port 80 (#0)
*   Trying 5.1.1.1... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: vhost1.rado.net
>
< HTTP/1.1 200 OK
< Date: Sat, 30 Mar 2013 23:42:20 GMT
< Server: Apache/2.2.22 (Ubuntu)
< Last-Modified: Sat, 30 Mar 2013 23:19:21 GMT
< ETag: "b46bb-47-4d92c9e17f040"
< Accept-Ranges: bytes
< Content-Length: 71
< Vary: Accept-Encoding
< Content-Type: text/html
<
<html><body><h1>It works!</h1>
<p>
This is VHOST 1
</p>
</body></html>
* Connection #0 to host 5.79.21.166 left intact
* Closing connection #0

How to calculate a number of new SSL/TCP connections per every 10ms

Hardware load balancers like F5 are a graet product that offers a lot of featreus still combined with a simple and intuitive management GUI. The only problem is the price you have to pay to buy it and than further to pay the support and the license fees.

When working with F5 I have run once into an interesting SSL/TLS problem. It is documented and described SOL6475: Overview of SSL TPS licensing limits.

The most important part from the solutions is:

The BIG-IP system measures SSL TPS based on client-side connection attempts to any
virtual server configured with a Client SSL profile. SSL TPS is enforced across a
sliding time window. The BIG-IP system utilizes a 10ms window (1/100 of a second)
to calculate the current TPS. If the number of TPS requests within any 10ms window
exceeds 1/100 of the licensed TPS, an error message regarding the TPS limit being
reached is sent to the /var/log/ltm file.

Problem

How to know what clients IPs cause the error to be logged. How to measure and calculate the number of SSL connection per seconds for even 10ms.

Solution

As there are no tools on F5 that helps you to find this out I thought that a simple way to get some visibility of it would be to capture all TCP SYN packets hitting the LB and then later do some analysis of it. An implementation of this ideas in a form of a python script can be found here [1].

Demonstration

To test sslAnalyze.py script we need first a tcpdump file. For this purpose we can use the nmap command and run a SYN flood. For the desciption of the nmap options you can take a look here [2].

$ nmap -P0 -TNormal -D 1.2.3.4,1.2.3.5,1.2.3.6,1.2.3.7,1.2.3.8,1.2.3.9,1.2.3.10 -iR 10

All what we have to do now is to run on one session a tcpump and on the other the nmap command. As we are only interested in the TCP SYN packets we should tailor the tcpdump filtering syntax properly. A tcpdump that will capture only the SYN packets:

$ tcpdump -vvv -nn -i eth0 -w /var/tmp/syn-flood-example.pcap 'tcp[13]&2!=0 and tcp[13]&16==0' 

All what we have to do is not run our script to see the statistics.

I have to quickly explain the script itself. Once run it will prints on stdout a listing of found connections and additionally will create a log file with a name sslConnHigh.txt for only these connections that are over the threshold.

The parameters that you have to specify are:

• param1 - tcpdump file (it has to have only SYN packets) 
• param2 - time fractions in microseconds ( 1000000 microseconds -> 1 second ) 
• param3 - connection threshold per time to log this result to a sslConnHigh.txt file

Examples

# Example 1: to see a  number of connection per 1 second 

$ python sslAnalyze.py  syn-flood-example.pcap 1000000 1

# Example 2: to see a number of connection per every 500ms (half a second)

$ python sslAnalyze.py  syn-flood-example.pcap 500000 1

# Example3: to see a number of connection per every 500ms (half a second) and log only
# these timestamps that have more than 100 connection in a single half a second
# some example output has been attached as well below

$ python sslAnalyze.py  syn-flood-example.pcap 500000 100

keeping the line: reading from file syn-flood-example.pcap, link-type EN10MB (Ethernet)
                     date     timestamp     sumOfConn [... 500000 microsecond periods ... ]
 Tue Aug 14 23:33:30 2012    1344983610       sum:183     0  183 
 Tue Aug 14 23:33:31 2012    1344983611        sum:95     6   89 
 Tue Aug 14 23:33:32 2012    1344983612       sum:614   430  184 
 Tue Aug 14 23:33:33 2012    1344983613       sum:520   216  304 

To better understand why F5 logs the error message and what trigger the TPS log error messages we have to run this command:

# 10 milliseconds = 10000 microseconds
$ python sslAnalyze.py  syn-flood-example.pcap 10000 [F5_SSL_total_TPS]
$ cat sslConnHigh.txt

In the output you are going to see the timestamps (rounded to 1 second) where the number of connections in a single 10ms window are above the licensing limit you device has. For further analize you can extract these data from the tcpdump with a help of tcpslice tool.

# 1268649656 is an example timestamp from above
$ tcpslice 1268649656  +1 syn-flood-example.pcap -w 1268649656.pcap

$ tcpdump -tt -nr 1268649656.pcap

reading from file 1268649656.pcap, link-type EN10MB (Ethernet)
1268649656.042723 vlan 4093, p 0, IP 19.26.168.192.4598 - 19.26.225.215.443: S 2973530156:2973530156(0) win 64512 mss 1460,nop,nop,sackOK
1268649656.056163 vlan 4093, p 0, IP 19.89.139.199.1622 - 19.26.225.23.443: S 1522394445:1522394445(0) win 64512 mss 1460,nop,wscale 0,nop,nop,sackOK

References

1. https://github.com/rtomaszewski/experiments/blob/master/sslAnalyze.py
2. http://www.hcsw.org/reading/nmapguide.txt
3. http://danielmiessler.com/study/tcpdump/