Search This Blog

Showing posts with label openssl. Show all posts
Showing posts with label openssl. Show all posts

Sunday, April 13, 2014

Description and demonstration of the Heartbleed bug in OpenSSL

There is a ton of posts on the Internet about the new bug in OpenSSL. I'm not going to repeat what others wrote  but rather give us a small demonstration.

Heartbeat packet description in SSL protocol suite

This is excellent blog posts we can take a look at the openssl code analysis and see where exactly the bug was hidden: Diagnosis of the OpenSSL Heartbleed Bug.

If you want to learn more how to build an potential exploid you can read and watch this: http://security.stackexchange.com/questions/55116/how-exactly-does-the-openssl-tls-heartbeat-heartbleed-exploit-work

A working code for a prof of concept can be found here:
http://www.garage4hackers.com/entry.php?b=2551
http://nakedsecurity.sophos.com/2014/04/08/anatomy-of-a-data-leak-bug-openssl-heartbleed/

Demonstration

How do I know if my site is vulnerable?

There are potentially many different ways how you can test if a site is vulnerable. As two extreme examples (a) we could write a simple SSL client and try to sent an hearbeat packet (not so trivial and requires some knowledge about the ssl protocol itself) or (b) search for a site on Internet that do the testing for us. I would definitively avoid (b). These sites can store the URL you provided and try to exploit you later.

A more simple and elegant solution can be built using openssl cli client tool instead. By running as single line script you can test if a server supports heartbeat or not. Next you have to find if the version of the OpenSSL you use is vulnerable.
 
$ openssl s_client -connect www.cloudflarechallenge.com:443 -tlsextdebug
CONNECTED(00000003)
TLS server extension "renegotiation info" (id=65281), len=1
0001 - <SPACES/NULS>
TLS server extension "EC point formats" (id=11), len=4
0000 - 03 00 01 02                                       ....
TLS server extension "session ticket" (id=35), len=0
TLS server extension "heartbeat" (id=15), len=1
0000 - 01                                                .
depth=4 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO Certification Authority
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
 3 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN - DATACorp SGC
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 4 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFLTCCBBWgAwIBAgIQSkGkHc+NJGGLqUs9YZlcxDANBgkqhkiG9w0BAQUFADBy
MQswCQYDVQQGEwJHQjEbMBkGA1UECBMSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYD
VQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0EgTGltaXRlZDEYMBYGA1UE
AxMPRXNzZW50aWFsU1NMIENBMB4XDTE0MDQxMDAwMDAwMFoXDTE0MDcwOTIzNTk1
OVowWDEhMB8GA1UECxMYRG9tYWluIENvbnRyb2wgVmFsaWRhdGVkMREwDwYDVQQL
EwhGcmVlIFNTTDEgMB4GA1UEAxMXY2xvdWRmbGFyZWNoYWxsZW5nZS5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCbQBaRWcPHl945y10L3tm2C+13
bm4oqaGMIekvJyYTF7VGJFKX+EYgvt/wWD+qJTO1Wbm5dknVQbt3PP7061M2H6/b
sG3M+xTfKK8d6/AAHWZMy0/ps+5cGPOzFFwL3JVwEFakoExGc3jT6S9RlhU5q4I+
q8Qd+jpHL7uKeklipCb8VIznRmtGKYI7H01kjyW8gwXYOrWKlKCHOIcR32LIxHfd
fv72QjT2kGupne3TmXAY+6cEL12ZqS2HCYpGBa8QQaZ7/dggc1X5OJL1yrQP8Le9
/faCOBHn0A4yzNp873BVMQ+7T+7k2PCSs7qAfB0TdvdfQFiPPFaTODDtPWClAgMB
AAGjggHXMIIB0zAfBgNVHSMEGDAWgBTay+qtWwhdzP/8JlTOSeVVxjj0+DAdBgNV
HQ4EFgQUbqyvF2sHtsjg5i82wBON35elvNQwDgYDVR0PAQH/BAQDAgWgMAwGA1Ud
EwEB/wQCMAAwNAYDVR0lBC0wKwYIKwYBBQUHAwEGCCsGAQUFBwMCBgorBgEEAYI3
CgMDBglghkgBhvhCBAEwTwYDVR0gBEgwRjA6BgsrBgEEAbIxAQICBzArMCkGCCsG
AQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8uY29tL0NQUzAIBgZngQwBAgEw
OwYDVR0fBDQwMjAwoC6gLIYqaHR0cDovL2NybC5jb21vZG9jYS5jb20vRXNzZW50
aWFsU1NMQ0EuY3JsMG4GCCsGAQUFBwEBBGIwYDA4BggrBgEFBQcwAoYsaHR0cDov
L2NydC5jb21vZG9jYS5jb20vRXNzZW50aWFsU1NMQ0FfMi5jcnQwJAYIKwYBBQUH
MAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTA/BgNVHREEODA2ghdjbG91ZGZs
YXJlY2hhbGxlbmdlLmNvbYIbd3d3LmNsb3VkZmxhcmVjaGFsbGVuZ2UuY29tMA0G
CSqGSIb3DQEBBQUAA4IBAQBlN1564xpz0f0EnCh5dKOjo6uk+kbLzEhkfaGd5Ydi
4diFQ9VYx3+Le1JCB/bDHMVUfwlqTpV0Eq8DZIWTO5wnP9BlRDiljVe7+y/jkQ/b
/B88kmBr2jjR9Aet1l8hOrqJycw6Ack6F+5hd/lYIvZ/0YH+h/qu9/Z6ii6rcUCd
UWERSKiTFsbM8PRmG/Cwb4Jm52N8ev6mcVYmxeBYIPmf51HBHEakN13oQcubCAjd
V9/8CugEMrl56lUpt7BYZMET2h4NyCDrfTlbFcDqQC+YBr5dLDOvLpe7T7Dv+r1P
wYJ+R0A4JC0F2RdUeIBWC5CycJcTx4h7ZSlNeWtFrZgJ
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=Free SSL/CN=cloudflarechallenge.com
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=EssentialSSL CA
---
No client certificate CA names sent
---
SSL handshake has read 6784 bytes and written 376 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: EF16DB45C3D67F69A480645C5267C4FDC44F41FD4CF4911194E986FC21E72F62
    Session-ID-ctx:
    Master-Key: 9DF3223AAF1520D6437E643E83E4AD5B1A590776F375B7ED082E024F3EC9EB43617A0D1F7715DF299EA483F905095465
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - c5 00 41 79 f6 38 12 30-bf 5f 85 54 f7 93 09 1c   ..Ay.8.0._.T....
    0010 - c1 60 e2 23 ca 90 8f 17-0c 4a 9f db cc 40 0e ea   .`.#.....J...@..
    0020 - 55 b0 f8 49 f1 7e b0 4e-78 0f 36 4a 58 3a 60 e2   U..I.~.Nx.6JX:`.
    0030 - b4 2b 22 a2 49 e8 c5 42-d0 00 ad a6 ec 49 b3 4d   .+".I..B.....I.M
    0040 - 28 b1 c3 ad 03 c6 53 de-a3 e7 ec c8 aa ed 5e 97   (.....S.......^.
    0050 - 75 12 5e 9f 5f eb cf a9-4a ab b7 85 bf cd e0 12   u.^._...J.......
    0060 - 2c ec 0b 05 4f cf ac 16-e9 65 40 1b a8 60 dc 3a   ,...O....e@..`.:
    0070 - 99 a0 cf 7a 65 0b 4c 74-a5 fc a5 16 11 48 e2 94   ...ze.Lt.....H..
    0080 - 19 0e 17 a8 03 d0 d0 4b-a4 14 7e 49 05 75 36 65   .......K..~I.u6e
    0090 - d4 70 63 fa a7 92 5a 14-63 97 00 cf 6b 5b 45 36   .pc...Z.c...k[E6

    Start Time: 1397426832
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
GET /heartbleed HTTP/1.1
Host: www.cloudflarechallenge.com

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 13 Apr 2014 22:02:32 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=86400

f61
<!doctype html>
<html>
<head>
  <title>Heartbleed Challenge</title>

From the output we can see that:
  • We connect to the server
  • There are many packages exchange between the client (our openssl cli tool) and the web server; the packets types and formats are defined in the relevant RFC documents for SSL/TLS
  • Option tlsextdebug instructs openssl to print out TLS extensions the server supports
  • We can immediately see if the option is supported by our www server; what we have o do next is to check if the version of OpenSSL that we run is vulnerable or not 
  • It is important to note that regardless if the www server supports the heartbeat extension or not you as a client can sent any legitimate HTTP requests; the whole problem is that if your client sent an heartbeat packet that was on purpose malicious the server in its response can reveal a lot more data that it should.
References

http://www.openssl.org/docs/apps/s_client.html
http://www.theregister.co.uk/2014/04/09/heartbleed_explained/
https://www.cloudflarechallenge.com/heartbleed


Tuesday, February 18, 2014

SSL certificate chain order matters

The certs we trust are usually stored in the CApath on Linux systems. The file is a simple text file with all the certs concatenated one after another.

Problem  

Does the order of certificated stored in the CAfile chain file matter for the client or server?

Analysis and verification  

The simple answer is it depends. As the certs from the CApath/CAfile are used by the client it is independent of the SSL/TLS server we are connecting to. The implementation details of the servers should matter.

That means that the certificate order is important only to the local client itself. In the SSL handshaking the content of this file is never sent to the server. An example handshaking can be found: here: http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_1-1/ssl.html.

To verify of the order of the certs matters for the openssl client we can run the following test. Both files ca1 and ca2 have the same certs but in different order. Example output.

$ openssl s_client -connect 1.1.1.1:443 -state -msg -CAfile ca1
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 2.0 [length 0077], CLIENT-HELLO
    01 03 01 00 4e 00 00 00 20 00 00 39 00 00 38 00
    ...
    ab 3b be 51 9d fa 43
SSL_connect:SSLv2/v3 write client hello A
<<< TLS 1.0 Handshake [length 002a], ServerHello
    02 00 00 26 03 01 2b ae 63 1e ec a0 82 a4 dc 25
    a9 4b 71 14 0a 54 2a ce 3d 6f 38 f5 26 e4 dd 8b
    7e e7 94 d5 02 b7 00 00 04 00
SSL_connect:SSLv3 read server hello A
<<< TLS 1.0 Handshake [length 0e16], Certificate
    11 11 0e 12 00 0e 0f 00 05 69 30 82 05 65 30 82
    22 22 a0 03 02 01 02 02 07 2b 86 02 70 e7 be 22
    ...
    09 0c 4d f6 a7 6b b4 99 84 65 ca 7a 88 e2 e2 44
    be 5c f7 ea 1c f5
depth=2 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
verify return:1
depth=1 /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
verify return:1
depth=0 /OU=Domain Control Validated/CN=mydomain.mysite.com
verify return:1
SSL_connect:SSLv3 read server certificate A
<<< TLS 1.0 Handshake [length 0004], ServerHelloDone
    0e 00 00 00
SSL_connect:SSLv3 read server done A
>>> TLS 1.0 Handshake [length 0106], ClientKeyExchange
    11 10 21 32 11 10 13 11 13 11 10 7b 1c c1 d1 10
    ...
    81 1f 71 f1 10 12
SSL_connect:SSLv3 write client key exchange A
>>> TLS 1.0 ChangeCipherSpec [length 0001]
    01
SSL_connect:SSLv3 write change cipher spec A
>>> TLS 1.0 Handshake [length 0010], Finished
    11 11 11 1c 1f 13 6f 1d 11 12 1a 19 ed 64 e8 4b
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
<<< TLS 1.0 ChangeCipherSpec [length 0001]
    01
<<< TLS 1.0 Handshake [length 0010], Finished
    14 00 11 1c ed 9d fd 1f ab db ee ef 29 9a 1c 32
SSL_connect:SSLv3 read finished A
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=mydomain.mysite.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
 1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
 2 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
AAAAAAAAAAAAAAAABBBBBBBBBBBBBBBCCCCCCCCCCCCCCDDDDDDDDEEEEEEFFFFF
...
111111111111111111111111111111111111111111111111111ah6I=
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/CN=mydomain.mysite.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
No client certificate CA names sent
---
SSL handshake has read 3710 bytes and written 431 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-MD5
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-MD5
    Session-ID:
    Session-ID-ctx:
    Master-Key: 11861BE5828519468B6C59B0F01D3FF3126EA2B59DFB985E1C7D88B68E63BF399BCDEF7451D68421C2CE344765CDE572
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1392721077
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

References

http://blog.edgecloud.com/post/19519955133/ssl-certificate-chain-order-matters
http://stackoverflow.com/questions/8431528/nginx-ssl-certificate-authentication-signed-by-intermediate-ca-chain
http://rtomaszewski.blogspot.co.uk/search/label/openssl
http://jw35.blogspot.co.uk/2010/05/doing-certificate-verification-in.html



Sunday, January 12, 2014

During TLS handshaking server can ignore a cipher from the preference list in ClientHello message

I've noticed recently that when my client tries to negotiated a TLS session its ciphers preference list is being ignored by the server.
  • Test 1
$ openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
  • Test 2
$ openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA
  • Test 3
# openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA:AES128-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: C7E7F65B4E927AFDE568E52FDAE52495CE815DAC2958B82C854146CD383FC00BB47ED691840C713CE762B6430FCB3230
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568347
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
  • Test 4
# openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA:DES-CBC-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: 1F0FB31410AAFFC234DC5598FFA2A676C63119A6367453D05578F9B1652DD085DB1D5C016694B9D293F3A2DFF769EF51
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568389
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

We can see in (1) and (2) that the TLS servers supports 2 different ciphers suits.
In (3) and (4) we see that even though the preference list was directly specified by the client the TLS server ignored the settings in (3).

Example ssldump output from (3).
 
# ssldump -A -n -i eth0 port 443 and host my_server_ip
New TCP connection #1: 162.13.0.27(51869) <-> my_server_ip(443)
1 1  0.1013 (0.1013)  C>SV3.1(61)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 23 b5 62 f5 08 38 14 53 d9 ac a4 51 c2 28
          0d e2 0e b6 41 89 2d af a1 50 55 bd a9 35 05 75
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.1984 (0.0970)  S>CV3.1(49)  Handshake
      ServerHello
        Version 3.1
        random[32]=
          2b ae 63 1e 9c da f3 38 b6 f9 dd c6 c0 4f ad 6c
          2c f0 f3 52 67 6e 28 29 8a 24 95 95 a5 67 f2 6f
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_AES_128_CBC_SHA
        compressionMethod                   NULL
1 3  0.1984 (0.0000)  S>CV3.1(674)  Handshake
      Certificate
        certificate[664]=
          50 82 02 94 50 82 01 fd 02 02 03 79 50 0d 06 09
          2a 86 48 86 f7 0d 01 01 05 05 00 50 81 b6 31 0b
          50 11 06 03 55 04 06 13 02 55 53 31 0e 50 0c 06
          03 55 04 08 13 05 54 65 78 61 73 31 14 50 12 06
          03 55 04 07 13 0b 53 61 6e 20 41 6e 74 6f 6e 69
          6f 31 12 50 10 06 03 55 04 0a 13 11 52 61 63 6b
          73 70 61 63 65 31 1e 50 1c 06 03 55 04 0b 13 15
          53 79 73 74 65 6d 20 41 64 6d 69 6e 69 73 74 72
          61 74 69 6f 6e 31 23 50 21 06 03 55 04 03 13 1a
...
          b3 79 0a 37 cd 27 93 af
1 4  0.1984 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
1 5  0.2126 (0.0141)  C>SV3.1(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          33 25 53 17 c2 45 b0 32 8a ca c1 66 39 6e f0 31
          98 96 4c 34 f2 e4 fd b7 0c e6 15 af c7 d3 fc e4
          0d 15 c9 c9 d4 e0 78 5a a1 13 dc 55 8b 5a bc 69
          68 24 f5 d1 50 6a 19 2e 71 9a 66 ee 3a 64 bc 1e
          d8 9a da d4 e0 44 96 b3 43 20 f1 a0 b6 4c 49 8e
          b2 ae 2b a6 12 68 78 19 eb 61 06 1c 34 8a 03 22
          ab 7e ff f8 88 44 89 97 cd 53 06 b0 b6 66 7b 77
          2a c0 0a 15 a3 54 2d 8c 5b 74 bc fe 31 3f 1f 8f
1 6  0.2126 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
1 7  0.2126 (0.0000)  C>SV3.1(48)  Handshake
1 8  0.3100 (0.0974)  S>CV3.1(1)  ChangeCipherSpec
1 9  0.3100 (0.0000)  S>CV3.1(48)  Handshake
1    1.4782 (1.1681)  C>S  TCP FIN
1    1.5740 (0.0958)  S>C  TCP FIN

Problem

Can TLS server ignore the cipher suite list passed from the client during TLS handshaking?

Analysis

The specification how the ClientHello message can be handled by the server during TLS handshaking has changed from TLS v1.0 to v1.2. The extract from the RFC comparing both relevant sections.
  • rfc2246
7.4.1.2. Client hello

   The CipherSuite list, passed from the client to the server in the
   client hello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first). Each CipherSuite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length) and a MAC algorithm.

   The server will select a cipher suite or, if no acceptable choices 
   are presented, return a handshake failure alert and close the 
   connection.
  • rfc5246
7.4.1.2.  Client Hello

   The cipher suite list, passed from the client to the server in the
   ClientHello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first).  Each cipher suite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length), a MAC algorithm, and a PRF.  The server will select a cipher
   suite or, if no acceptable choices are presented, return a handshake
   failure alert and close the connection.

   If the list contains cipher suites the server does not recognize, 
   support, or wish to use, the server MUST ignore those cipher suites,
   and process the remaining ones as usual.

In the yellow parts we can see that a server implementing TLS v1.2 has the ability to ignore ciphers specified by the client in its list ClientHello message. The criteria are very wide. The TLS server can ignore a cipher only because it doesn't wish to use it.

References

http://www.openssl.org/docs/apps/ciphers.html
DES and IDEA ciphers are deprecated in the latest TLS protocol
http://www.ietf.org/rfc/rfc2246.txt - The TLS Protocol, Version 1.0
http://tools.ietf.org/html/rfc5246.txt - The Transport Layer Security (TLS) Protocol, Version 1.2
http://support.f5.com/kb/en-us/solutions/public/10000/200/sol10209.html - ssldump

A definitely interesting reading about ciphers in the web browser to show some parasitical view at the cipher list topic.
https://groups.google.com/forum/#!topic/mozilla.dev.tech.crypto/36na1B2brGU

How to run my TLS server

Problem

How to quickly deploy and run SSL/TLS server for testing?

Solution
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -tls1
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -ssl3
openssl s_server  -accept 443 -key ./server.key -cert server.crt -state -debug -cipher DES-CBC-SHA:AES256-SHA

DES and IDEA ciphers are deprecated in the latest TLS protocol

When incorporating security into your solution and applications it is important to maintain a high level view and follow security best practices. That means you need a FW. The FW should have a DMZ and Inside segments. To actively protect your web applications you can deploy WAF or another kind of IPS. To passively monitor traffic you can implement IDS additionally.

But as our solution is being extend by new and more sophisticated network devices it is still important to understand and maintain the low level security parameters for the network protocols. When I mean low level I mean the low level details of the TLS/SSL network protocols that are being used when using HTTPS for example.

Problem

Is that secure or recommended to enable and support DES or IDEA ciphers in application or SSL-offloading load balancers?

Analysis and results discussion

According to RFC 5469 IDEA and DES should not be used any more. The reasons are listed in the RFC.

To verify if your server responds to clients using these ciphers you can try:
 
# (1)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -ssl3
# or
# (2)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA -tls1
# or 
# (3)
# openssl s_client -connect server_ip:443 -cipher DES-CBC-SHA 
CONNECTED(00000003)
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
 0 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
   i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
MIICATCCAWoCCQCxkFtlc6Bd0TANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB
VTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
cyBQdHkgTHRkMB4XDTE0MDExMjIxMzMwN1oXDTE1MDExMjIxMzMwN1owRTELMAkG
A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0
IFdpZGdpdHMgUHR5IEx0ZDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAtVtD
BfrmHU/T9m4xvvlP7+J4zJ2BYFY8QfSvQ1tyQw+BwvPyh9zyzgd0Zw4iOa6ThlQ3
GTr7e3FMQooMWpK0XXTYKbbWGqyVfnkcwmWjapJxOv8OaXlDS5TIc7MursFXp16e
oOjvpyuddX2gilQLiO6n1b6vyKsFfPW0eoPPmf8CAwEAATANBgkqhkiG9w0BAQUF
AAOBgQBGd8xD6ZINxy8Vf1jFrX+4EyPEL3+DkAU4lInd83kIuDd8i2fzia4YOfKh
JB3/ML8kLGLMh6R0WpHbaoGQvNM5qn7GdFL+DDBvXqlyZtIrfKamx+s5GxUiP0SV
5miO9Oh1mkxhXUqaVHaJR0DeTYEAuA0dc1lMoJlPoSMedlgJBg==
-----END CERTIFICATE-----
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd
---
No client certificate CA names sent
---
SSL handshake has read 710 bytes and written 273 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
SSL-Session:
    Protocol  : SSLv3
    Cipher    : DES-CBC-SHA
    Session-ID: A5568C18EFB2DA77B729A247EA8E605BEBC4DF478129357D002C26DFA89F96C7
    Session-ID-ctx:
    Master-Key: F9CDF6CD91F3E4F5117758104906C779E18493062397EFFE7E4C518F0894398A01D969D5EE07804ED436A24444CD92FA
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Compression: 1 (zlib compression)
    Start Time: 1389565902
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

An example output from the (3) showing that the server supports the legacy and depreciated ciphers.
 
# ssldump -A -n -i lo port 443
New TCP connection #1: 127.0.0.1(50211) <-> 127.0.0.1(443)
1 1  0.0007 (0.0007)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 18 25 a9 86 1c 58 ff f0 90 ca fe ba f8 eb
          c8 23 46 fd 5b 7a 4a aa 51 c2 37 40 6a 8b dc 01
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0010 (0.0003)  S>CV3.2(58)  Handshake
      ServerHello
        Version 3.2
        random[32]=
          52 d3 18 25 95 9c 3e 34 80 d8 00 3d fe 02 8f bf
          3c 1a 72 5d d1 4f 30 8c 6c 3b fa 64 0e 82 1c 6c
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_DES_CBC_SHA
        compressionMethod                 unknown value
1 3  0.0021 (0.0011)  S>CV3.2(527)  Handshake
      Certificate
        certificate[517]=
          30 82 02 01 30 82 01 6a 02 09 00 b1 90 5b 65 73
          a0 5d d1 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05
          05 00 30 45 31 0b 30 09 06 03 55 04 06 13 02 41
          55 31 13 30 11 06 03 55 04 08 0c 0a 53 6f 6d 65
          2d 53 74 61 74 65 31 21 30 1f 06 03 55 04 0a 0c
          18 49 6e 74 65 72 6e 65 74 20 57 69 64 67 69 74
          73 20 50 74 79 20 4c 74 64 30 1e 17 0d 31 34 30
          31 31 32 32 31 33 33 30 37 5a 17 0d 31 35 30 31
          31 32 32 31 33 33 30 37 5a 30 45 31 0b 30 09 06
          03 55 04 06 13 02 41 55 31 13 30 11 06 03 55 04
          08 0c 0a 53 6f 6d 65 2d 53 74 61 74 65 31 21 30
          1f 06 03 55 04 0a 0c 18 49 6e 74 65 72 6e 65 74
          20 57 69 64 67 69 74 73 20 50 74 79 20 4c 74 64
          30 81 9f 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01
          05 00 03 81 8d 00 30 81 89 02 81 81 00 b5 5b 43
          05 fa e6 1d 4f d3 f6 6e 31 be f9 4f ef e2 78 cc
          9d 81 60 56 3c 41 f4 af 43 5b 72 43 0f 81 c2 f3
          f2 87 dc f2 ce 07 74 67 0e 22 39 ae 93 86 54 37
          19 3a fb 7b 71 4c 42 8a 0c 5a 92 b4 5d 74 d8 29
          b6 d6 1a ac 95 7e 79 1c c2 65 a3 6a 92 71 3a ff
          0e 69 79 43 4b 94 c8 73 b3 2e ae c1 57 a7 5e 9e
          a0 e8 ef a7 2b 9d 75 7d a0 8a 54 0b 88 ee a7 d5
          be af c8 ab 05 7c f5 b4 7a 83 cf 99 ff 02 03 01
          00 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05
          00 03 81 81 00 46 77 cc 43 e9 92 0d c7 2f 15 7f
          58 c5 ad 7f b8 13 23 c4 2f 7f 83 90 05 38 94 89
          dd f3 79 08 b8 37 7c 8b 67 f3 89 ae 18 39 f2 a1
          24 1d ff 30 bf 24 2c 62 cc 87 a4 74 5a 91 db 6a
          81 90 bc d3 39 aa 7e c6 74 52 fe 0c 30 6f 5e a9
          72 66 d2 2b 7c a6 a6 c7 eb 39 1b 15 22 3f 44 95
          e6 68 8e f4 e8 75 9a 4c 61 5d 4a 9a 54 76 89 47
          40 de 4d 81 00 b8 0d 1d 73 59 4c a0 99 4f a1 23
          1e 76 58 09 06
1 4  0.0021 (0.0000)  S>CV3.2(4)  Handshake
      ServerHelloDone
1 5  0.0085 (0.0063)  C>SV3.2(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          71 83 c8 f4 af ab be 5e a6 e0 ec 06 ab 14 be e3
          41 25 5f f9 9e b3 29 a1 a5 1a a9 25 8d c8 1e 3d
          f2 06 3b 50 68 58 ca 1b bf 9b 1a e5 3f 4d c7 f5
          43 67 93 a1 fc f8 16 9e 35 24 7f a6 4c ad 9b 0f
          c4 db 6e a8 3d 97 5e 5f 96 0f 40 7b a3 42 62 e4
          7c 07 f9 65 97 a4 52 1a 30 cc 11 d6 43 06 7d 85
          4b e9 d5 1e 2e af 9a bd 90 cd 4d 6e aa 9e 00 29
          07 12 cd 96 bd 59 ca 5c dc a3 88 00 53 6e 8f ec
1 6  0.0085 (0.0000)  C>SV3.2(1)  ChangeCipherSpec
1 7  0.0085 (0.0000)  C>SV3.2(56)  Handshake
1 8  0.0099 (0.0014)  S>CV3.2(170)  Handshake
      TLS_RSA_WITH_RC4_128_MD51 9  0.0476 (0.0376)  S>CV3.2(1)  ChangeCipherSpec
1 10 0.0476 (0.0000)  S>CV3.2(56)  Handshake
1    0.7913 (0.7436)  C>S  TCP FIN
1    0.7917 (0.0004)  S>C  TCP FIN

Output proving the ciphers are not supported.
 
# ssldump -A -n -i eth0 port 443 and host 31.222.129.61
New TCP connection #1: 162.13.0.27(34228) <-> 31.222.129.61(443)
1 1  0.0017 (0.0017)  C>SV3.1(59)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 19 53 c5 78 4c 06 8c e7 fc 47 a1 92 ec a4
          90 63 ca a2 6e a5 7e 58 bb 72 9b a1 be c1 84 3a
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.0021 (0.0003)  S>CV3.1(2)  Alert
    level           fatal
    value           handshake_failure
1    0.0021 (0.0000)  S>C  TCP FIN
1    0.0044 (0.0022)  C>S  TCP FIN

 
# openssl s_client -connect 31.222.129.61:443 -state -msg -cipher DES-CBC-SHA
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> TLS 1.1  [length 003b]
    01 00 00 37 03 02 52 d3 19 53 c5 78 4c 06 8c e7
    fc 47 a1 92 ec a4 90 63 ca a2 6e a5 7e 58 bb 72
    9b a1 be c1 84 3a 00 00 04 00 09 00 ff 02 01 00
    00 09 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state
SSL3 alert read:fatal:handshake failure
<<< TLS 1.0 Alert [length 0002], fatal handshake_failure
    02 28
SSL_connect:error in unknown state
139646822749888:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:741:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 64 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

References

DES and IDEA Cipher Suites for Transport Layer Security (TLS)
http://www.ietf.org/rfc/rfc5469.txt

The TLS Protocol, Version 1.0
http://www.ietf.org/rfc/rfc2246.txt

The Transport Layer Security (TLS) Protocol, Version 1.2
http://tools.ietf.org/html/rfc5246.txt

Sunday, November 10, 2013

How to use wildcard certificate with alternative name extensions or server name indication (SNI) certificates

There are number of ways how you can incorporate security into your web site. One and the most common method is to use SSL/TLS protocol to create and maintain a secure channel between the client and server.

Normally, by default for every site (for example for every home page URLlike ww.example.com) that you want to protect you need to set up separate SSL/TLS configuration. The most important part of the configuration is the private key and certificate. In standard SSL deployments this leads to a situation that for every new site you have a new public IP that is tight through DNS to URL name that is used as a CN(common name) in the new certificate.

The security of the TLS/SSL protocol heavily depends on the method how the client verifies and confirms and  the identity of your site. The most common and the most important part of the client check is to evaluate and compare the site URL with the CN value embedded in the certificate.

From high level point of view to grantee your sites security you need to protect and mange all your private keys and certificates on all devices like web servers, load balancers etc.

Problem

How to use a single certificate to protect multiple different sites (domains).
How to use a single public IP to host multiple SSL sites.

Solution 1: wildcard plus alternative names
  • Wildcard 
To use a single certificate for multiple sites we can use wildcard certificate. This certificate can be used for all domains with a shared name, like for example *.rado.com. There is a limitation that the wildcard can only be used to mask one single domain level name. That means:

subdomain1.rado.com - ok
sub2.subdomain1.rado.com - bad
  • Alternative names
A certificate can be used to protect 2 and more different domains. For example: ww.rado.com and www.radoninja.com. All what you need is to provide one or more alternative names when registering and buying a certificate.
  • Combine alternative names and wildcard in a single certificate
You can combine these to options. You can have a wildcard certificate with multiple alternative names using wildcard domains, example:

*.rado.com - CN
*.subdomain1.rado.com - alternative name to overcome the wildcard limitation
*.radoninja.com - alternative name for 2th domain
*.subdomain.radoninja.com - another alternative name, etc...

Solution 2

Alternatively to use a single certificate with multiple domains uou can use the newer TLS extension called SNI.

The disadvantage is that SNI is relatively new. There are some older web clients, for example Win XP or some mobile browsers that don't support it yet. That means that  your site may not be available for these clients if you supports only SNI.

Example

http://www.ssltools.com/certificate_lookup/www.wikipedia.org

SSL Certificate

Common Name : *.wikipedia.org 
Subject Alternative Names : *.wikipedia.org, wikipedia.org, m.wikipedia.org, *.m.wikipedia.org, wikibooks.org, m.wikibooks.org, *.wikibooks.org, *.m.wikibooks.org, wikidata.org, m.wikidata.org, *.wikidata.org, *.m.wikidata.org, wikimedia.org, m.wikimedia.org, *.wikimedia.org, *.m.wikimedia.org, wikimediafoundation.org, m.wikimediafoundation.org, *.wikimediafoundation.org, *.m.wikimediafoundation.org, wikinews.org, m.wikinews.org, *.wikinews.org, *.m.wikinews.org, wikiquote.org, m.wikiquote.org, *.wikiquote.org, *.m.wikiquote.org, wikisource.org, m.wikisource.org, *.wikisource.org, *.m.wikisource.org, wikiversity.org, m.wikiversity.org, *.wikiversity.org, *.m.wikiversity.org, wikivoyage.org, m.wikivoyage.org, *.wikivoyage.org, *.m.wikivoyage.org, wiktionary.org, m.wiktionary.org, *.wiktionary.org, *.m.wiktionary.org, mediawiki.org, *.mediawiki.org, m.mediawiki.org, *.m.mediawiki.org 
Issuer Name : DigiCert High Assurance CA-3 
Serial Number : 07:24:ee:a9:7c:55:f2:57:5e:28:8b:a4:cc:f2:0e:8e 
SHA1 Thumbprint : DA:AA:A4:9B:AD:0C:1F:A3:29:71:D8:CC:62:BA:72:D1:A4:DB:94:9F 
Key Length : 2048 bit 
Signature Algorithm : sha1WithRSAEncryption 
Secure Renegotiation: Supported


References

http://en.wikipedia.org/wiki/Server_Name_Indication
http://en.wikipedia.org/wiki/Subject_Alternative_Name
http://stackoverflow.com/questions/2115611/wildcard-ssl-on-sub-subdomain

http://en.wikipedia.org/wiki/Server_Name_Indication
http://www.delantek.com/san.html
https://devcentral.f5.com/articles/multiple-certs-one-vip-tls-server-name-indication-via-irules#.Un832_lpmYI

3.1.  Server Identity
http://www.ietf.org/rfc/rfc2818.txt

http://www.networksorcery.com/enp/protocol/tls.htm


Wednesday, November 6, 2013

How to recognise encrypted private SSL key

Problem

How to recognise an encrypted private key.

Solution

You can recognise and see if a key is encrypted by looking at the first lines of your private SSL key file. Below is an example:
  • Key is encrypted  
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,123456ABCD719580

diYueU...
-----END RSA PRIVATE KEY-----
  • Key is not encrypted 
-----BEGIN RSA PRIVATE KEY-----
MIIEogI...
-----END RSA PRIVATE KEY-----


Wednesday, May 29, 2013

How to extract a single SSL connection from tcpdump

There is not better tool for SSL troubleshooting than ssldump (a very useful how to use in a form of F5 solution can be found here: SOL10209: Overview of packet tracing with the ssldump utility.

The ssldump tool is not perfect although. It can produce only text output. The output is a mixture of SSL handshaking requests and data connections.

This little tool https://github.com/akozadaev/ssld-extract can help to extract a single SSL session. An example usage is provided below.
root@server:~/ssld-extract/# ssldump -n -r example1.pcap  > example1.pcap.txt
root@server:~/ssld-extract/pp# python ssld-extract.py -c -n1 ~/ssld-extract/example1.pcap.txt
New TCP connection #1: 192.168.0.2(57122) <-> 72.26.232.202(443)
1 1  0.1946 (0.1946)  C>S  Handshake
      ClientHello
        Version 3.1
        resume [32]=
          7b 9a 08 2f 3f c0 5e 70 c8 9e b6 f8 61 a0 4e 9e
          d9 84 07 e5 94 13 f8 e8 87 33 96 0d f4 a4 9f 6a
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc014
        Unknown value 0x88
        Unknown value 0x87
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        Unknown value 0xc012
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
...
        compression methods
                  NULL
1 2  0.3973 (0.2027)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[32]=
          d4 65 5e b6 3d 33 88 8c bd 7e 56 65 13 71 9f 52
          30 47 ea e1 c0 d6 1f 72 12 b9 2f 8f 6b 42 b2 68
        cipherSuite         TLS_RSA_WITH_RC4_128_SHA
        compressionMethod                   NULL
1 3  0.3974 (0.0001)  S>C  Handshake
      Certificate
1 4  0.3974 (0.0000)  S>C  Handshake
      ServerHelloDone
1 5  0.4006 (0.0031)  C>S  Handshake
      ClientKeyExchange
1 6  0.4006 (0.0000)  C>S  ChangeCipherSpec
1 7  0.4006 (0.0000)  C>S  Handshake
1 8  0.5794 (0.1788)  S>C  ChangeCipherSpec
1 9  0.5794 (0.0000)  S>C  Handshake
1 10 0.5814 (0.0019)  C>S  application_data
1 11 0.5819 (0.0004)  C>S  application_data
1 12 0.7806 (0.1987)  S>C  application_data
As you can see it was able to extract the single connection what is a huge help if you need to analyze a big tcpdump file.

Tuesday, May 21, 2013

Openssl cheat sheet

  • How to extract certificates
Usual certificate files have not only the certificate itself but include the chain as well.

# example chain cert file

-----BEGIN CERTIFICATE-----
687f687asfafaufyaufasfyfyasifyayfafvG74WlTANBgkqhkiG9w0BAQUFADBm
678fa6auyasfyasf8a7fa9f7a9sfiauy987safasffQgSW5jMRkwFwYDVQQLExB3
....
UW9iatnbVzOcOdJJaBK7obGALVFBAQ==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
76af5at7fat8f7astfuyasftuftauftaufasfasfOzANBgkqhkiG9w0BAQUFADBs
....
tfiuytaiutauisftiaustfiuasftuiastfasf2oWGU4K8K2Eyl2Us1p292E=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
tfuastfuiatfutfuiatfiuatsifuastifuayrkYldzANBgkqhkiG9w0BAQUFADBs
....
tuaftiautfuaftiuatfiuastfuastfiuastfiastfuatsifutafutafutasuftap
+OkuE6N36B9K
-----END CERTIFICATE-----

To extract each certificate and save it in a separate file you can use this little tick.
 
root@server:~# csplit -k cert.txt '%-----BEGIN CERTIFICATE-----%' '/-----END CERTIFICATE-----/+1' {9}
2362
2260
1367
csplit: `/-----END CERTIFICATE-----/+1': match not found on repetition 3
1

root@server:~# ll 
-rw-r--r-- 1 root root 2362 May 21 16:50 cert.txt
-rw-r--r-- 1 root root 2362 May 21 16:50 xx00
-rw-r--r-- 1 root root 2260 May 21 16:50 xx01
-rw-r--r-- 1 root root 1367 May 21 16:50 xx02
-rw-r--r-- 1 root root    1 May 21 16:50 xx03

# because it is irrelevant
root@server:~# rm xx03 
  • How to verify that the certificate and key belong together
$ openssl x509 -noout -modulus -in server.crt | openssl md5
$ openssl rsa -noout -modulus -in server.key | openssl md5
  • How to verify what certificate and what certificate chain does a https server sends
$ openssl s_client -connect :443 -showcerts

Without the -showcerts option the openssl shows only a site certificate (a top certificate in the chain), hiding the remaining certs received in server hello handshaking message. Please be aware that in the regular output you can still see there were intermediate certs although:.

Certificate chain
 0 s:/1.3.6.1.4.1.311.60.2.1.3=GB/2.5.4.15=Private Organization/serialNumber=04168207/C=GB/ST=Greater Manchester/L=Manchester/O=Party Delights Limited/OU=Web Development/CN=www.example.com
   i:/C=US/O=thawte, Inc./OU=Terms of use at https://www.thawte.com/cps (c)06/CN=thawte Extended Validation SSL CA

 1 s:/C=US/O=thawte, Inc./OU=Terms of use at https://www.thawte.com/cps (c)06/CN=thawte Extended Validation SSL CA
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

 2 s:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA
   i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

References
  1. https://kb.wisc.edu/middleware/page.php?id=4064