Dirty trick how to analysis ASA performance based on interface overruns and underruns

There are number of firewall vendors on the market you can chose from (other links to Gartner magic quadrant for firewalls here and here). Every vendor has a product line ranging from the low to high end firewalls. An example product list for Cisco ASA can be seen here: http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html#~tab-b

Problem

I can see in my firewalls interface stats underruns and overrruns and the counters increase.

Solution

This is rather a dirty trick and your monitoring system should be able to graph the interface stats. But if you are in a position like me where you have no visibility to interface statistics like you could have in Zenoss, Cacti, Zabbix or other monitoring system you may need to manually check this...

• We need to first start collecting data so we can look at it later.

Run at least one a day the command and save in a file 1.txt, 2.txt, etc.

sh clock
sh int

• After some time you should have a collection of files 

$ ls -1 *.txt
1.txt
2.txt
3.txt
4.1.txt
4.2.txt
5.1.txt
6.1.txt
8.txt

An example outpout form the file can be seen here: https://github.com/rtomaszewski/experiments/blob/master/asa-interfaces-example.txt

• Download the script asa-interfaces.sh and run it 

bash asa-interfaces.sh

Base on the files you collected it will generate stats for every interface (time stamp is in the last column). Example output:

Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        734 input errors, 0 CRC, 0 frame, 734 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        1502 input errors, 0 CRC, 0 frame, 1502 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        1794 input errors, 0 CRC, 0 frame, 1794 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        1881 input errors, 0 CRC, 0 frame, 1881 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1921 input errors, 0 CRC, 0 frame, 1921 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1971 input errors, 0 CRC, 0 frame, 1971 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        364 input errors, 0 CRC, 0 frame, 364 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        382 input errors, 0 CRC, 0 frame, 382 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        392 input errors, 0 CRC, 0 frame, 392 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        468 input errors, 0 CRC, 0 frame, 468 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        707 input errors, 0 CRC, 0 frame, 707 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        756 input errors, 0 CRC, 0 frame, 756 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        640 input errors, 0 CRC, 0 frame, 640 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        658 input errors, 0 CRC, 0 frame, 658 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        683 input errors, 0 CRC, 0 frame, 683 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        797 input errors, 0 CRC, 0 frame, 797 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        811 input errors, 0 CRC, 0 frame, 811 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        863 input errors, 0 CRC, 0 frame, 863 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        984 input errors, 0 CRC, 0 frame, 984 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1052 input errors, 0 CRC, 0 frame, 1052 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014



Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        646983182 packets output, 473597063148 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        700155558 packets output, 509814505730 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        753341661 packets output, 546026853810 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        1025937535 packets output, 734304301602 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        1054530605 packets output, 761409276094 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        1105491616 packets output, 798565630885 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        1264871240 packets output, 907739984962 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        1315876113 packets output, 943680519398 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        985243431 packets output, 309823858329 bytes, 459 underruns 07:48:45.631 cst Wed Feb 12 2014
        1070533450 packets output, 336205856058 bytes, 459 underruns 05:38:08.573 cst Thu Feb 13 2014
        1159894277 packets output, 366047579951 bytes, 483 underruns 03:37:26.853 cst Fri Feb 14 2014
        1596471490 packets output, 500836893219 bytes, 483 underruns 03:52:52.523 cst Wed Feb 19 2014
        1635530484 packets output, 511489408071 bytes, 483 underruns 10:03:08.799 cst Wed Feb 19 2014
        1722164227 packets output, 536769375853 bytes, 483 underruns 06:11:22.244 cst Thu Feb 20 2014
        2032554621 packets output, 636075162304 bytes, 2831 underruns 08:35:21.315 cst Sun Feb 23 2014
        2174454722 packets output, 688313076839 bytes, 2831 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        1968362 packets output, 524301440 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1987058 packets output, 526612914 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2005883 packets output, 528940672 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        4036852 packets output, 3167775775 bytes, 2831 underruns 03:52:52.523 cst Wed Feb 19 2014
        13676338 packets output, 15853359823 bytes, 2831 underruns 10:03:08.799 cst Wed Feb 19 2014
        23861856 packets output, 16649050850 bytes, 3052 underruns 06:11:22.244 cst Thu Feb 20 2014
        66743830 packets output, 20187731129 bytes, 5941 underruns 08:35:21.315 cst Sun Feb 23 2014
        80290600 packets output, 21286340673 bytes, 6860 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        16582048 packets output, 17699836232 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        17971640 packets output, 19234649922 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        19380417 packets output, 20791969660 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        26970739 packets output, 29162172960 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        27259841 packets output, 29471830004 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        29612954 packets output, 32141440890 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        39077736 packets output, 42912691094 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        42074827 packets output, 46322035220 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        1863787 packets output, 265441230 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1977505 packets output, 281732398 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2091970 packets output, 298145244 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        2718733 packets output, 387975068 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        2750523 packets output, 392530668 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        2855417 packets output, 407567752 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        3242848 packets output, 463113612 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        3366749 packets output, 480878922 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        229534738 packets output, 55157234973 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        249682890 packets output, 59948227086 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        272763726 packets output, 66350185657 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        378020447 packets output, 91307448807 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        384704374 packets output, 93165635304 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        402556578 packets output, 97469565455 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        492798902 packets output, 119550853698 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        564346002 packets output, 137603523999 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        142287604 packets output, 56966204294 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        154809474 packets output, 62049309926 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        167733332 packets output, 67152657884 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        231962627 packets output, 93689642614 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        235640974 packets output, 95384548398 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        249769631 packets output, 103735197461 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        290301462 packets output, 119748550482 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        303003248 packets output, 125098088305 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 packets output, 0 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        0 packets output, 0 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        0 packets output, 0 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        0 packets output, 0 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        0 packets output, 0 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        0 packets output, 0 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014

Once you have the data in-front of you can easily see how the stats were changing over time, over longer period of time like a week.

In my case we suspected that the Firewall hit the capacity limit but further investigation confirmed that the device is doing well and no upgrade is necessary.

References

http://www.gossamer-threads.com/lists/cisco/nsp/152428
http://ccna2ccnp.blogspot.co.uk/2012/12/ciscoasa-oversubcription-maximizing.html
http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1904.html
http://en.wikipedia.org/wiki/Buffer_underrun
http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.html

On Cisco ASA firewall how to find the real Interface MAC address

Normally the output from 'sh interface' shows interfaces MAC addresses. This is truth when you have a single ASA.

Problem

How to find a real interface MAC address on HA ASA cluster node.

Resolution

There are no floating IPs in ASA cluster design. Instead there active IP will be moved between the ASA nodes when a failover occurs. For guys who work on different cluster implementation it may be very confusing.

When a firewall is part of an HA active/standby cluster the physical interface MAC address (showed in the output form sh interface) and the IP assigned to it has always a value of the primary unit.

When a failover happens both ASAs swaps IP and MAC during.

For example, if we have assigned an IP 1.1.1.1 to the primary unit on our ASA cluster this IP will be once held by the unit A once by the unit B. That means, when you try to connect to this IP you never know to what physical ASA unit you are actually connecting.

To find out the real MAC of an interface you need to look at the sh version output.

fw-1092388-553262/pri/stby# sh ver | i Gig
 0: Ext: GigabitEthernet0/0  : address is 1111.aaaa.deea, irq 9   ------  real address on the ASA 
 1: Ext: GigabitEthernet0/1  : address is 1111.aaaa.deeb, irq 9
 2: Ext: GigabitEthernet0/2  : address is 1111.aaaa.deec, irq 9
 3: Ext: GigabitEthernet0/3  : address is 1111.aaaa.deed, irq 9
 
fw-1092388-553262/pri/stby# sh int | i MAC|Int
Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        MAC address 2222.fd52.ac28, MTU 1500                     ------ active MAC that migrates every time a failover happens

Best practice

If you want to know to what physical ASA unit you connected take a look at the output from sh version. Never relay on the values in sh int output.

References

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/ef.html#wp1929064
http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/ha_overview.html

During TLS handshaking server can ignore a cipher from the preference list in ClientHello message

I've noticed recently that when my client tries to negotiated a TLS session its ciphers preference list is being ignored by the server.

• Test 1

$ openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA

• Test 2

$ openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA
# filtered out  ....
New, TLSv1/SSLv3, Cipher is DES-CBC-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DES-CBC-SHA

• Test 3

# openssl s_client -connect server_ip:443 -state -msg -cipher DES-CBC-SHA:AES128-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: C7E7F65B4E927AFDE568E52FDAE52495CE815DAC2958B82C854146CD383FC00BB47ED691840C713CE762B6430FCB3230
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568347
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

• Test 4

# openssl s_client -connect server_ip:443 -state -msg -cipher AES128-SHA:DES-CBC-SHA
# skiped ...
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID:
    Session-ID-ctx:
    Master-Key: 1F0FB31410AAFFC234DC5598FFA2A676C63119A6367453D05578F9B1652DD085DB1D5C016694B9D293F3A2DFF769EF51
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1389568389
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

We can see in (1) and (2) that the TLS servers supports 2 different ciphers suits.
In (3) and (4) we see that even though the preference list was directly specified by the client the TLS server ignored the settings in (3).

Example ssldump output from (3).

 
# ssldump -A -n -i eth0 port 443 and host my_server_ip
New TCP connection #1: 162.13.0.27(51869) <-> my_server_ip(443)
1 1  0.1013 (0.1013)  C>SV3.1(61)  Handshake
      ClientHello
        Version 3.2
        random[32]=
          52 d3 23 b5 62 f5 08 38 14 53 d9 ac a4 51 c2 28
          0d e2 0e b6 41 89 2d af a1 50 55 bd a9 35 05 75
        cipher suites
        TLS_RSA_WITH_DES_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        Unknown value 0xff
        compression methods
                unknown value
                  NULL
1 2  0.1984 (0.0970)  S>CV3.1(49)  Handshake
      ServerHello
        Version 3.1
        random[32]=
          2b ae 63 1e 9c da f3 38 b6 f9 dd c6 c0 4f ad 6c
          2c f0 f3 52 67 6e 28 29 8a 24 95 95 a5 67 f2 6f
        session_id[0]=

        cipherSuite         TLS_RSA_WITH_AES_128_CBC_SHA
        compressionMethod                   NULL
1 3  0.1984 (0.0000)  S>CV3.1(674)  Handshake
      Certificate
        certificate[664]=
          50 82 02 94 50 82 01 fd 02 02 03 79 50 0d 06 09
          2a 86 48 86 f7 0d 01 01 05 05 00 50 81 b6 31 0b
          50 11 06 03 55 04 06 13 02 55 53 31 0e 50 0c 06
          03 55 04 08 13 05 54 65 78 61 73 31 14 50 12 06
          03 55 04 07 13 0b 53 61 6e 20 41 6e 74 6f 6e 69
          6f 31 12 50 10 06 03 55 04 0a 13 11 52 61 63 6b
          73 70 61 63 65 31 1e 50 1c 06 03 55 04 0b 13 15
          53 79 73 74 65 6d 20 41 64 6d 69 6e 69 73 74 72
          61 74 69 6f 6e 31 23 50 21 06 03 55 04 03 13 1a
...
          b3 79 0a 37 cd 27 93 af
1 4  0.1984 (0.0000)  S>CV3.1(4)  Handshake
      ServerHelloDone
1 5  0.2126 (0.0141)  C>SV3.1(134)  Handshake
      ClientKeyExchange
        EncryptedPreMasterSecret[128]=
          33 25 53 17 c2 45 b0 32 8a ca c1 66 39 6e f0 31
          98 96 4c 34 f2 e4 fd b7 0c e6 15 af c7 d3 fc e4
          0d 15 c9 c9 d4 e0 78 5a a1 13 dc 55 8b 5a bc 69
          68 24 f5 d1 50 6a 19 2e 71 9a 66 ee 3a 64 bc 1e
          d8 9a da d4 e0 44 96 b3 43 20 f1 a0 b6 4c 49 8e
          b2 ae 2b a6 12 68 78 19 eb 61 06 1c 34 8a 03 22
          ab 7e ff f8 88 44 89 97 cd 53 06 b0 b6 66 7b 77
          2a c0 0a 15 a3 54 2d 8c 5b 74 bc fe 31 3f 1f 8f
1 6  0.2126 (0.0000)  C>SV3.1(1)  ChangeCipherSpec
1 7  0.2126 (0.0000)  C>SV3.1(48)  Handshake
1 8  0.3100 (0.0974)  S>CV3.1(1)  ChangeCipherSpec
1 9  0.3100 (0.0000)  S>CV3.1(48)  Handshake
1    1.4782 (1.1681)  C>S  TCP FIN
1    1.5740 (0.0958)  S>C  TCP FIN

Problem

Can TLS server ignore the cipher suite list passed from the client during TLS handshaking?

Analysis

The specification how the ClientHello message can be handled by the server during TLS handshaking has changed from TLS v1.0 to v1.2. The extract from the RFC comparing both relevant sections.

• rfc2246

7.4.1.2. Client hello

   The CipherSuite list, passed from the client to the server in the
   client hello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first). Each CipherSuite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length) and a MAC algorithm.

   The server will select a cipher suite or, if no acceptable choices 
   are presented, return a handshake failure alert and close the 
   connection.

• rfc5246

7.4.1.2.  Client Hello

   The cipher suite list, passed from the client to the server in the
   ClientHello message, contains the combinations of cryptographic
   algorithms supported by the client in order of the client's
   preference (favorite choice first).  Each cipher suite defines a key
   exchange algorithm, a bulk encryption algorithm (including secret key
   length), a MAC algorithm, and a PRF.  The server will select a cipher
   suite or, if no acceptable choices are presented, return a handshake
   failure alert and close the connection.

   If the list contains cipher suites the server does not recognize, 
   support, or wish to use, the server MUST ignore those cipher suites,
   and process the remaining ones as usual.

In the yellow parts we can see that a server implementing TLS v1.2 has the ability to ignore ciphers specified by the client in its list ClientHello message. The criteria are very wide. The TLS server can ignore a cipher only because it doesn't wish to use it.

References

http://www.openssl.org/docs/apps/ciphers.html
DES and IDEA ciphers are deprecated in the latest TLS protocol
http://www.ietf.org/rfc/rfc2246.txt - The TLS Protocol, Version 1.0
http://tools.ietf.org/html/rfc5246.txt - The Transport Layer Security (TLS) Protocol, Version 1.2
http://support.f5.com/kb/en-us/solutions/public/10000/200/sol10209.html - ssldump

A definitely interesting reading about ciphers in the web browser to show some parasitical view at the cipher list topic.
https://groups.google.com/forum/#!topic/mozilla.dev.tech.crypto/36na1B2brGU

ASA performance troubleshooting tips

This is more a work in progress. Below are couple of tips and ideas how to deal with high traffic performance issues.

Limit connection per IP

Often a load can be generated from unique single (or a group of IPs). To limit the number of connection.

access-list http_conn_limit extended permit tcp any any eq www 
! access-list http_conn_limit extended permit tcp any any eq https
! you can add any other ACL to catch the intresting traffic 

class-map http_conn_limit_class
 match access-list http_conn_limit

policy-map http_conn_limit_map
 class http_conn_limit_class
  set connection per-client-max 100 

service-policy global_policy global
service-policy http_conn_limit_map interface outside

Reference:
http://rtomaszewski.blogspot.co.uk/2013/12/cisco-asa-connection-table-state.html
http://www.itlibrary.net/index.php/cisco-asa/8-limiting-connections-rate-for-traffic-destined-on-port-80
http://www.cisco.com/en/US/docs/security/asa/asa72/configuration/guide/mpc.html
http://blog.ine.com/2009/04/19/understanding-modular-policy-framework/

Kick off a client sessions

If you identify a client that you want to deny traffic and close all its connections.

access-list 101 extended deny ip host [ip] any
shun [ip]
no shun [ip]

Cisco ASA connection table state description and examples

On ASA in the connection table you can find protocol sessions (TCP, UDP, ICMP and others) that describe the state of the session (like TCP/IP) when the command was run.

In the session you can find all currently managed sessions by the ASA. From this output you can understand as well as from what IPs your clients are coming from and to what services they connect.

Session statutes

 
fw-asa# sh conn  

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,
       B - initial SYN from outside, C - CTIQBE media, D - DNS, d - dump,
       E - outside back connection, F - outside FIN, f - inside FIN,
       G - group, g - MGCP, H - H.323, h - H.225.0, I - inbound data,
       i - incomplete, J - GTP, j - GTP data, K - GTP t3-response
       k - Skinny media, M - SMTP data, m - SIP media, n - GUP
       O - outbound data, P - inside back connection, p - Phone-proxy TFTP connection,
       q - SQL*Net data, R - outside acknowledged FIN,
       R - UDP SUNRPC, r - inside acknowledged FIN, S - awaiting inside SYN,
       s - awaiting outside SYN, T - SIP, t - SIP transient, U - up,
       V - VPN orphan, W - WAAS,
       X - inspected by service module

Example flags meaning from the session entities

 
UB
U - up,
B - initial SYN from outside,

UO
U - up,
O - outbound data,

UIB
U - up,
I - inbound data,
B - initial SYN from outside,

UIOB
U - up,
I - inbound data,
O - outbound data,
B - initial SYN from outside,

UfIB
U - up,
f - inside FIN,
I - inbound data,
B - initial SYN from outside,

UfrO
U - up,
f - inside FIN,
r - inside acknowledged FIN,
O - outbound data,

UfIOB 
U - up,
f - inside FIN,
I - inbound data,
O - outbound data,
B - initial SYN from outside,

UfFIOB
 the same like UfIOB 
 F - outside FIN,

UfFRIOB
the same like UfFIOB
R - UDP SUNRPC,

UfrIOB
U - up,
f - inside FIN,
r - inside acknowledged FIN
I - inbound data,
O - outbound data,
B - initial SYN from outside,

SaAB
S - awaiting inside SYN,
a - awaiting outside ACK to SYN,
A - awaiting inside ACK to SYN, 
B - initial SYN from outside,

aB
a - awaiting outside ACK to SYN,
B - initial SYN from outside,

Example flow you can find in the ASA firewall connection table

Usually a lot entries with these state.

 
fw-asa# sh conn detail long

flags UfIOB TCP outside:1.165.177.125/1965 (1.165.177.125/1965) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 52m38s, uptime 54m21s, timeout 1h0m, bytes 3063
flags UfIOB TCP outside:1.172.130.64/1485 (1.172.130.64/1485) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 41m38s, uptime 43m12s, timeout 1h0m, bytes 3063

flags UB TCP outside:1.189.22.195/16208 (1.189.22.195/16208) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UB, idle 45m6s, uptime 48m17s, timeout 1h0m, bytes 0
flags UB TCP outside:1.56.45.22/24654 (1.56.45.22/24654) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UB, idle 45m54s, uptime 49m4s, timeout 1h0m, bytes 0

Common but less frequent state

 
flags UfFIOB TCP outside:1.55.216.14/14104 (1.55.216.14/14104) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFIOB, idle 41m51s, uptime 43m24s, timeout 1h0m, bytes 3002
flags UfFIOB TCP outside:110.81.84.50/20230 (110.81.84.50/20230) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFIOB, idle 52m55s, uptime 54m28s, timeout 1h0m, bytes 3063

flags UfFRIOB TCP outside:109.109.38.148/4760 (109.109.38.148/4760) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFRIOB, idle 3s, uptime 15s, timeout 5m0s, bytes 2261
flags UfFRIOB TCP outside:112.12.221.155/3753 (112.12.221.155/3753) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfFRIOB, idle 0s, uptime 0s, timeout 5m0s, bytes 1008

flags UfIB TCP outside:121.35.47.128/1481 (121.35.47.128/1481) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIB, idle 23m54s, uptime 26m28s, timeout 1h0m, bytes 1106
flags UfIB TCP outside:183.11.2.56/4589 (183.11.2.56/4589) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIB, idle 47m15s, uptime 49m48s, timeout 1h0m, bytes 1106

flags SaAB TCP outside:112.72.135.224/7494 (112.72.135.224/7494) inside:192.168.55.172/4567 (1.2.157.172/4567), flags SaAB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
flags SaAB TCP outside:113.170.107.218/4472 (113.170.107.218/4472) inside:192.168.55.172/4567 (1.2.157.172/4567), flags SaAB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

flags UfrO TCP outside:202.168.215.226/80 (202.168.215.226/80) inside:192.168.55.172/3845 (1.2.157.172/3845), flags UfrO, idle 6s, uptime 8s, timeout 10m0s, bytes 1182

flags UIOB TCP outside:61.187.244.179/9571 (61.187.244.179/9571) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 38m13s, uptime 39m46s, timeout 1h0m, bytes 2897
flags UIOB TCP outside:67.47.251.34/14921 (67.47.251.34/14921) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 48m14s, uptime 49m50s, timeout 1h0m, bytes 3348

TCP outside:1.2.27.69/49856 (1.2.27.69/49856) FW-INSIDE:192.168.100.112/80 (11.22.192.112/80), flags UIB, idle 0s, uptime 0s, timeout 1h0m, bytes 581

flags UO TCP outside:202.168.215.226/80 (202.168.215.226/80) inside:192.168.55.172/3848 (1.2.157.172/3848), flags UO, idle 7s, uptime 7s, timeout 1h0m, bytes 1182

TCP outside:220.135.240.219/61139 (220.135.240.219/61139) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
TCP outside:220.135.240.219/61138 (220.135.240.219/61138) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

# without the 'long' parameter
TCP outside 94.5.94.11:59458 FW-DMZ-LB 192.168.67.79:80, idle 0:04:31, bytes 19424, flags UfrIOB
TCP outside 94.5.94.11:59463 FW-DMZ-LB 192.168.67.72:80, idle 0:04:05, bytes 7181, flags UfrIOB

You can specify additional parameters to filter output for specific connection entries state.

 
fw-asa# sh conn detail  long state tcp_embryonic all

TCP outside:220.135.240.219/61139 (220.135.240.219/61139) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0
TCP outside:220.135.240.219/61138 (220.135.240.219/61138) inside:192.168.55.172/4567 (1.2.157.172/4567), flags aB, idle 0s, uptime 0s, timeout 1m0s, bytes 0

fw-asa# sh conn long state data_out

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 0s, uptime 3m48s, timeout 1h0m, bytes 72509
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 1s, uptime 6m12s, timeout 1h0m, bytes 139249
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m37s, uptime 7m14s, timeout 1h0m, bytes 3002
TCP outside:171.118.104.53/64054 (171.118.104.53/64054) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 8s, uptime 7m27s, timeout 1h0m, bytes 98878
TCP outside:219.139.32.90/4141 (219.139.32.90/4141) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 7s, uptime 7m32s, timeout 1h0m, bytes 94113

fw-asa# sh conn long state data_in

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 4s, uptime 3m37s, timeout 1h0m, bytes 44907
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 1s, uptime 6m1s, timeout 1h0m, bytes 137801

fw-asa# sh conn long state finin

TCP outside:138.91.170.208/1264 (138.91.170.208/1264) inside:192.168.55.172/80 (1.2.157.172/80), flags UfFRIOB, idle 0s, uptime 0s, timeout 5m0s, bytes 5052
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 4m45s, uptime 6m21s, timeout 1h0m, bytes 3002
TCP outside:2.176.137.197/1653 (2.176.137.197/1653) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m6s, uptime 6m43s, timeout 1h0m, bytes 3002

fw-asa# sh conn long state up

TCP outside:112.65.211.244/6680 (112.65.211.244/6680) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 0s, uptime 2m50s, timeout 1h0m, bytes 37914
TCP outside:113.247.3.129/3253 (113.247.3.129/3253) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UIOB, idle 4s, uptime 5m14s, timeout 1h0m, bytes 78789
TCP outside:2.176.137.197/1950 (2.176.137.197/1950) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 4m39s, uptime 6m16s, timeout 1h0m, bytes 3002
TCP outside:171.118.104.53/64054 (171.118.104.53/64054) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 0s, uptime 6m29s, timeout 1h0m, bytes 89118
TCP outside:219.139.32.90/4141 (219.139.32.90/4141) inside:192.168.55.172/80 (1.2.157.172/80), flags UIOB, idle 9s, uptime 6m35s, timeout 1h0m, bytes 82689
TCP outside:2.176.137.197/1653 (2.176.137.197/1653) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m1s, uptime 6m38s, timeout 1h0m, bytes 3002
TCP outside:2.176.137.197/1589 (2.176.137.197/1589) inside:192.168.55.172/4567 (1.2.157.172/4567), flags UfIOB, idle 5m13s, uptime 6m49s, timeout 1h0m, bytes 3002

Advance types of NAT

In a previous post (NAT order of operation on Cisco ASA firewall) we took a look how many NAT configuration types Cisco ASA supports and what the differences are. In  advance (firewall) configuration there can be other types of address and port translation that may obey more or less restrictive rules. Examples and a short listing of them can be found bellow (a full description can be found in reference section):

• Symmetric NAT

Standard, very restrictive. Only the original source and destination hosts can communicate together.

• Full-cone NAT

Any external host can use the NAT binding (the entry from the connection table about the NAT) and communicate with the internal server. Neither external IP or external port are checked when processing TCP/UDP packets.

• Restricted-cone NAT 

Only the single remote host can use the NAT binding. The port is irrelevant.

• Port-restricted-cone NAT

Upside down to the "Restricted-clone NAT". Every host can reuse the NAT binding as long as it is using the original destination port number that was used when the NAT binding was 

created and stored in the NAT table in the FW memory. Take a look at the "X" and "V" paths at the illustration bellow:

Despite a good theoretical explanation that you can find on Cisco I wasn't able to find a working example for ASA. Found only some spare documents for the wireless routes in the context of SIP protocol. Does ASA supports these NAT types?

References

http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_7-3/anatomy.html
https://supportforums.cisco.com/thread/2178132

ASA IPsec VPN filters explained

There is a standard ACL that we use to control the ingress and egress traffic of an interface on the ASA firewall. When it comes to IPsec VPN there are vpn-filter ACLs that can be used additionally (or instead) to control the traffic on a more granular basis.

Problem

How does a vpn-filter (VPN filter ACL/VPN filter access list ) works and how it is different from a standard ACL.

Solution description

The way the ASA is processing and applying the standard ACL is different from how vpn filter ACL (vpn-filter ACL) work.

Normally when defining the VPN filter ACL rules you will specify them in this format:

access-list <acl-no> <permit/deny> ip <remote network> <local network>

where,
 - local network are the FW local segments or segments we want a VPN client to have access to
 - remote network is the network the VPN traffic (or the VPN user traffic) is coming from

Below are some extracts from available documentation I found:

Description from various documentation links:

• PIX/ASA 7.x and Later: VPN Filter (Permit Specific Port or Protocol) Configuration Example for L2L and Remote Access

When you construct the ACLs for use with the vpn-filter feature the ACLs are constructed with the post-decrypted traffic (inbound VPN traffic) in mind. However, they are also applied to the traffic originated in the opposite direction.

• Configuring Tunnel Groups, Group Policies, and Users from Cisco ASA 5500 Series Configuration Guide using the CLI, 8.3

A vpn-filter command is applied to post-decrypted traffic after it exits a tunnel and pre-encrypted traffic before it enters a tunnel. An ACL that is used for a vpn-filter should NOT also be used for an interface access-group. When a vpn-filter command is applied to a group policy that governs Remote Access VPN client connections, the ACL should be configured with the client assigned IP addresses in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.

When a vpn-filter command is applied to a group-policy that governs a LAN to LAN VPN connection, the ACL should be configured with the remote network in the src_ip position of the ACL and the local network in the dest_ip position of the ACL.

• vpn-filter from Cisco ASA 5500 Series Command Reference, 8.2

By design, the vpn-filter feature allows for traffic to be filtered in inbound direction only. The outbound rule is automatically compiled.

• VPN-Filter v Interface ACLs on ASA 5525

What you also have to consider especially with L2L VPN Filter ACL is that their format is different from interface ACL and Client VPN Filter ACL.

In the L2L VPN Filter ACLs you ALWAYS define the source address as the "remote network". This creates every now and then confusion on how the rules should actually look like. I also potentially allows more traffic than you want as the single ACL rule is bidirectional.

• ASA unidirectional VPN tunnel

The ASA has this strange little thing called a filter list.  It is basically an acl that works in both directions for vpn traffic.  The challenge is that it is one acl that works in both directions.  So the granularity is limited a bit, but you can control things based on the tcp/udp port.

• VPN-filter question in 8.2.3 or 8.3.2

Well firstly the group-policy and hence the vpn-filters will take effect only after tunnel comes up. So you would not be able to restrict what traffic brings up the tunnel using vpn-filters.

Conclusion and things to remember 

1. The vpn-filters work on top of crypto domain; you need to first define an interesting traffic to bring tunell up in crypto domain and later it can be more filtered by the VPN filters
2. Single VPN filter ACL is stateful (once a rule allows the traffic through the return traffic will be allowed as well)
3. But for every single VPN filter ACL the ASA engine will create another implicit ACL rule; the 2th rule will permit the other peer to initiate traffic and sent it over VPN tunnel
4. You define the vpn-filter rule from the remote FW perspective; on our FW the vpn filter inspect the ingress traffic; once the decrypted packet enters FW and is encrypted the src and destination and ports are checked by our filter
5. VPN filters are not checking egress VPN traffic (how to inspect an encrypted traffic any way)
6. The FW incoming traffic on its local networks is controlled by the implicit VPN filter rules
7. In another words the vpn filter control explicitly packets AFTER they are decrypted by the FW and the implicit FW rules control the traffic BEFORE it enters the tunnel; even if you want to filter only the traffic BEFORE it enters the VPN you need to specify the explicit AFTER VPN filter instead
8. The vpn filter control the in and out VPN traffic in the same time; you can't have 2 VPN filters configured 
9. Changes to the VPN filter using DENY statements take affect immediately
10. Changes to the VPN filter using the PERMIT statements requires the tunnel to be restarted

References

1. http://www.alfredtong.com/cisco/cisco-asa-vpn-filter-tips-and-misc/
2. http://netribe.blogspot.co.uk/2009/03/these-cisco-asa-vpn-filters-could-trip.html

NAT order of operation on Cisco ASA firewall

There are many types of NAT you can configure on the ASA FW. This is a short summary with examples for ASA 8.2/8.3 software.

• Dynamic NAT

! The pool-number parameter ( 2, in this case) binds the 'global' and 'nat' commands
nat (dmz) 2 10.10.10.128 255.255.255.128
global (out) 2 172.16.16.129-172.16.16.254 netmask 255.255.255.128

• Dynamic PAT

! Configuring PAT using the global address 172.16.16.126
ASA2(config)# nat (dmz) 1 10.10.10.0 255.255.255.0
ASA2(config)# global (out) 1 172.16.16.126

• Identity NAT

! Configuring Identity NAT
ASA2(config)# nat (dmz) 0 10.10.10.128 255.255.255.192

• Static NAT

! Static translation for an individual host (/32 is the default netmask)
ASA2(config)# static (dmz,out) 172.16.16.140 10.10.10.140

• Static Policy NAT

ASA2(config)# access-list STATIC-POLICY1 extended permit ip host 10.10.10.148 host 172.16.16.200
ASA2(config)# static (dmz,out) 172.18.18.148 access-list STATIC-POLICY1

! For other destinations source address 10.10.10.148 is translated to 172.16.16.148
ASA2(config)# static (dmz,out) 172.16.16.148 10.10.10.148 netmask 255.255.255.255

• Dynamic Policy NAT

! Packets from 10.10.10.128/27 travelling to host 172.16.16.200 undergo Policy NAT
access-list DYN-POLICY1 extended permit ip 10.10.10.128 255.255.255.224 host 172.16.16.200
!
nat (dmz) 4 access-list DYN-POLICY1
global (out) 4 172.17.17.129-172.17.17.158 netmask 255.255.255.224
!

• Dynamic Policy PAT

! Policy PAT for source subnet 10.10.10.128/26 going to host 172.16.16.200
access-list DYN-POLICY2 extended permit ip 10.10.10.128 255.255.255.192 host 172.16.16.200
!
ASA2(config)# nat (dmz) 3 access-list DYN-POLICY2
ASA2(config)# global (out) 3 172.16.16.125

• NAT Exemption

! Connections between 10.10.10.128/28 and 172.16.16.0/24 are exempted from NAT
access-list NONAT extended permit ip 10.10.10.128 255.255.255.240 172.16.16.0 255.255.255.0
!
nat (dmz) 0 access-list NONAT

Problem

In what order and precedence is ASA firewall processing various NAT configurations.

NAT precedence rules 

Step 1.
NAT Exemption: This is always the first to be checked and has precedence over any other type of NAT rule that eventually conflicts with it.

Step 2.
Static Policy NAT: The motivation for this type of rule is to allow the selection of distinct global addresses for a given laddr, depending on the destination address (faddr) being contacted. This can be thought of as an exception to a generic static statement and, as such, needs to be configured before regular statics.

Step 3.
Static NAT: This is checked before all the Dynamic, Dynamic Policy, and Identity NAT rules. If some hosts that fall within a NAT Exemption range require static translation, the pertinent exceptions in the nat 0 access-list command need to be created, as previously illustrated in Example 8-14.

Step 4.
Dynamic Policy NAT/PAT: These rules are checked before the Dynamic and Identity NAT rules. If two rules of this category exist, the precedence is given by the order in which they were configured (does not matter if is a Policy PAT or Policy NAT).

Step 5.
Identity NAT: This unidirectional translation bypass method is evaluated before any Dynamic NAT or Dynamic PAT rule.

Step 6.
Dynamic NAT: This category has precedence over Dynamic PAT only.

Step 7.
Dynamic PAT: If after running through all the previous NAT types, ASA does not find a match, it still searches for a Dynamic PAT. If no matching rule is located, the implicit deny rule that characterizes the NAT-control model is enforced.

References

• Cisco Firewalls By: Alexandre M.S.P Moraes, chapter "Outbound NAT Analysis" 
• http://blog.ciscoinferno.net/you-me-and-nat-8-3
• Network Address Translation (NAT) order of operation in the PIX Firewall
• Packet Flow through Cisco ASA Firewall
• Order of NAT Commands Used to Match Real Addresses

ASA ssh login problem

Working for ISP is big fun. From all the work you do there is one routine like swapping of network devices (for example Cisco ASA firewall) that you are going to do. Not going into too much details the process is straight forward and requires:

• copy the config to new device
• rack the new device
• make sure that the switches and VLANs are configured properly
• change routing info if needed 

  Problem

After putting new ASA FW into rack you can connect using serial line but you can't access it over SSH. You getting this error message.

 
$ ssh 1.1.1.77
ssh_exchange_identification: Connection closed by remote host

Troubleshooting and solution

From serial console access enable debugging:

 
# debug ssh

Connect over ssh. You are going to see this logs on console:

 
Device ssh opened successfully.
SSH0: SSH client: IP = '212.100.225.42'  interface # = 2
SSH: unable to retrieve default host public key.  Please create a defauth RSA key pair before using SSH
SSH0: Session disconnected by SSH server - error 0x00 "Internal error"

Searching for 'unable to retrieve default host public key' finds the links in reference sections.  To fix this we need:

 
fw-asa(config)# crypto key generate rsa
INFO: The name for the keys will be: 
Keypair generation process begin. Please wait...

Once ASA has its own RSA key to use for SSH handshaking the logs from a sucessful SSH session looks like:

 
fw-asa# 
Device ssh opened successfully.
SSH0: SSH client: IP = '212.100.225.42'  interface # = 2
SSH: host key initialised
SSH: license supports 3DES: 2
SSH: license supports DES: 2
SSH0: starting SSH control process
SSH0: Exchanging versions - SSH-2.0-Cisco-1.25
SSH0: send SSH message: outdata is NULL
server version string:SSH-2.0-Cisco-1.25SSH0: receive SSH message: 83 (83)
SSH0: client version is - SSH-2.0-OpenSSH_4.3
client version string:SSH-2.0-OpenSSH_4.3SSH0: begin server key generation
SSH0: complete server key generation, elapsed time = 1830 ms
SSH2 0: SSH2_MSG_KEXINIT sent
SSH2 0: SSH2_MSG_KEXINIT received
SSH2: kex: client->server aes128-cbc hmac-md5 none
SSH2: kex: server->client aes128-cbc hmac-md5 none
SSH2 0: expecting SSH2_MSG_KEXDH_INIT
SSH2 0: SSH2_MSG_KEXDH_INIT received
SSH2 0: signature length 143
SSH2: kex_derive_keys complete
SSH2 0: newkeys: mode 1
SSH2 0: SSH2_MSG_NEWKEYS sent
SSH2 0: waiting for SSH2_MSG_NEWKEYSSSH0: TCP read failed, error code = 0x86300003 "TCP connection closed"
SSH0: receive SSH message: [no message ID: variable *data is NULL]

SSH2 0: Unexpected mesg type receivedSSH0: Session disconnected by SSH server - error 0x00 "Internal error"

References

1. http://www.myteneo.net/blog/-/blogs/accessing-cisco-asa-using-ssh/
2. http://ciscotalk.wordpress.com/2011/08/31/enabling-ssh-on-a-cisco-asa/

Best practices on how to learn multi vendor networking hardware with a help of modern virtualization

When I read my fist book to CCNA back in at the university I didn't know much about network simulators or emulators. I learn the stuff from the books and by practicing on our uni lab. But since my recent come back to networking again I needed today something that helps me to simulate or even reproduce some network config scenarios. As I'm not interested in buying any second hand Cisco gear (or any other vendor) or renting remote labs or setting complex repros in our datacenter I have found a simple solution for me: GNS3.

I didn't have a big chance yet to play with it extensively but at the first glimpse it has a huge potential. It can boot and emulate many vendor devices like [2]:

• Cisco switches and routers
• Cisco PIX and ASA firewalls
• Vyatta firewalls
• Juniper
• Linux Open vSwitch  
• generic VM

These are some further features described on the main page [1]:

Features overview

Design of high quality and complex network topologies.
Emulation of many Cisco IOS router platforms, IPS, PIX and ASA firewalls, JunOS.
Simulation of simple Ethernet, ATM and Frame Relay switches.
Connection of the simulated network to the real world!
Packet capture using Wireshark.
 

References

1. http://www.gns3.net/
2. http://www.gns3.net/appliances/ http://www.gns3.net/hardware-emulated/
3. http://www.gns3.net/screenshots/