Dirty trick how to analysis ASA performance based on interface overruns and underruns

There are number of firewall vendors on the market you can chose from (other links to Gartner magic quadrant for firewalls here and here). Every vendor has a product line ranging from the low to high end firewalls. An example product list for Cisco ASA can be seen here: http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html#~tab-b

Problem

I can see in my firewalls interface stats underruns and overrruns and the counters increase.

Solution

This is rather a dirty trick and your monitoring system should be able to graph the interface stats. But if you are in a position like me where you have no visibility to interface statistics like you could have in Zenoss, Cacti, Zabbix or other monitoring system you may need to manually check this...

• We need to first start collecting data so we can look at it later.

Run at least one a day the command and save in a file 1.txt, 2.txt, etc.

sh clock
sh int

• After some time you should have a collection of files 

$ ls -1 *.txt
1.txt
2.txt
3.txt
4.1.txt
4.2.txt
5.1.txt
6.1.txt
8.txt

An example outpout form the file can be seen here: https://github.com/rtomaszewski/experiments/blob/master/asa-interfaces-example.txt

• Download the script asa-interfaces.sh and run it 

bash asa-interfaces.sh

Base on the files you collected it will generate stats for every interface (time stamp is in the last column). Example output:

Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        31 input errors, 0 CRC, 0 frame, 31 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        94 input errors, 0 CRC, 0 frame, 94 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        104 input errors, 0 CRC, 0 frame, 104 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        719 input errors, 0 CRC, 0 frame, 719 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        734 input errors, 0 CRC, 0 frame, 734 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        1502 input errors, 0 CRC, 0 frame, 1502 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        1794 input errors, 0 CRC, 0 frame, 1794 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        1881 input errors, 0 CRC, 0 frame, 1881 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1921 input errors, 0 CRC, 0 frame, 1921 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1971 input errors, 0 CRC, 0 frame, 1971 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1 input errors, 0 CRC, 0 frame, 1 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        364 input errors, 0 CRC, 0 frame, 364 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        382 input errors, 0 CRC, 0 frame, 382 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        392 input errors, 0 CRC, 0 frame, 392 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        444 input errors, 0 CRC, 0 frame, 444 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        468 input errors, 0 CRC, 0 frame, 468 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        707 input errors, 0 CRC, 0 frame, 707 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        756 input errors, 0 CRC, 0 frame, 756 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        640 input errors, 0 CRC, 0 frame, 640 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        658 input errors, 0 CRC, 0 frame, 658 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        683 input errors, 0 CRC, 0 frame, 683 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        797 input errors, 0 CRC, 0 frame, 797 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        811 input errors, 0 CRC, 0 frame, 811 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        863 input errors, 0 CRC, 0 frame, 863 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        984 input errors, 0 CRC, 0 frame, 984 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        1052 input errors, 0 CRC, 0 frame, 1052 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 07:48:45.631 cst Wed Feb 12 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 05:38:08.573 cst Thu Feb 13 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:37:26.853 cst Fri Feb 14 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 03:52:52.523 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10:03:08.799 cst Wed Feb 19 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 06:11:22.244 cst Thu Feb 20 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:35:21.315 cst Sun Feb 23 2014
        0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 08:22:57.704 cst Mon Feb 24 2014



Interface GigabitEthernet0/0 "outside", is up, line protocol is up
        646983182 packets output, 473597063148 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        700155558 packets output, 509814505730 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        753341661 packets output, 546026853810 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        1025937535 packets output, 734304301602 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        1054530605 packets output, 761409276094 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        1105491616 packets output, 798565630885 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        1264871240 packets output, 907739984962 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        1315876113 packets output, 943680519398 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/1 "dmz", is up, line protocol is up
        985243431 packets output, 309823858329 bytes, 459 underruns 07:48:45.631 cst Wed Feb 12 2014
        1070533450 packets output, 336205856058 bytes, 459 underruns 05:38:08.573 cst Thu Feb 13 2014
        1159894277 packets output, 366047579951 bytes, 483 underruns 03:37:26.853 cst Fri Feb 14 2014
        1596471490 packets output, 500836893219 bytes, 483 underruns 03:52:52.523 cst Wed Feb 19 2014
        1635530484 packets output, 511489408071 bytes, 483 underruns 10:03:08.799 cst Wed Feb 19 2014
        1722164227 packets output, 536769375853 bytes, 483 underruns 06:11:22.244 cst Thu Feb 20 2014
        2032554621 packets output, 636075162304 bytes, 2831 underruns 08:35:21.315 cst Sun Feb 23 2014
        2174454722 packets output, 688313076839 bytes, 2831 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/2 "myapp1", is up, line protocol is up
        1968362 packets output, 524301440 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1987058 packets output, 526612914 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2005883 packets output, 528940672 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        4036852 packets output, 3167775775 bytes, 2831 underruns 03:52:52.523 cst Wed Feb 19 2014
        13676338 packets output, 15853359823 bytes, 2831 underruns 10:03:08.799 cst Wed Feb 19 2014
        23861856 packets output, 16649050850 bytes, 3052 underruns 06:11:22.244 cst Thu Feb 20 2014
        66743830 packets output, 20187731129 bytes, 5941 underruns 08:35:21.315 cst Sun Feb 23 2014
        80290600 packets output, 21286340673 bytes, 6860 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet0/3 "state-failover", is up, line protocol is up
        16582048 packets output, 17699836232 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        17971640 packets output, 19234649922 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        19380417 packets output, 20791969660 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        26970739 packets output, 29162172960 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        27259841 packets output, 29471830004 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        29612954 packets output, 32141440890 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        39077736 packets output, 42912691094 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        42074827 packets output, 46322035220 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface Management0/0 "lan-failover", is up, line protocol is up
        1863787 packets output, 265441230 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        1977505 packets output, 281732398 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        2091970 packets output, 298145244 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        2718733 packets output, 387975068 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        2750523 packets output, 392530668 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        2855417 packets output, 407567752 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        3242848 packets output, 463113612 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        3366749 packets output, 480878922 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/0 "inside", is up, line protocol is up
        229534738 packets output, 55157234973 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        249682890 packets output, 59948227086 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        272763726 packets output, 66350185657 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        378020447 packets output, 91307448807 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        384704374 packets output, 93165635304 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        402556578 packets output, 97469565455 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        492798902 packets output, 119550853698 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        564346002 packets output, 137603523999 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/1 "app2", is up, line protocol is up
        142287604 packets output, 56966204294 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        154809474 packets output, 62049309926 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        167733332 packets output, 67152657884 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        231962627 packets output, 93689642614 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        235640974 packets output, 95384548398 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        249769631 packets output, 103735197461 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        290301462 packets output, 119748550482 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        303003248 packets output, 125098088305 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014
Interface GigabitEthernet1/2 "", is administratively down, line protocol is down
        0 packets output, 0 bytes, 0 underruns 07:48:45.631 cst Wed Feb 12 2014
        0 packets output, 0 bytes, 0 underruns 05:38:08.573 cst Thu Feb 13 2014
        0 packets output, 0 bytes, 0 underruns 03:37:26.853 cst Fri Feb 14 2014
        0 packets output, 0 bytes, 0 underruns 03:52:52.523 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 10:03:08.799 cst Wed Feb 19 2014
        0 packets output, 0 bytes, 0 underruns 06:11:22.244 cst Thu Feb 20 2014
        0 packets output, 0 bytes, 0 underruns 08:35:21.315 cst Sun Feb 23 2014
        0 packets output, 0 bytes, 0 underruns 08:22:57.704 cst Mon Feb 24 2014

Once you have the data in-front of you can easily see how the stats were changing over time, over longer period of time like a week.

In my case we suspected that the Firewall hit the capacity limit but further investigation confirmed that the device is doing well and no upgrade is necessary.

References

http://www.gossamer-threads.com/lists/cisco/nsp/152428
http://ccna2ccnp.blogspot.co.uk/2012/12/ciscoasa-oversubcription-maximizing.html
http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1904.html
http://en.wikipedia.org/wiki/Buffer_underrun
http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/115985-asa-overrun-product-tech-note-00.html