Automation options for network devices
There are various networking vendors out there (example Cisco, Juniper, F5, ...). Every vendor provides command line access to its device. The networking shell is usually something unique for a vendor and requires some time to master it. The most popular way to access the shell is to use SSH protocol although.
It is sad that many networking vendors doesn't supply any form of API for device configuration and management (only very few, example F5 iControl). Usually there is only command line shell or GUI.
From operational point of view it is a tedious job to try to automate your routine tasks. In practice, at the end you are going to be forced to write your own scripts and programs. Below is a list of frameworks and libraries I've found to help me to get my job done.
As a site note, the good thing is that it is changing. Probably the most significant force behind the new technologies is SDN, Openflow and cloud.
What Perl libraries can I use to write my scripts to manage Cisco devices
http://search.cpan.org/~oliver/Net-Appliance-Session-4.131260/lib/Net/Appliance/Session.pm
http://search.cpan.org/~oliver/Net-CLI-Interact-2.131260/lib/Net/CLI/Interact.pm
http://search.cpan.org/~mingzhang/Angel_101/
What Python libraries can I use to manage Cisco devices
http://pydoc.net/Python/ciscolib/0.1/
More like a source code browser.
http://code.ohloh.net/search?s=cisco&browser=Default&pp=0&fl=Python&mp=1&ml=0&me=1&md=1&ff=1&filterChecked=true
You can always mange your SSH session manually using Python SSH libraries
http://stackoverflow.com/questions/5238000/persistent-ssh-session-to-cisco-router
http://linuxdynasty.org/219/howto-manage-your-networked-devices-using-python-and-pexpect/
What Ruby libraries can I use to manage Cisco devices
https://github.com/net-ssh?tab=repositories
Summary
From my investigation it looks that Perl has the most mature option when it comes to network devices automation libraries.
There is only few Python code available that natively supports network appliance management over SSH command line. Situation looks identical with Ruby. To write a bigger thing you would have need to manually write all code including SSH session(s) management, command execution, output delivery and redirections as well as error handling.
Search This Blog
Thursday, June 20, 2013
Wednesday, June 19, 2013
F5 Network BigIp cheat sheet
This post is a work in progress...
net self 10.2.2.2/30 { vlan FAILOVER
net self 10.176.30.100/19 { vlan hybridServiceNet-140
net self 10.176.30.102/19 { vlan hybridServiceNet-140
net self 10.176.94.132/19 { vlan hybridServiceNet-142
Or
# tmsh list net self | egrep -v 'floating|unit|allow-service' | xargs -n 7
net self 10.178.191.49/18 { vlan rackconnect110 }
net self 10.179.63.181/18 { vlan rackconnect112 }
- How to generate a list with one self ip and vlans per line
net self 10.2.2.2/30 { vlan FAILOVER
net self 10.176.30.100/19 { vlan hybridServiceNet-140
net self 10.176.30.102/19 { vlan hybridServiceNet-140
net self 10.176.94.132/19 { vlan hybridServiceNet-142
Or
# tmsh list net self | egrep -v 'floating|unit|allow-service' | xargs -n 7
net self 10.178.191.49/18 { vlan rackconnect110 }
net self 10.179.63.181/18 { vlan rackconnect112 }
- How to simulate F5 health check requests with empty Host header
- How to parse tmsh output
Labels:
automation,
bash,
cheat sheet,
command,
f5,
linux
Tuesday, June 18, 2013
How does SFTP work
Every network administrator knows how painful is it to troubleshoot FTP protocol issues. Many of them are related to the FTP design concept of independent control and data channels. Because both FTP data and control channels are handled in separate TCP session often network devices are allowing one TCP session but blocking the other. The end result is that you can't connect to FTP server or you can't download a file or your download hangs and never completes.
There are number of alternatives. On of them is SFTP (Secure File Transfer Protocol). The design of this protocols is very different from FTP even though it shares its name.
In FTP we have concept of separate control and data channels. The client can sent commands over the control TCP session and data transfers happens in the data TCP sessions. Both the control and data TCP session has its own TCP conventions. Passive and active FTP mode make it even more convoluted.
In SFTP there is one session (by default the client connect on port 22 from random source port). The single session is used to exchange a mix of control and data commands. This is still truth that for a file transfer you need a new TCP session but the session is initiated in a similar way on port 22. From network point of view there is not much difference how the first or second TCP session looks. The sessions are independent but connected.
Example tcpdump showing a single file transfer (100* is client, 200* is server):
Example loggin session from FileZilla:
References
There are number of alternatives. On of them is SFTP (Secure File Transfer Protocol). The design of this protocols is very different from FTP even though it shares its name.
In FTP we have concept of separate control and data channels. The client can sent commands over the control TCP session and data transfers happens in the data TCP sessions. Both the control and data TCP session has its own TCP conventions. Passive and active FTP mode make it even more convoluted.
In SFTP there is one session (by default the client connect on port 22 from random source port). The single session is used to exchange a mix of control and data commands. This is still truth that for a file transfer you need a new TCP session but the session is initiated in a similar way on port 22. From network point of view there is not much difference how the first or second TCP session looks. The sessions are independent but connected.
Example tcpdump showing a single file transfer (100* is client, 200* is server):
Nr Time Source Destination Protocol flags Length vlan hsrc hdst sport dport Window size value Seq# Ack# Info # connecting to the SFTP server 130 0.017634 100.0.0.1 200.0.0.1 TCP 0x02 66 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 65535 0 55373 > ssh [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=128 SACK_PERM=1 131 0.002186 200.0.0.1 100.0.0.1 TCP 0x12 66 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 8192 0 1 ssh > 55373 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1 132 0.000033 100.0.0.1 200.0.0.1 TCP 0x10 54 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32768 1 1 55373 > ssh [ACK] Seq=1 Ack=1 Win=4194304 Len=0 133 0.004121 200.0.0.1 100.0.0.1 SSHv2 0x18 87 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 258 1 1 Server Protocol: SSH-2.0-1.82_sshlib GlobalSCAPE\r 134 0.000234 100.0.0.1 200.0.0.1 SSHv2 0x18 97 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32767 1 34 Client Protocol: SSH-2.0-PuTTY_Local:_Jan__8_2012_14:40:35\r 135 0.000055 100.0.0.1 200.0.0.1 TCP 0x18 566 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32767 44 34 [TCP segment of a reassembled PDU] 136 0.000018 100.0.0.1 200.0.0.1 SSHv2 0x18 182 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32767 556 34 Client: Key Exchange Init 137 0.002431 200.0.0.1 100.0.0.1 SSHv2 0x18 542 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 258 34 44 Server: Key Exchange Init 138 0.000001 200.0.0.1 100.0.0.1 TCP 0x10 60 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 256 522 684 ssh > 55373 [ACK] Seq=522 Ack=684 Win=65536 Len=0 139 0.000076 100.0.0.1 200.0.0.1 SSHv2 0x18 70 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32763 684 522 Client: Diffie-Hellman Key Exchange Init 140 0.002168 200.0.0.1 100.0.0.1 SSHv2 0x18 334 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 256 522 700 Server: Diffie-Hellman Key Exchange Reply 141 0.018620 100.0.0.1 200.0.0.1 SSHv2 0x18 326 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32761 700 802 Client: Diffie-Hellman GEX Init 142 0.024606 200.0.0.1 100.0.0.1 SSHv2 0x18 902 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 254 802 972 Server: Diffie-Hellman GEX Reply 144 0.005229 100.0.0.1 200.0.0.1 SSHv2 0x18 70 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32768 972 1650 Encrypted request packet len=16[Malformed Packet] 145 0.000113 100.0.0.1 200.0.0.1 SSHv2 0x18 142 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32768 988 1650 Encrypted request packet len=88 146 0.001946 200.0.0.1 100.0.0.1 TCP 0x10 60 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 254 1650 1076 ssh > 55373 [ACK] Seq=1650 Ack=1076 Win=65024 Len=0 147 0.000001 200.0.0.1 100.0.0.1 SSHv2 0x18 174 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 254 1650 1076 Encrypted response packet len=120 148 0.000249 100.0.0.1 200.0.0.1 SSHv2 0x18 158 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 32767 1076 1770 Encrypted request packet len=104 149 0.002327 200.0.0.1 100.0.0.1 SSHv2 0x18 122 Cisco_ee:08:c0 Dell_75:49:e3 22 55373 254 1770 1180 Encrypted response packet len=68 ...... # we initiated a file transfer # new tcp session is created, similar like in passive FTP mode 1683 0.064012 100.0.0.1 200.0.0.1 TCP 0x02 66 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 65535 0 55375 > ssh [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=128 SACK_PERM=1 1684 0.002196 200.0.0.1 100.0.0.1 TCP 0x12 66 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 8192 0 1 ssh > 55375 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1380 WS=256 SACK_PERM=1 1685 0.000035 100.0.0.1 200.0.0.1 TCP 0x10 54 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32768 1 1 55375 > ssh [ACK] Seq=1 Ack=1 Win=4194304 Len=0 1686 0.004089 200.0.0.1 100.0.0.1 SSHv2 0x18 87 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 258 1 1 Server Protocol: SSH-2.0-1.82_sshlib GlobalSCAPE\r 1687 0.000506 100.0.0.1 200.0.0.1 SSHv2 0x18 97 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32767 1 34 Client Protocol: SSH-2.0-PuTTY_Local:_Jan__8_2012_14:40:35\r 1688 0.000053 100.0.0.1 200.0.0.1 TCP 0x18 566 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32767 44 34 [TCP segment of a reassembled PDU] 1689 0.000015 100.0.0.1 200.0.0.1 SSHv2 0x18 182 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32767 556 34 Client: Key Exchange Init 1690 0.002266 200.0.0.1 100.0.0.1 SSHv2 0x18 542 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 258 34 44 Server: Key Exchange Init 1691 0.000001 200.0.0.1 100.0.0.1 TCP 0x10 60 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 256 522 684 ssh > 55375 [ACK] Seq=522 Ack=684 Win=65536 Len=0 1692 0.000109 100.0.0.1 200.0.0.1 SSHv2 0x18 70 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32763 684 522 Client: Diffie-Hellman Key Exchange Init 1693 0.002230 200.0.0.1 100.0.0.1 SSHv2 0x18 334 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 256 522 700 Server: Diffie-Hellman Key Exchange Reply 1697 0.000446 100.0.0.1 200.0.0.1 SSHv2 0x18 326 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32761 700 802 Client: Diffie-Hellman GEX Init 1698 0.023470 200.0.0.1 100.0.0.1 SSHv2 0x18 902 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 254 802 972 Server: Diffie-Hellman GEX Reply 1699 0.027365 100.0.0.1 200.0.0.1 SSHv2 0x18 70 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32768 972 1650 Encrypted request packet len=16[Malformed Packet] 1700 0.000111 100.0.0.1 200.0.0.1 SSHv2 0x18 142 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32768 988 1650 Encrypted request packet len=88 1701 0.001906 200.0.0.1 100.0.0.1 TCP 0x10 60 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 254 1650 1076 ssh > 55375 [ACK] Seq=1650 Ack=1076 Win=65024 Len=0 1702 0.000001 200.0.0.1 100.0.0.1 SSHv2 0x18 174 Cisco_ee:08:c0 Dell_75:49:e3 22 55375 254 1650 1076 Encrypted response packet len=120 1703 0.000303 100.0.0.1 200.0.0.1 SSHv2 0x18 158 Dell_75:49:e3 Cisco_ee:08:c0 55375 22 32767 1076 1770 Encrypted request packet len=104 ... # once the tranfer is finished both sessions are closed 2498 0.015696 100.0.0.1 200.0.0.1 TCP 0x14 54 Dell_75:49:e3 Cisco_ee:08:c0 55373 22 0 3768 3158 55373 > ssh [RST, ACK] Seq=3768 Ack=3158 Win=0 Len=0
Example loggin session from FileZilla:
Status: Connecting to 200.0.0.1... Response: fzSftp started Command: open "user_name@200.0.0.1" 22 Command: Trust new Hostkey: Once Command: Pass: ********** Status: Connected to 200.0.0.1 Status: Retrieving directory listing... Command: pwd Response: Current directory is: "/" Command: ls Status: Listing directory / Status: Calculating timezone offset of server... Command: mtime "Archive" Response: 1365109008 Status: Timezone offsets: Server: 0 seconds. Local: 3600 seconds. Difference: 3600 seconds. Status: Directory listing successful Status: Connecting to 200.0.0.1... Response: fzSftp started Command: open "user_name@200.0.0.1" 22 Command: Trust new Hostkey: Once Command: Pass: ********** Status: Connected to 200.0.0.1 Status: Starting download of /paht/to/the/file.txt Command: cd "/" Response: New directory is: "/" Command: get "/paht/to/the/file.txt" "c:\Users\file.txt" Status: remote:/paht/to/the/file.txt => local:c:\Users\file.txt Status: File transfer successful, transferred 360 bytes in 1 second Status: Disconnected from server Status: Disconnected from server
References
Monday, June 17, 2013
Tables in Bash
There are many situation where a simple Bash script is more than enough to get a job done. But as much as I like Bash its primitive and quite sensitive syntax make me think twice before I code something more complex. Below is a nice trick I found how to deal with tables in Bash.
The most up to date gist can be found here: https://gist.github.com/rtomaszewski/5799274
The most up to date gist can be found here: https://gist.github.com/rtomaszewski/5799274
#!/bin/bash TESTS[0]=a,b,c TESTS[1]=1,2,3 for row in "${TESTS[@]}"; do IFS="," set $row col1=$1 col2=$2 col3=$3 echo "row was: $row" echo "col1 is $col1, col2 is $col2, col3 is $col3" done
Labels:
bash,
linux,
programming
Openstack or Linux or bash cheat sheet
This post is a work in progress.
# echo a b c | xargs -n 1 echo 'this is '
this is a
this is b
this is c
Practical demo of how to delete all your cloud servers
# nova --no-cache list | grep '[|]' | awk '{print $2}' | tail -n +2 | xargs -n1 echo nova delete
nova delete 0dafascd-e7e5-4531-9542-25132338a3fc
nova delete ffasff56-ef5a-42e8-aa96-594d14538def
nova delete ad509afa-0cc8-111b-a681-7c56cc354957
nova delete b9bfafaf-073d-4732-a9c0-2e6720938357
Connection to 92.52.111.222 80 port [tcp/http] succeeded!
netstat -nntulpa &> $file
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq
cat $file 2 | grep udp
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq
CLOSE_WAIT
CLOSING
ESTABLISHED
FIN_WAIT1
FIN_WAIT2
LAST_ACK
LISTEN
SYN_RECV
SYN_SENT
TIME_WAIT
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq | while read STATE; do echo $STATE; grep $STATE $file | wc -l; done
CLOSE_WAIT
2
CLOSING
8
ESTABLISHED
53
FIN_WAIT1
15
FIN_WAIT2
0
LAST_ACK
136
LISTEN
20
SYN_RECV
166
SYN_SENT
0
TIME_WAIT
2
Remove the fist line
Remove the line #2 and #3
Remove the first 2 lines
Extract source IP and port
Strip of the port number
- How to generate a list of commands base on input list.
# echo a b c | xargs -n 1 echo 'this is '
this is a
this is b
this is c
# nova --no-cache list | grep '[|]' | awk '{print $2}' | tail -n +2 | xargs -n1 echo nova delete
nova delete 0dafascd-e7e5-4531-9542-25132338a3fc
nova delete ffasff56-ef5a-42e8-aa96-594d14538def
nova delete ad509afa-0cc8-111b-a681-7c56cc354957
nova delete b9bfafaf-073d-4732-a9c0-2e6720938357
- Testing if you can establish a TCP session
Connection to 92.52.111.222 80 port [tcp/http] succeeded!
- some of the useful CLIs
fold - Filter for folding lines. This breaks the lines to have a maximum of x width column position (or bytes).
column - columnate lists
- How to check TCP / UDP network and socket statistics
netstat -nntulpa &> $file
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq
cat $file 2 | grep udp
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq
CLOSE_WAIT
CLOSING
ESTABLISHED
FIN_WAIT1
FIN_WAIT2
LAST_ACK
LISTEN
SYN_RECV
SYN_SENT
TIME_WAIT
cat $file | grep tcp | awk ' { print $6 } ' | sort | uniq | while read STATE; do echo $STATE; grep $STATE $file | wc -l; done
CLOSE_WAIT
2
CLOSING
8
ESTABLISHED
53
FIN_WAIT1
15
FIN_WAIT2
0
LAST_ACK
136
LISTEN
20
SYN_RECV
166
SYN_SENT
0
TIME_WAIT
2
Other useful links: link1, link2, link3
52 MB ./lib/libwireshark.so.2.0.2
17 MB ./lib/x86_64-linux-gnu/libicudata.so.48.1.1
- How to sort files based on file size
52 MB ./lib/libwireshark.so.2.0.2
17 MB ./lib/x86_64-linux-gnu/libicudata.so.48.1.1
- How to cat and highlight a word in text
$echo -n 'ello' | ( read a; read -u1 b ; echo "1st read : - $a -"; echo "2th read : = $b =" ) test 1st read : - ello - 2th read : = test =
- How to truncated and shrink the text output to your terminal screen width
$ cat tmp1 a1 a2 a3 a4
Remove the fist line
$ cat tmp1 | tail -n+2 a2 a3 a4
Remove the line #2 and #3
cat tmp1 | sed '2,3d' a1 a4
Remove the first 2 lines
$ cat tmp1 | tail -n+3 a3 a4
- How to extract IP address from tcpdump output
$ tcpdump -nr attack.log 21:35:49.553423 IP 162.13.0.27.22 > 82.44.149.5.51227: Flags [P.], seq 567291273:567291325, ack 2916928547, win 312, length 52 21:35:49.573227 IP 82.44.149.5.51227 > 162.13.0.27.22: Flags [.], ack 52, win 16516, length 0
Extract source IP and port
$ tcpdump -nr attack.log | tmp.xt |awk '{print $3}' 162.13.0.27.22 82.44.149.5.51227
Strip of the port number
$ tcpdump -nr attack.log | awk '{print $3}' | grep -oE '[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}' 162.13.0.27 82.44.149.5
- How to count strings in a text using awk
$ cat | awk ' { count+=NF } END { print count;}' 1 2 aaaa :rrr :ddjf -dd rrd ccc zz 1 2 3 4 444; -d df 16
Labels:
bash,
cheat sheet,
linux,
openstack
Devops language
Devops is a new discipline that brings a mix of experience from the operational and development area. You definitely need to understand what the development team is doing but not necessarily be able to do yourself. You need to know as well hows live in the operational/admin team looks like ;).
This is a nice slide that brings some of the hot/buzz words you hear often when people talk about devops:
The full presentation can be found here: http://www.slideshare.net/mtesauro/devops-ci-apis-oh-my-texas-linux-fest-2012.
References
This is a nice slide that brings some of the hot/buzz words you hear often when people talk about devops:
The full presentation can be found here: http://www.slideshare.net/mtesauro/devops-ci-apis-oh-my-texas-linux-fest-2012.
References
Labels:
devops
Thursday, June 13, 2013
How to crash your PC with fork-bomb Denial-of-service attack
You learn something new every day. When it comes to computer attacks I've thought I saw many things but this little one really make me smile.
THIS ONE WILL CRASH YOU LINUX BOX. DON'T RUN IN ON YOUR PRODUCTION SYSTEM. If you want to test it create some cloud server or better read the explanation of what it does on the wiki.
References
THIS ONE WILL CRASH YOU LINUX BOX. DON'T RUN IN ON YOUR PRODUCTION SYSTEM. If you want to test it create some cloud server or better read the explanation of what it does on the wiki.
:(){ :|: & };:
References
PS.
I dedicate this post to Tim ;)
Tuesday, June 11, 2013
Sublime keyboard shortcat list
This is work in progress...
Ctrl-B - show/hide results panel
Ctrl-D - highlight a word
Alt-F3 - multi selection for all highlighted words
F4 - (in the search results Ctrl+Shift+F) opens a file and takes you to highlighted line
Ctrl-F2 - marks a point in a line and create a point for multi selection if needed
Alt-F2 - edit all the marked lines at once (create a multi selection from all the marks above)
Alt-F2, Ctrl-F2 - removes all the marks in the file
Ctrl+Alt+Enter - In the Find/Replace window at the bottom it 'Replace All' matches
More keyboard shortcats https://gist.github.com/eteanga/1736542
More function to play with can be found here: http://www.sublimetext.com/docs/2/api_reference.html
- Below is a list of most common keyboard shortcats I use in Sublime (this is a work in progress post)
Ctrl-B - show/hide results panel
Ctrl-D - highlight a word
Alt-F3 - multi selection for all highlighted words
F4 - (in the search results Ctrl+Shift+F) opens a file and takes you to highlighted line
Ctrl-F2 - marks a point in a line and create a point for multi selection if needed
Alt-F2 - edit all the marked lines at once (create a multi selection from all the marks above)
Alt-F2, Ctrl-F2 - removes all the marks in the file
Ctrl+Alt+Enter - In the Find/Replace window at the bottom it 'Replace All' matches
More keyboard shortcats https://gist.github.com/eteanga/1736542
- Even more shortcats http://www.sublimetext.com/forum/viewtopic.php?f=2&t=10615
- The Menu -> Edit -> Permute Lines menu options provide a few more ways to sort, including Reverse, which sorts in reverse alpha order, starting with lowercase letters, Unique, which removes duplicated lines, and Shuffle, which sorta kinda randomizes the text.
- Sublime multi-selection keyboard shortcats
- How to show Build Results panel in Sublime editor
- How to enable API debug in Sublime and see all commands name in console (http://robdodson.me/blog/2012/06/23/sublime-text-2-tips-and-shortcuts/)
Sublime has a built in console terminal with ctrl+` . When type sublime.log_commands(True) there you enable verbose logging. When enabled you are going to see all commands that Sublime is executes as you use it. Just remember to turn logging off when you’re done :)
More function to play with can be found here: http://www.sublimetext.com/docs/2/api_reference.html
Labels:
cheat sheet,
editor,
shortcut,
sublime
Monday, June 10, 2013
Remote execution and configuration management framework written in Python
There are number of tools out there that you can use for automation and/or configuration management. Many of these tools are flexible and offer a range of command and GUI applications interfaces as well as programming libraries and API. If you are paranoid or have very unique requisitions you can writhe with a help of them the automaton scripts yourself. This is a simple example code to automate a task using the paramiko Python library.
Likely, the most popular one are puppet and chef. Puppet is written in Ruby where Chef is in Ruby and Erlang (the newest version).
Problem
Is there any automation and configuration framework that is natively written in Python.
Analisis and results description
Automation tool called Salt or Saltstack is written in Python and provide a native support for modules written in Python.
The tool is rapidly evolving and getting popular. On the main project page we can find number of resources like email group, wiki and IRC (http://saltstack.com/community). There is even a YouTube channel. Below is an introduction and example how salt works.
http://docs.saltstack.com/topics/tutorials/walkthrough.html
http://www.linuxjournal.com/content/getting-started-salt-stack-other-configuration-management-system-built-python
Likely, the most popular one are puppet and chef. Puppet is written in Ruby where Chef is in Ruby and Erlang (the newest version).
Problem
Is there any automation and configuration framework that is natively written in Python.
Analisis and results description
Automation tool called Salt or Saltstack is written in Python and provide a native support for modules written in Python.
The tool is rapidly evolving and getting popular. On the main project page we can find number of resources like email group, wiki and IRC (http://saltstack.com/community). There is even a YouTube channel. Below is an introduction and example how salt works.
http://docs.saltstack.com/topics/tutorials/walkthrough.html
http://www.linuxjournal.com/content/getting-started-salt-stack-other-configuration-management-system-built-python
Labels:
automation,
cloud,
puppet,
python,
salt
Sunday, June 9, 2013
SR-IOV technology enables low level network virtualization
In the virtualization space the SR-IOV technology was introduced in about 2008/2010 [1]. The technical details can be found under the links in the reference section but in plain English the technology allows to create many virtual devices base on single a physical device. For this to work the hardware (CPU, North chip) and operating system need to have SR-IOV support.
Below is a video demonstrating packet processing for an Intel Ethernet card that supports SR-IOV.
Interesting slides showing the concept from the video and reference links:
Below is a video demonstrating packet processing for an Intel Ethernet card that supports SR-IOV.
Interesting slides showing the concept from the video and reference links:
- After the frame enters the physical port on the NIC the low level driver/firmware (supporting the SR-IOV) distributes the packet (based on header classifications/hash value/etc) to separate virtual queues
- Each virtual queue is assigned directly to a virtual device
- Once the packet is in the queue it can be deliver to the VM DIRECTLY without the usual software hypervisor overhead
- Packets don't have to be copied from physical port buffer(s) to OS RAM and than from OS RAM to VM OS buffers. The data can be sent directly from the physical port to VM OS buffers. That way the hypervisor processing overhead can be minimised.
- Critical part for the technology is CPU and chipset virtualization support
- As access to physical RAM need to be protected between hypervisor and VMs as well as VMs themselves the virtual memory address is translated to the physical location by the north chipset
- For the DMA request to copy the packets the address translation between the hypervisor address space and the VM address space is transparent (north chip take care of it)
- Another view how the packet is delivered from physical port to the VM
- http://www.intel.com/content/dam/doc/application-note/pci-sig-sr-iov-primer-sr-iov-technology-paper.pdf
- http://www.intel.com/content/dam/doc/white-paper/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.pdf
- http://communities.intel.com/community/wired/blog/2010/09/07/sr-iov-explained
Labels:
cpu,
hardware,
hypervisor,
network,
sr-iov,
virtualisation
Thursday, June 6, 2013
Network appliance architecture
Networking is a fascinating topic. There are thousands of books and RFC describing protocols from layer 1 to layer 7. In every big network we find variety of network devices that handle traffic and provide additional enhanced services. Example of such devices can be switches, routers, load balancers, traffic accelerators, firewalls, IDS, DDOS mitigation devices and others. Services could be QoS, security and traffic deduplication etc.
Looking at the network devices someone could ask one interesting questions: how to build a network device; how would you describe an internal device architecture?
Of course every company has its own patents, secrets, methods etc. how they built, manage and operate network appliances. Below are some information I found when working with them or reading about them.
Looking at the network devices someone could ask one interesting questions: how to build a network device; how would you describe an internal device architecture?
Of course every company has its own patents, secrets, methods etc. how they built, manage and operate network appliances. Below are some information I found when working with them or reading about them.
- Arista network operating system switch architecture
- Dell DNOS or FTOS architecture
- F5 Network has its Linux based TMOS architecture (link1/link2)
Labels:
architecture,
network,
os
What OS is used to manage data centre class network appliances from Dell
Dell acquired Force10 in 2011 and that way gained access to data center networking technologies like 10 and 40 Gigabit Ethernet switches.
Like many hardware devices a network appliance needs an (network) operating system (NOS) to control resources and to provide monitoring and management interface. Base on the publicly available information a simplified view of a hardware and software architecture for Force10 network devices can be described as:
Like many hardware devices a network appliance needs an (network) operating system (NOS) to control resources and to provide monitoring and management interface. Base on the publicly available information a simplified view of a hardware and software architecture for Force10 network devices can be described as:
- Packet processing is accelerated with the help of an modern ASIC chip(s)
- There is a low level microcode/firmware ( operating system driver) to manage the ASIC
- All of this is integrated within a Linux or *BSD (NetBSD) operating system
- Dell uses 2 different NOS: Dell Networking Operating System (DNOS) or Force10 Operating System (FTOS) depending on the hardware
- http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Force10_FTOS_Spec_Sheet.pdf
- http://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Force10_Product_Quick_Reference_Guide.pdf
Labels:
architecture,
dell,
hardware,
linux
Cisco is leaving the ADC market and recommends NetScaler from Citrix
The networking giant Cisco is leaving the competitive Application Delivery Controller (ADC) market. The Cisco CSS and Cisco ACE products were not able to keep the pace with the competitors like F5 Networks, Citrix, Radware and Brocade.
The doc [3] provides some guidance how to choice the right hardware.
References
The doc [3] provides some guidance how to choice the right hardware.
References
Wednesday, June 5, 2013
Devops chalanges in companies
Devops team in your company
The DevOps name has been used for a while in IT industry. After we established what the name mean there is a time now to ask another questions:
I've copied 2 interesting facts from it: (a) what skills are essential and (b) what obstacles you may face if you would like to join a devops team :). More can be found in the report.
The DevOps name has been used for a while in IT industry. After we established what the name mean there is a time now to ask another questions:
- What is DevOps in your company and how your company/team is using it
- Would you consider yourself a devops engineer
- In what tasks is devops paradigm helping you and why do you like it
I've copied 2 interesting facts from it: (a) what skills are essential and (b) what obstacles you may face if you would like to join a devops team :). More can be found in the report.
Sunday, June 2, 2013
Sublime multi-selection keyboard shortcats
We have discussed the benefit of Sublime editor in our previous posts. In this one we will show how to use efficiently the multi selection feature.
Keyboard shortcat for random multi selection
With Ctrl + D you can create multiple selections when editing text. But when combining it with Ctrl + K you can skip some matches.
Example:
Copy the following text into your editor:
`Example text. This is an example text. Text has only one line`
Test 1:
Ctrl-F to find all `text` string.
Press Ctrl+D twice times.
Escape to get back to editor from the search dialogue.
You created a three word multi selection. You can edit word in 3 different places at once.
Test 2:
Ctrl-F to find all `text` string.
Press Ctrl+D once. It creates 2 word multi-selection.
Press Ctrl+K once. It ignores the last selected work. It gets back to single word selection.
Press Ctrl+D second time. It creates 2 word multi-selection.
Escape to get back to editor from the search dialogue.
You created a 2 word multi selection. You can edit the 2 words simultaneously. We skipped the word in the middle.
Of course you can always use ctrl+u to return to a previous selection :).
Sublime Cheat sheet
A short summary (more can be found here and on internet)
Ctrl + D repeat multi-select word
Ctrl + K skip multi-select word
Keyboard shortcat for random multi selection
With Ctrl + D you can create multiple selections when editing text. But when combining it with Ctrl + K you can skip some matches.
Example:
Copy the following text into your editor:
`Example text. This is an example text. Text has only one line`
Test 1:
Ctrl-F to find all `text` string.
Press Ctrl+D twice times.
Escape to get back to editor from the search dialogue.
You created a three word multi selection. You can edit word in 3 different places at once.
Test 2:
Ctrl-F to find all `text` string.
Press Ctrl+D once. It creates 2 word multi-selection.
Press Ctrl+K once. It ignores the last selected work. It gets back to single word selection.
Press Ctrl+D second time. It creates 2 word multi-selection.
Escape to get back to editor from the search dialogue.
You created a 2 word multi selection. You can edit the 2 words simultaneously. We skipped the word in the middle.
Of course you can always use ctrl+u to return to a previous selection :).
Sublime Cheat sheet
A short summary (more can be found here and on internet)
Ctrl + D repeat multi-select word
Ctrl + K skip multi-select word
Subscribe to:
Posts (Atom)