Problem runing ssh or scp from a python script using the paramiko module

Paramiko project [1] is a native SSH Python library for scritpting. It provides an extensible API that allows you to imitate a SSH session, control it and later as well as execute commands.

I tried to use it to implement one of my Rackconnect scritps:

• create a cloud server, let's call it a bastion server 
• connect to bastion server over SSH 
• execute ssh command from bastion against other cloud server (10.178.7.217/ServiceNet) or
• execute scp command to copy a file from bastion to a cloud server over local ServiceNet 

I failed with my first attempt to solve this problem. I discovered that the SSH session doesn't have a tty attached to it.

First attempt.
Below is the error message I always got.

$ python -u example_paramiko_notty.py

 test 1
 cmd pwd; ls; date

stdout : /root
stdout : check_rackconnect.sh
stdout : Sun Aug 19 19:04:03 UTC 2012

 test 2
 cmd scp -q  -o NumberOfPasswordPrompts=1 -o StrictHostKeyChecking=no /root/check_rackconnect.sh root@10.178.7.217:~/; echo $? done.

stdout : 1 done.
stderr : lost connection

 test 3
 cmd scp -q -v -o NumberOfPasswordPrompts=1 -o StrictHostKeyChecking=no /root/check_rackconnect.sh root@10.178.7.217:~/; echo $? done.

stdout : 1 done.
stderr : Executing: program /usr/bin/ssh host 10.178.7.217, user root, command scp -v -t ~/
stderr : OpenSSH_5.3p1 Debian-3ubuntu3, OpenSSL 0.9.8k 25 Mar 2009
stderr : debug1: Reading configuration data /etc/ssh/ssh_config
stderr : debug1: Applying options for *
stderr : debug1: Connecting to 10.178.7.217 [10.178.7.217] port 22.
stderr : debug1: Connection established.
stderr : debug1: permanently_set_uid: 0/0
stderr : debug1: identity file /root/.ssh/identity type -1
stderr : debug1: identity file /root/.ssh/id_rsa type -1
stderr : debug1: identity file /root/.ssh/id_dsa type -1
stderr : debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu3
stderr : debug1: match: OpenSSH_5.3p1 Debian-3ubuntu3 pat OpenSSH*
stderr : debug1: Enabling compatibility mode for protocol 2.0
stderr : debug1: Local version string SSH-2.0-OpenSSH_5.3p1 Debian-3ubuntu3
stderr : debug1: SSH2_MSG_KEXINIT sent
stderr : debug1: SSH2_MSG_KEXINIT received
stderr : debug1: kex: server-client aes128-ctr hmac-md5 none
stderr : debug1: kex: client-server aes128-ctr hmac-md5 none
stderr : debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024 1024 8192) sent
stderr : debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
stderr : debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
stderr : debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
stderr : debug1: Host '10.178.7.217' is known and matches the RSA host key.
stderr : debug1: Found key in /root/.ssh/known_hosts:5
stderr : debug1: ssh_rsa_verify: signature correct
stderr : debug1: SSH2_MSG_NEWKEYS sent
stderr : debug1: expecting SSH2_MSG_NEWKEYS
stderr : debug1: SSH2_MSG_NEWKEYS received
stderr : debug1: SSH2_MSG_SERVICE_REQUEST sent
stderr : debug1: SSH2_MSG_SERVICE_ACCEPT received
stderr : debug1: Authentications that can continue: publickey,password
stderr : debug1: Next authentication method: publickey
stderr : debug1: Trying private key: /root/.ssh/identity
stderr : debug1: Trying private key: /root/.ssh/id_rsa
stderr : debug1: Trying private key: /root/.ssh/id_dsa
stderr : debug1: Next authentication method: password
stderr : debug1: read_passphrase: can't open /dev/tty: No such device or address
stderr : debug1: Authentications that can continue: publickey,password
stderr : debug1: No more authentication methods to try.
stderr : Permission denied (publickey,password).
stderr : lost connection

The problems is that the exec_command() function [2] doens't open a SSH session that has a terminal attached. I couldn't find a working solution using it so I have written and to use invoke_shell() function [3] instead. Most of the code was inspired and copied from an example found here [4].Below is my working script.

Second attempt This is the output when we run it this time.

$ python -u example_paramiko_with_tty.py 

 test 2
 cmd scp -q  -o NumberOfPasswordPrompts=1 -o StrictHostKeyChecking=no /root/check_rackconnect.sh root@10.178.7.217:~/; echo $? done.

Linux rctest 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
Last login: Sun Aug 19 19:47:09 2012 from bbb.rrr.com

root@rctest:~# 
/root/check_rackconnect.sh root@10.178.7.217:~/; echo $? done.no  

root@10.178.7.217's password: 

0 done.
root@rctest:~# 
command was successful:True

As we can see the invoke_shell() function is very different from the exec_command() one. To make it work we have to deal with all the terminal outputs and make sure we sent the command string at a right time. We can see as well as that with default terminal settings the output is not showing all text. We can see only part of the overlapped command string we sent in line #11

References

1. http://www.lag.net/paramiko/
2. http://www.lag.net/paramiko/docs/paramiko.SSHClient-class.html#exec_command
3. http://www.lag.net/paramiko/docs/paramiko.SSHClient-class.html#invoke_shell
4. http://stackoverflow.com/questions/1911690/nested-ssh-session-with-paramiko

Others interesting links

5. http://www.minvolai.com/blog/2009/09/how-to-ssh-in-python-using-paramiko/
6. http://jessenoller.com/2009/02/05/ssh-programming-with-paramiko-completely-different/