Home directory and dotfiles management

In Linux you can customize your environment by creating custom aliases and various scripts. After some time this flexibility becomes difficult to managed if you work on many different machines. And if some of the hosts are cloud servers that are being deleted and recreated setting up your home directory can become a boring and annoying task.

Problem

How to customize and maintain your home directory config files on many servers.

Solution

1. Create a repository for your configs on github. You can track your config files in a repository like mine: dotfiles on github. There are many existing repositories that that you can clone or simply reuse.
2. On the server/cloud server install https://github.com/andsens/homeshick. This bash script will help us to maintain the config files.
3. Download your dotfiles (config files) repository on the server
4. Install it in your home directory
5. Add the customization to .bashrc file

References

https://github.com/rtomaszewski/dotfiles
https://github.com/andsens/homeshick
http://dotfiles.github.io/

How to extract a single SSL connection from tcpdump

There is not better tool for SSL troubleshooting than ssldump (a very useful how to use in a form of F5 solution can be found here: SOL10209: Overview of packet tracing with the ssldump utility.

The ssldump tool is not perfect although. It can produce only text output. The output is a mixture of SSL handshaking requests and data connections.

This little tool https://github.com/akozadaev/ssld-extract can help to extract a single SSL session. An example usage is provided below.

root@server:~/ssld-extract/# ssldump -n -r example1.pcap  > example1.pcap.txt
root@server:~/ssld-extract/pp# python ssld-extract.py -c -n1 ~/ssld-extract/example1.pcap.txt
New TCP connection #1: 192.168.0.2(57122) <-> 72.26.232.202(443)
1 1  0.1946 (0.1946)  C>S  Handshake
      ClientHello
        Version 3.1
        resume [32]=
          7b 9a 08 2f 3f c0 5e 70 c8 9e b6 f8 61 a0 4e 9e
          d9 84 07 e5 94 13 f8 e8 87 33 96 0d f4 a4 9f 6a
        cipher suites
        Unknown value 0xc00a
        Unknown value 0xc014
        Unknown value 0x88
        Unknown value 0x87
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        Unknown value 0xc012
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
...

        compression methods
                  NULL
1 2  0.3973 (0.2027)  S>C  Handshake
      ServerHello
        Version 3.1
        session_id[32]=
          d4 65 5e b6 3d 33 88 8c bd 7e 56 65 13 71 9f 52
          30 47 ea e1 c0 d6 1f 72 12 b9 2f 8f 6b 42 b2 68
        cipherSuite         TLS_RSA_WITH_RC4_128_SHA
        compressionMethod                   NULL
1 3  0.3974 (0.0001)  S>C  Handshake
      Certificate
1 4  0.3974 (0.0000)  S>C  Handshake
      ServerHelloDone
1 5  0.4006 (0.0031)  C>S  Handshake
      ClientKeyExchange
1 6  0.4006 (0.0000)  C>S  ChangeCipherSpec
1 7  0.4006 (0.0000)  C>S  Handshake
1 8  0.5794 (0.1788)  S>C  ChangeCipherSpec
1 9  0.5794 (0.0000)  S>C  Handshake
1 10 0.5814 (0.0019)  C>S  application_data
1 11 0.5819 (0.0004)  C>S  application_data
1 12 0.7806 (0.1987)  S>C  application_data

As you can see it was able to extract the single connection what is a huge help if you need to analyze a big tcpdump file.

How to checkout Nova Folsom or Grizzly branch on github

Since Openstack Folsom release the source code is tracked on github. The previous releases like Essex, Diablo ... were maintained on Canonical Launchpad.

https://launchpad.net/nova/essex
https://github.com/openstack/nova

Problem

How to checkout Folsom release

Solution

We start with checking out latest nova code.

 
git clone git://github.com/openstack/nova.git
Cloning into 'nova'...
remote: Counting objects: 174497, done.
remote: Compressing objects: 100% (40106/40106), done.
remote: Total 174497 (delta 140401), reused 164045 (delta 130984)
Receiving objects: 100% (174497/174497), 116.95 MiB | 2.01 MiB/s, done.
Resolving deltas: 100% (140401/140401), done.

We can see what branches are available.

 
cd nova
 git branch
* master

At first it looks like something went wrong. We can see some tags although.

 
git tag
0.9.0
2011.1rc1
2011.2
...

To see all the branches (local and remote) we need to use -r or -a options.

 
git branch -r
  origin/HEAD -> origin/master
  origin/master
  origin/stable/folsom
  origin/stable/grizzly

git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/stable/folsom
  remotes/origin/stable/grizzly

When combining with -v we can see the last commits ID. We see that local master is the same as remotes/origin/master.

 
git branch -v -a
* master                        e4f05ba Imported Translations from Transifex
  remotes/origin/HEAD           -> origin/master
  remotes/origin/master         e4f05ba Imported Translations from Transifex
  remotes/origin/stable/folsom  6740c41 Check QCOW2 image size during root disk creation
  remotes/origin/stable/grizzly 159fdd2 Merge "Detach volume fails when using multipath iscsi"

We can see more info about the remote branch.

 
 git remote -v show origin
* remote origin
  Fetch URL: git://github.com/openstack/nova.git
  Push  URL: git://github.com/openstack/nova.git
  HEAD branch: master
  Remote branches:
    master         tracked
    stable/folsom  tracked
    stable/grizzly tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

To checkout a remote branch we use the standard checkout with -b options.

 
git checkout -b folsom  remotes/origin/stable/folsom
Checking out files: 100% (2510/2510), done.
Branch folsom set up to track remote branch stable/folsom from origin.
Switched to a new branch 'folsom'

git branch -a
* folsom
  master
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/stable/folsom
  remotes/origin/stable/grizzly

 git branch -a -v
* folsom                        6740c41 Check QCOW2 image size during root disk creation
  master                        e4f05ba Imported Translations from Transifex
  remotes/origin/HEAD           -> origin/master
  remotes/origin/master         e4f05ba Imported Translations from Transifex
  remotes/origin/stable/folsom  6740c41 Check QCOW2 image size during root disk creation
  remotes/origin/stable/grizzly 159fdd2 Merge "Detach volume fails when using multipath iscsi" into stable/grizzly

As you can see it created a new local branch and set the pointer to the remote one. There is another way as well to do the same. We use the Grizzly branch to demonstrate this.

 
git checkout --track remotes/origin/stable/grizzly
Checking out files: 100% (2355/2355), done.
Branch stable/grizzly set up to track remote branch stable/grizzly from origin.
Switched to a new branch 'stable/grizzly'

git branch -a -v
  folsom                        6740c41 Check QCOW2 image size during root disk creation
  master                        e4f05ba Imported Translations from Transifex
* stable/grizzly                159fdd2 Merge "Detach volume fails when using multipath iscsi" into stable/grizzly
  remotes/origin/HEAD           -> origin/master
  remotes/origin/master         e4f05ba Imported Translations from Transifex
  remotes/origin/stable/folsom  6740c41 Check QCOW2 image size during root disk creation
  remotes/origin/stable/grizzly 159fdd2 Merge "Detach volume fails when using multipath iscsi" into stable/grizzly

git branch -a
  folsom
  master
* stable/grizzly
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/stable/folsom
  remotes/origin/stable/grizzly

Once we are done with the checking out we can always get back to the most up to date and latest code.

 
git checkout master
Switched to branch 'master'

References

1. http://git-scm.com/book/ch3-5.html
2. http://stackoverflow.com/questions/9537392/git-fetch-remote-branch

How to find your public ip from bash

There are many sites that can show you your current public IP. But only a few of them are designed in a way so you can interact with them from command line using curl for example.

Below is a nice and elegant way to find out your public IP using curl

 
alias myip='curl --silent checkip.dyndns.org | egrep --only-matching "[0-9\.]+"'

root@server:~# myip
1.79.21.123

References

1. https://github.com/pex/config/blob/master/bash/shortcuts.sh
2. https://github.com/rtomaszewski/dotfiles
3. http://dotfiles.github.io/

How to merge changes from a custom branch into master branch on github

As per our release policy for the api-challenge we have created a new branch called challenge5 to release the code that implements task #5. After the initial release we found some bugs. We fixed them and committed the changes back to our challenge5 repository using this method How to modify files in your branch on github. These are the changes made to the challenge5 branch:

https://github.com/rtomaszewski/api-challenge/commit/7872d14cc22e96b3b9f2049ae0d76a7697af22c6

Problem

The changes made on the challenge5 branch are not visible in master repository. How to copy the changes from challenge5 to master branch.

Solution

To copy the changes from one branch to another you can use github merging feature.

 
PS > git status .\challenge5.py
# On branch master
nothing to commit (working directory clean)

PS > git branch
  challenge2
  challenge3
  challenge4
  challenge5
* master

PS> git merge challenge5
Updating d609784..7872d14
Fast-forward
 challenge5.py |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

PS > git status .\challenge5.py
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
nothing to commit (working directory clean)

PS > git push
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:rtomaszewski/api-challenge
   d609784..7872d14  master -> master

References

1. http://learn.github.com/p/branching.html

How to modify files in your branch on github

You created and released a new branch following the RERO development philosophy paradigm. Later on you discover that there are some bugs that you would like to fix in your branch and publish.

Problem

How to fix a file in a branch.

Solution

We need to checkout the branch from repository.
Modify the file and fix the issue.
Commit the changes back to our branch.

An example session is showed below.

 
git checkout challenge2
#now you modify the file, once done
git commit -m 'fixed delete_images func' -a
git push origin

# after modification we want to return in default branch
git checkout master
git branch

How to create a branch on github to freeze your code in a particular version

The nature of developing is that the code is in constant change. These could be new features, bug fixes, code refactoring or some modifications to polishing the code.

Sooner or later you are going to realize that on some occasions you need to have access and a reference to the code you tested or officially published. This is a big topic and I don't want to go into deep details about it. If you are interested to know more try to search on the web for Software Release Management topic.

As a simple example please take a look at my previous blog Challenge 1 script. I tested my code and documented the expected results. Some days after as I worked on the other scripts I made changes to my original code as well. For anyone who would like to take a look what I was describing previous it would be very difficult. Looking the latest code from github is not what you really want.

Problem

How to create a branch on github to release a particular version of your code.

Solution

 
$ git branch
* master
$ git checkout 47677b8c1cf79e6517b2caf7b32aca59b4d38795
HEAD is now at 47677b8... prints ipv4 ip

$ git branch
* (no branch)
  master

$ git branch challange1
$ git branch
* (no branch)
  challange1
  master

$ git checkout challange1
Switched to branch 'challange1'

$ git branch
* challange1
  master

$ git push origin challange1
Warning: Permanently added 'github.com,204.232.175.90' (RSA) to the list of known hosts.
Counting objects: 33, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (23/23), done.
Writing objects: 100% (33/33), 6.10 KiB, done.
Total 33 (delta 13), reused 15 (delta 4)
To git@github.com:rtomaszewski/api-challenge.git
 * [new branch]      challange1 - challange1

After the branch is created you can always download the code for it: https://github.com/rtomaszewski/api-challenge/tree/challange1

References

1. https://github.com/Kunena/Kunena-2.0/wiki/Create-a-new-branch-with-git-and-manage-branches 
2. http://git-scm.com/book/en/Git-Branching-Basic-Branching-and-Merging

Howto add a file to api-challenge repository

The full summary of all posts for API Rackspace challenge can be found here:
Rackspace api-challenge summary

Adding my first file to the api-challenge repo 

Before we start coding our first program let's checkout the api-challenge repository we created.

 
cd C:\Users\rado4752\Desktop\docs\development\_mycoding\api-challenge
git clone git@github.com:rtomaszewski/api-challenge.git

As soon as you create a file you are working on you can add it to track changes.

 
git add challange1.py
git commit -m 'init'
git push origin master

After you have completed some of your work and would like to upload a new version of the file to the master repository.

 
git commit -m 'changed output' .\challange1.py
git pull origin master
git push origin master

References

1. http://rtomaszewski.blogspot.co.uk/search/label/github
2. http://rtomaszewski.blogspot.co.uk/search/label/git

Practical online github course

There are many modern distributed versioning systems and github is one of them. This link is going to take you to a simple online but practical course where you can practice the basic git commands.

Learning by doing: http://try.github.com/ (http://www.codeschool.com/courses/try-git)

Commands reference:

 
$ git init
$ git status
$ git add octocat.txt
$ git status
$ git commit -m "Add cute octocat story"
$ git add '*.txt'
$ git commit -m 'Add all the octocat txt files'
$ git log
$ git remote add origin git@github.com:rtomaszewski/try_git.git
$ git push -u origin master
$ git pull origin master
$ git diff HEAD
$ git add octofamily/octodog.txt
$ git diff --staged
$ git reset octofamily/octodog.txt
$ git checkout -- octocat.txt
$ git branch clean_up
$ git checkout clean_up
$ git rm '*.txt'
$ git commit -m "Remove all the cats"
$ git checkout master
$ git merge clean_up
$ git branch -d clean_up
$ git push