Remote execution and configuration management framework written in Python

There are number of tools out there that you can use for automation and/or configuration management. Many of these tools are flexible and offer a range of command and GUI applications interfaces as well as programming libraries and API. If you are paranoid or have very unique requisitions you can writhe with a help of them the automaton scripts yourself. This is a simple example code to automate a task using the paramiko Python library.

Likely, the most popular one are puppet and chef. Puppet is written in Ruby where Chef is in Ruby and Erlang (the newest version).

Problem

Is there any automation and configuration framework that is natively written in Python.

Analisis and results description

Automation tool called Salt or Saltstack is written in Python and provide a native support for modules written in Python.

The tool is rapidly evolving and getting popular. On the main project page we can find number of resources like email group, wiki and IRC (http://saltstack.com/community). There is even a YouTube channel. Below is an introduction and example how salt works.

http://docs.saltstack.com/topics/tutorials/walkthrough.html
http://www.linuxjournal.com/content/getting-started-salt-stack-other-configuration-management-system-built-python

Devops chalanges in companies

Devops team in your company 

The DevOps name has been used for a while in IT industry. After we established what the name mean there is a time now to ask another questions:

• What is DevOps in your company and how your company/team is using it
• Would you consider yourself a devops engineer
• In what tasks is devops paradigm helping you and why do you like it

These are only examples. To help to spread the knowledge further and to increase understanding of devops mentality even more Puppet Labs has conducted a survey. You can find more about it on its blog Get More Agile: Learn How to Automate One Small Thing with Puppet Enterprise ( a direct link to the survey report is there 2013 State of DevOps ).

I've copied 2 interesting facts from it: (a) what skills are essential and (b) what obstacles you may face if you would like to join a devops team :). More can be found in the report.

Popularity of automation and orchestration frameworks

It is difficult to list all possible solutions that exist but there are certain names that everyone knows in the industry when it comes to automation: Puppet Chef. Below is a small diagram showing their steady increase of popularity.

Among these big names there are many other often complementary or competing tools than can be used (the order is random): Fabric saltstack Ansible CFEngine Cobbler Foreman Jenkins VeeWee Juju.

Below is a graph from Google trends (if you can't see it you need to be logged in a Google account)

Openstack auto provisioning with Puppet and razor

To build and operate big Openstack infrastructure solutions you have to be able to deploy and provision quickly and effectively many new servers.

This is not definitive list but as a simple task you will need to make sure that all your servers have the right OS version, all dependency packages are installed and finally that the right Openstack code (projects like nova, cinder, etc) are deployed  This list is only as very simple example what you need to think about. As a demonstration in this blog I wanted to show an example how this can be achieved with a Puppet razor tool.

Problem

How to provision and configure Openstack servers.

Analysis and results description

Openstack+puppet+razor: all details of how to run this can be found here: http://wiki.debian.org/OpenStackRazorHowto

References

1. http://puppetlabs.com/blog/introducing-razor-a-next-generation-provisioning-solution/
2. http://wiki.debian.org/OpenStackRazorHowto
3. http://purevirtual.eu/2012/07/02/how-to-get-started-with-razor-and-puppet-part-1/

Dev vs Ops team and DevOps anti patterns

In this video from Puppet conf Roy Tyler talks about important aspects and rules that sysadmin, engineering and devops teams do and follow. He gives many examples and tries to explain the principles that lead into many bad ideas within organisations when it comes to run operation, engineering or devs teams. He calls the bad ideas ops anti patterns. The full video can be found here: We'll Do It Live - Operations Anti-Pattern. Below are couple of notes I took when watching it :).

• ~4.20m; use configuration management; we are in transition and your infrastructure should be treated not different than a software code
• ~6.00m; use automatic testing
• ~7.40m; if your infrastructure is treated as code please do implement a release management process
• ~8.40m; scripts written by ops team are a great way to start but without some good software engineering practices it doesn't work in long time
• ~9.20m; don't use root user for code deployments into production but rather a well known good tools design for it 
• ~11.40m; Ops team tends to be reactive instead of proactive; this is dangerous because your SPOF (single point of failures) are changing as your application matures and gets more complex
• ~14.30m; dev team may take care of HA in software but there is a lot more that needs to be done on the infrastructure level as well; dedicate resources or highly skilled contractors to work on it as soon as possible
• ~14.40m; HA is never extensively tested so failures should be expected
• ~15.30m; when building in cloud you have to assume that everything, I mean everything is going to fail
• ~16.00m; make sure that your alerting system is not over logging
• ~18.45m; this sysadmin attitude is wrong: never touch a running system
• ~19.30m; use continues integration to minimize long-term risk
• ~21.40m; isolated silos like a separate dev and ops team that don't talk to each other is wrong
• ~24.30m; share the necessary information about your production to developers
• ~25.00m; strict control is an enemy of creativity
• ~25.30m; too much IT and ops control will lead to wrong and bad workarounds
• ~26.50m; silos and poor communication will lead to waist of resources
• ~29.00m; don't jump and use the hype-bla products/tools only because they are popular on Internet; by choosing the right tools you have to evaluate a risk between:
• from personal experience knowing the bad design and limitations the old tools have vs
• knowing the advantages the new products have from only reading about them
• ~30.40m; don't invent tools in house before you evaluate potential solutions
• ~33.50m; don't build your own packaging system; use the one that opensource or some vendors offer
• ~36.40m; don't netboot all our hardware; there is a reason the servers have local disk that can be used
• ~39.20m; don't delete your production data to test your backup
• ~40.30m; don't trust your vendor
• ~43.00 don't use multi data center deployments if your application is not ready for it; use other methods to implement disaster recovery if needed
• ~45.00 have a centralized location for code that is used for production deployments

References

1. http://www.agileweboperations.com/devops-anti-patterns
2. http://unethicalblogger.com/2012/10/09/video-well-do-it-live.html

Redhat application deployment automaton with Puppet module and F5

I have found this youtube video [1] on the the Puppet channel. It is an interesting presentation about the day to day issues the folks from Redhat run into when developing, managing and deploying new code for customer facing sites they host.

On of the things they mention that was difficult for them and why the started this project were:

• Lack of a cross functional team who could understand the whole architecture.
• People tend to have a limited understanding of other areas that they don't work in; Examples are SysAdmin about NetAdmin, NetAdmin about Develpers etc.
• On new depoyment a difficulties to find a team/person who can take an ownership of an issue as the problem may be above the area of their own expertise.
• Diverse and inconsistencies dev, staging and production systems.

To solve the problem they decided to refactor the architecture and automate as much as it was possible. The video [1] shows what issues they run into and how they solved it with a help of Puppet module they created and the F5 load balancer.

In short a message they convey in the presentation is: automate, standardize, and automate once again.

References

1. Managing F5 LTM with Puppet - Matthew Carpenter and Bret McMillan of Red Hat
2. http://www.youtube.com/user/PuppetLabsInc