Rackspace Openstack lesson learned from continues integration and continuous deployment

When a software like Openstack gets bigger and bigger and some of the old features matures and new are added over time, there are going to be a number of challenges and complexities that you will see.

These challenges will be for the engineering team who wants to keep the development pace and rolling out new features, for the operational team that needs to look after the existing deployed infrastructure and be capable of planning and executing upgraded when needed, as well as for the support team who could be the first front of people who are notified when things get broken and things get broken when you move fast.

As Rackspace is one of the biggest Openstack consumer out there there are couple of lesson we can learn from them.

• Continues delivery is an obligatory task if you want to make sure your infrastructure is running the latest code 

We’re still plugging away on efforts to do continuous delivery from Nova trunk to our production environments. Our deploy pace has slowed as we are learning how to scale this effort and keep up with the pace of commits. [1]

• You need to have as much automatic testing as possible

To help facilitate progress, we’ve open-sourced our testing framework, Cloud Cafe, and engaged with the community to talk about how we might move our extensive test suite further upstream in the process.[1]

Continuous Delivery: While developing our OpenStack-based cloud, we created a Continuous Delivery/Continuous Integration model through which every change to the code goes through automated tests.  This Continuous Delivery pipeline lets us make a lot of changes quickly and gives us the ability to rapidly and confidently deploy new features.[2]

• Your deployment model can take the benefit of the cloud agility 

We’re eating our own dog food. We actually use OpenStack to run the control systems for our public cloud.

If we need more API nodes or additional scheduling capacity, we spin up more instances in our infrastructure cloud, which we call iNova. When upgrading nodes, we spin up new instances with the updated software and roll those into the load balancer. If things work as expected, the old instances are deleted. If there are issues, rollback is as simple as putting the old instances back in rotation. This blue/green deployment approach is a well known pattern. But, doing it in the cloud means we don’t have to have two sets of physical hardware standing around for the task. iNova is now being adopted across our cloud engineering teams for development, testing and production uses.[2]

References

1. http://www.rackspace.com/blog/ramping-up-our-openstack-investment-involvement/
2. http://www.rackspace.com/blog/how-rackspace-re-wrote-the-cloud-with-openstack-continuous-delivery/
3. https://github.com/stackforge/cloudcafe
4. http://www.slideshare.net/openstackindia/introduction-to-tempest
5. https://github.com/openstack/tempest/tree/master/tempest