Continuous deployment for SaaS: A natural evolution from continuous integration
I have just read "The Software Revolution Behind LinkedIn’s Gushing Profits" on Wired, and here is my thoughts on the continuous deployment, and decided to share my thoughts on the practice.
In such practice, branching the development repository is often discouraged as it increases the NVA (*) of the need of "code fixes for integration". Such NVA quickly grows as much as the branch has been sitting by itself without being integrated for a period of time.
* Non-Value-Added: In Lean management, all the activities that does not directly add customer values, such as submitting status reports to supervisors, are regarded as "wastes" that should be reduced.
In a strict continuous integration environment, the developers may be expected to check-in their code to the main trunk in 60 minutes. That means one has to break down their work finely enough upfront so that each check-in would be marginal changes. In many cases, it a huge mental shift for developers in terms of the coding habit as well as the engineering approach itself. One would often face seemingly impossible cases.
At a company that I used to work, we shared one code base among over 400 developers, and we operated in continuous integration environment. We were not developing a SaaS solution, so it was not continuous deployment.
...anything sitting in trunk must be releasable at any point in time, and if it’s not releasable it’s just as significant as a site emergency. Stop all forward software development and everyone is all hands on deck to get trunk fixed.
The developers must be very disciplined to make continuous deployment or continuous integration work: When they see the build is failing on the build sever, everybody must stop checking-in the new code except for the one that fixes the problem. But because they integrate in high frequency and fix things before adding new uncertainties, the NVA of integration effort is often much smaller than the traditional practice (Like the way git users tend to do feature branching casually and often), making the build more stable and releasable more often.
The automated test coverage also must be very high. If we need to stage the testing sever to do manual testing to know if the existing features have been broken, forget about doing continuouss deployment.
One thing I want to know from LinkedIn that was not written in the article is that how to do CD when the development requires major data migration. At minimum, it requires clean architectures so that data integration issues are shielded from the application developments.
All in all, it is understandable that:
Shifting from feature-branch-based development to the new continuous deployment system required halting all development for two months as LinkedIn trained staff, migrated old code, and built out the automated tools it needed to make the new system work.
I'm sure it was a risky investment decision for a big company like LikedIn. I admire the LinkedIn management team because they:
- Have in-depth understandings of the technology to recognize the need for such investment prioritized over releasing the new features
- Patiently listened the engineers for the long term strategical decisions
- Are wise enough not to forget the staff training as that is arguably the most important and difficult part of practicing CD
It certainly has a big overhead for a company with legacy products, but the pay-off can be very big as in the LinkedIn case.
Dr. Daigo Tanaka, Certified Scrum Professional
Original post: April 18, 2013 | Last updated: April 18, 2013