The DevOps Pilgrimage – Part I – The Abyss

Introduction

For a large legacy project, the transformation from the current manual process of testing and release management to adopt the new Agile/DevOps-centric model is a journey fraught with obstacles and dead-ends. What we need is our own handy guide who can lead us from the current AdHoc hell to DevOps heaven.

In this article, in the first part, I describe the infernal pain we suffer from the ad-hoc, manual processes that characterise the current situation. In part II, I then describe the steps through which we purge the system of the ad-hoc processes, and automate the testing and release steps. Then in Part III, I detail the sublime benefits that we can get by extended automation. This is where we meet and transcend customer expectations since we are able to develop, test and release software with speed, quality and ease.

Part I – The Abyss

These are some of the characteristics of ad-hoc and manual processes

• Long-lived branches: Developer teams create branches for doing large chunks of work, and these branches persist for months and even quarters. Eventually, a couple of weeks before the testing phase, the branches are merged. This merge process is, unsurprisingly, painful because of all the accumulated changes that create conflicts. There is a lot of finger-pointing and blame-gaming when the system seems broken beyond repair and it takes weeks to merge all the changes and compile it error-free.
• Long Testing Cycles: Weeks and weeks of SIT and UAT testing where there is much consternation that the programmers did not finish the features or that the testers do not understand the system. Consequently a rash of defects end up being raised and late nights and weekend work becomes the order of the day.
• Delivery Delays: Due to the merging issues and the testing issues, the delivery dates slip; and it is difficult even to estimate when the delivery is likely to take place since new bugs keep appearing on bug-fix releases.
• Long Bug-Fixing Cycles: Since the developers and testers need to test all the scenarios manually for each change, the bug fixing cycle is long. This also delays the feedback where the developers are given bugs on changes that they made weeks or months after they moved on to other things.
• Non-reproducible builds: Because the builds (even release builds) are happening in developer machines where the machine configuration is not controlled, it is pretty much impossible to reproduce the same build from the same version of the source code. This causes much heartburn when an emergency fix needs to be made on a released version, but in the rebuild, other unrelated functionalities also are broken.
• Resistance to refactoring: Due to the long testing cycles, and the lack of confidence in the comprehensiveness of the testing, developers become averse to refactoring; while refactoring may make their jobs easier in the long run, the possibly of testing delays and the fear of production outages due to errors in refactoring keep them away from doing it.
• Gradual Performance Degradation: Build by build, release by release, the performance of an application may degrade but goes unnoticed until the performance numbers breach a threshold. But, by this time, it is difficult to identify the exact change that caused the degradation since a bunch of changes have gone in and any one of them might have caused the issue.
• Security Vulnerabilities: Since security vulnerabilities are difficult to test and analyse, when time is short, these tests get the chop first. This leaves the application open to security breaches.
• “Works For Me” Syndrome: Code that works in one environment mysteriously fails in another environment. And because the environments are dissimilar, the programmers are at a loss as to how to reproduce and debug the failure.
• Non-reproducible servers: The production servers have been so long configured by hand that no operations folks have the confidence that they can replicate the servers in other locations or data centres.xtended automation, to meet and transcend customer expectation

About the Author

Krishna has over 25 years of experience in the software industry including several start-up companies in the networking, search and big data space as well as in IT-services and web companies. His prior experience includes web search, big data and cloud infrastructures. His current responsibilities include leading the adoption of automation technologies both in the work that Hexaware performs for its customers as well as in the IT and business processes of the customers. He also leads Hexaware’s investments in the next generation of technologies which promises to transform the high tech industry landscape.

Krishna holds a degree in electrical engineering from IIT Madras. He is based out of our Chennai, India office.

Krishna Kumar

Read this case study to learn how a US-based Healthcare Firm Leverages Agile DevOps for Operational Efficiency, Innovation, Automation and Better ROI. Read more

Read this case study to understand how a US-based Provider of Specialized Payment Products & Services Leverages Agile DevOps for ASM & Development.Read more

For more insights please feel free to connect with us on marketing@hexaware.com.

Visit related articles