Since I started my career in software development, I’ve witnessed many changes in how we release software.

My first job was as a technical consultant at a large ERP software vendor customizing their Student Administration package for a large state university. I worked on several things* on that project, and at one point, I was in charge of change management from the Dev to Test system environments. This was the process:

* On a weekly basis, email all the developers on the team, asking them for their finished code projects.
* Create a spreadsheet listing the complete set of projects to be migrated.
* Get various approvals from the team leads for the final migration list.
* Manually migrate each project from the Dev database to the Test database.
* Finally, let the testers know they’ve got new stuff to test.

Waterfall can best be described as a “walled-garden” methodology. Each function — development, test, and operations — mostly worked in silos. Interactions between teams only happened occasionally — weeks at best, months at worst. And even then, only a person with people skills, like me, handled the interaction between the teams.

This was the peak of waterfall software development. A complete code cycle from initial development to production only happened a few times a year. (“Once per quarter” was a pretty aggressive release goal for ERP projects in those days.) No one thought this was a “bad” process. Tedious, yes, but not terrible.

A few years later, I found myself working for a hip, boutique software consultancy in Chicago. I became the technical lead on an in-house project to replace the company’s time and expense system. This company was really into XP and Agile, so we “ate our own dog food,” and ran the project The Agile Way, the same way we ran our client projects. We had weekly iterations, and pushed to production about once a month. Initial project kick-off to first production users was about 3 months.

The big innovation I noticed with Agile was the distinct feeling that development no longer lived in a “walled garden” with regards to testing. The wall that traditionally existed between development and testing no longer existed. In fact, every developer wrote tests. We even wrote cross browser testing tools to make the task easier for the development team. And we used continuous integration to make sure our tests were run all the time.

We were good-to-above-average for a typical agile project. We only pushed to production once a month, but we were proud of ourselves. During this whole time, no one thought this was a bad process. In fact, general consensus was that it was good, bordering on great — especially compared to the Waterfall horror stories everyone knew of.

More recently, I co-founded a startup focused on solving software testing problems. As a small startup with a new Amazon EC2 account, cloud computing provided us direct and fast access for deploying code to production whenever we wanted.

With no more long cycles waiting for IT to procure machines, and no lengthy approval processes for dealing with production DBAs, I could hear another wall collapsing. We had obliterated the line between dev/test and operations. Some people call this “DevOps.” In my opinion, it should be called “DevTestOps,” because no sane person would push to production without running tests first!

At first, even though we now had little process friction to go from dev to test to production, we still did things the Agile Way. We coded features at a weekly or biweekly scale, and pushed to production once a week or every two weeks. This was still an improvement over my first Agile project, though.

With the walls down between development, test, and operations, we’ve started to optimize for speed. As we strive to improve our process, it helps to see pioneers show the way.

We now release at a much faster pace than a typical Agile project. Our internal goal is to *always* push to production at least once a day. We often do better than that. The faster you go, the more motivated you are to go even faster.

One benefit of fast release cycles is that it frees up time to tear down more walls in the development process. We have production metrics to watch user behavior in production — what they do and what they click on — that can inform what we work on next week. And we even have time to talk to the sales team for input on what to work on. Wherever we can, we destroy friction in our development process.

We’ve noticed that as our release cycle speeds up, so does our reliance on fast feedback systems. Our tests and our continuous integration system is the heartbeat of development. If the build is green, we deploy. If not, we have to wait. When you get used to pushing new code to production several times a day, it’s really painful to go back to the old way of doing it. After every process improvement, I look back at the old way we used to do things, and wonder what took us so long to get here.

Footnote:
* My first assignment was writing the program for the Admissions department that had a simple workflow. Input: GPA and ACT/SAT score. Output: a “personalized” (mail-merged) acceptance or rejection letter. I still remember thinking how a bug in my code would alter the course of someone’s life. If you didn’t get into Boise State University around 1999-2000, sorry about that!