As I write this I’m taking a bit of time between jobs. I decided to take a couple of weeks off to, you know… write blog posts and hack on some stuff. I won’t get into my reasons for leaving Breezy but it was an amiable separation. I’ll likely post on this some time down the road.
Working as CTO in a seed stage startup, was probably about the most fun I’ve ever had at any job. Actually, I’m not even sure I’d call it a job. Sure, I got paid, but I was getting to do what I love (build stuff) and hang out with other smart people doing the same. Most of the time it didn’t feel like a job.
I could go on about things I’ve learned over the last year but one thing I found myself thinking about a lot was what engineering practices we should follow. You know, should unit tests be required for all of our products? How do we do QA? What about deployments? etc. These sorts of questions didn’t keep me up at night but they were genuine problems to solve and fun ones at that.
A bit about my background: Ignoring all the random jobs I did before my first engineering gig (pond installations, lifeguard, server, etc.), my “career” in software officially started at Walt Disney World. This place was about as far from a startup as anyone could contrive. Hundreds of engineers, cubicles, change control policies. It defined enterprise. After that I spent some time at Vertigo Software, relatively speaking a much smaller shop with about 50 people. These guys operate very differently and move much more quickly. It was an enlightening adjustment.
Having worked in both large and small environments (and medium-ish) I think I’ve become very pragmatic about engineering practices. What I mean by that is there’s no religion to it. There are no absolutes. I refute any argument that certain engineering practices should always be followed because it’s the “right” way. Frankly, that would be too easy an answer. As with most things in life, pragmatic engineering is about maximizing for the right variable at the right time.
Optimizing for Ship Time
When I started at Breezy, we were a small team. I mean really small, as in just the founder (Jared Hansen) and myself. He was working on scaring up some customers, meanwhile we needed a product people could touch ASAP. I wrote a lot of code those days building out our Rails API, our Windows app, etc. Jared and I both split QA responsibilities and deployments were typically pushed right after the code was committed. It was strategically expensive to do anything that wasn’t product so very few unit tests were written. Also, because it was such a small team things like branching strategies had very little value.
After our MVP (Minimum Viable Product) shipped and our team grew, eventually reaching 5 engineers, a few things changed. When our intern started, his first task was to retro-fit our API with unit tests. This added immediate value and helped us discover a few bugs. More importantly, we were able to refactor with confidence and going forward all new API development happened with unit tests.
Another huge process improvement was the addition of Continuous Integration. Our Windows application had 4 different builds: 2 distributions (Enterprise & Consumer) x 2 environments (PROD & TEST). Our Android application had 3 builds: DEV, Marketplace, and Enterprise. We also had a similar build configuration for Blackberry but unfortunately never got around to setting it up for iOS. Having all of this automated saved huge amounts of time. Furthermore, it decreased the likelihood that human intervention would introduce errors into some build destined for Production.
There were two levers that I saw impacting how much process we implemented. The first was product stage. Early in the game, having something to show interested, potential customers was more important than worrying about how easy it would be to refactor a given piece of code. We were building an MVP and that has a very different set of goals associated with it than a piece of software for a company that has, well… revenue.
The second, and in my opinion more significant one, was team size. I think you reach a point at which, no matter what stage your company or product is at, some process is needed to manage the complexity. Things like branching strategies and standups are very useful for any team larger than 2 engineers for example. You get a greater degree of flexibility if the team members are developing on independent products. However, within the same company, especially at a startup, that’s rarely the case.
Here’s a short list of a few elements that made up our engineering processes. There are several more I could add to the list (standups, backup schedules, etc.) but these stood out as being more significant.
Task/Bug Tracking – I can’t understate the importance of having a solid process in place for tracking on a task list (or backlog). When things kicked off a Breezy I started by using a spreadsheet. We also wrote tons of lists all over our walls. At the time our office walls were covered in IdeaPaint so we had significant amounts of whiteboard square footage! I eventually started using a combination of Pivotal and Asana, which are great solutions to this problem but support slightly different use cases (more on that in another post).
Automated Builds – As I mentioned earlier, setting up a Continuous Integration/Automated Build process was a huge process improvement that saved us significant amounts of time. One downside to CI though is that it’s expensive to setup and maintain, and I mean that from an “engineer-time” perspective. I think our setup was pretty typical, EC2 + Jenkins, but when you factor in environment configuration and maintenance, it adds up. While I would have liked to have had it earlier it was set up about 3 months into the game (post – MVP).
Unit Testing – Few things inspire crusade-like fervor within engineers quite like the debate over unit testing. My humble opinion is that Unit Tests are very useful, but they’re also expensive. For some products/teams they are affordable, however for some they are not. When we were pre-MVP we could not afford to build unit tests and so none were written. They were eventually retro-fitted to our Rails API but we never wrote them for our mobile applications. I’ve been asked if looking back I feel like this was the right decision and I do.
Engineering Your Engineering Team
It seems to me that there are many approaches to building software. In my opinion, different approaches simply seek to optimize different variables. The best engineers I’ve seen are the ones that are able to shift between approaches at the right times. Building a good software product (quickly) is a complex equation and there are no right or wrong ways to do it, only more and less efficient approaches. I think the same goes for building an engineering team.