Delivering software is a challenge. The complexity of getting the job done doesn’t really change whether you are building an internal system for your employer, or building a cloud service or an application that’s packaged and distributed. Software development is both an art and a science and how software gets built can range from the extraordinarily disciplined engineering approach NASA takes to kids with no formal training quickly slapping something together and simply launching it on the world to see what happens.

Every developer and every team is different and is constantly striking a balance between cost, planning, implementing, deploying, and maintaining.

Some teams place heavy emphasis on up-front planning and preach piss poor planning produces piss poor products - i.e. the “seven P’s” (or eight P’s as I like to say when they get dogmatic about it). Other teams stress working software over documentation and collaboration and interaction over formal processes and contract negotiations.

I’ve been professionally building software for just over three decades. The biggest questions on every endeavor have always been:

1. Will it work?
2. How much will it cost?
3. Is it done, yet?

Over the years, I have seen teams of every size from 2 to over 500 grapple with defining the scope of the deliverables, the time to build it in, and the cost to produce it all. There is one book I read when I was nineteen that, 45 years since its publication, is still remarkably spot on - The Mythical Man-month by Fred Brooks. In particular, there are five points the author makes that describes the vast majority of software teams remarkably well:

First, our techniques of estimating are poorly developed. More seriously, they reflect an unvoiced assumption which is quite untrue, i.e., that all will go well.

Second, our estimating techniques fallaciously confuse effort with progress, hiding the assumption that men and months are interchangeable.

Third, because we are uncertain of our estimates, software managers often lack the courteous stubbornness of Antoine's chef.

Fourth, schedule progress is poorly monitored. Techniques proven and routine in other engineering disciplines are considered radical innovations in software engineering.

Fifth, when schedule slippage is recognized, the natural (and traditional) response is to add manpower. Like dousing a fire with gasoline, this makes matters worse, much worse. More fire requires more gasoline, and thus begins a regenerative cycle which ends in disaster.

My own observations in tune with the above:

1. Developers are notoriously optimistic. Because of this trait, they build things many others thought not possible or reasonable. Because of this trait, they often fail at accurately estimating effort to build.

2. Developers’ estimating technique generally suffer because they’re basically just guessing most of the time. It is challenging and mentally exhausting to drill down in enough detail up-front to perform accurate estimates. Its also expensive. So, virtually every team seeks to strike some sort of balance with tolerance for schedule slippage, and cost of being more accurate with schedules.

3. Software managers are almost always developers promoted to rank. They bring their inherent optimism with them, and more often than not, they try to assuage their bosses’ mandates to get things done quickly by agreeing to schedules before asking their development team “how much effort?”

4. Developer oriented metrics are more often misleading than truly reflective of progress vs. effort involved. Rarely do they lead to a direct and accurate insight into the overall status of a project, let alone provide an accurate measurement of each developer’s contribution to the whole.

5. Schedule slippage is still, by and large, solved by either working more hours or by adding more bodies to the effort. If not careful, the former quickly becomes habit and consequentially, leads to high burnout and turnover rates. We developers call it, “The Deathmarch.” Adding more bodies slows things down due to the overhead of extra communication and training, collaboration to bring new members up to speed. Adding more bodies rapidly is worse than adding more bodies at a measured pace.

Now in my third decade as a technologist, I reflect upon all the different ways I tried to make the whole process pleasant, predictable, and valuable, not only to myself but to every stakeholder I have ever produced software for. What follows is an attempt to distill some of that wisdom.

Estimating Effort

Estimations are the bane of all software delivery teams. Quite simply and in direct support of Brooks’ first point: We are optimistic to a fault when it comes to producing estimates. I once had a manager in my 20’s that said I was great at estimating, but never right. No matter what I quoted to him, he always multiplied by four and usually nailed the deadline. I tried doubling and tripling my estimates and he still multiplied by four! The sad thing, at least for me, I still missed by a factor of four and he remained right.

Ever since then, I have endeavored to find a way to improve my estimating skills as a software developer. I’m here to tell you that after decades of refining and improving, I’m still not accurately nailing estimates. Certainly, I’ve gotten far better and this is what I’ve learned over the years: The only time I get estimates right better than 90% of the time is for tasks that take 2 hours or less to do. I’m reasonably consistent for those that take 4 to 6 hours (about 80% of the time). Anything over that, it becomes gut-feel and definitely influenced by my abundant optimism.

The conclusion: If a client absolutely must know time to delivery and bank on the estimate, then no task in the entire scope of effort can be estimated to be more than 4 hours, and the vast majority of them better be 2 hours or less. This constraint especially applies to all scope that is under the critical path.

Unfortunately, this level of detail up-front can kill many projects before they even get started. It is a lot of work to estimate to this level of detail. Not only on the developer’s shoulder, who must do the work, but on the project manager’s shoulder who must constantly translate the constraints of the technical against the desires of the business and negotiate to an acceptable schedule and scope of effort across the board.

The compromise: Don’t estimate 100% of the effort to this level of detail, but do estimate enough to get within striking distance of 80% confidence of the estimates projected. And that means, in practicality, driving to the point that the critical path work is fully to this detail level, while fringe scope tasks are at least under 12 hour guesses (two work days). If a large majority of the critical path tasks are under four hours, we have a reasonable shot at nailing the intended deadlines, yet reducing overall estimating effort up-front to 20% of the effort to prepare a fully drilled down estimate.

The key with your client is to be clear just how confident you are about the estimates and why at all times. In short, you’re engaged in a sales/negotiation effort with clear parameters to set expectations against which to manage discussions of schedule, cost, and ongoing effort in a way that is generally positive all the way down the road.

Measuring One Developer Against Another

• How many lines of code written by each developer last month?
• How many commits per day per developer?
• How many bugs squashed per week per developer?
• Which developer has the highest and lowest sprint velocity?
• Who delivered the most features and fewest features?

Sound familiar? How anyone can truly measure progress or effectiveness of the team using metrics like these is beyond me. I’ve tried ‘em all over the years. It doesn’t work. It doesn’t even matter. Who cares if developer A cranks 500 lines of code a day, while developer B produces only 100? I’m notoriously famous for removing 1,000’s of lines of code in a project in a day. In fact, I once joined a team and turned a 300K lines of code project into a 40K lines of code project with quadrupled the features in six month’s time. How would you measure me with such a metric? That was the year the company threw away lines of code written as a way to identify top developers and award bonuses.

Measuring, analyzing and reporting on these sort of stats does one thing very, very well: It kills time - time that could otherwise be channeled into coding, coaching, collaboration, defining and refining what should and should not be built - Any number of activities that can directly boost output of deliverables.

While on the topic of individual developer performance, Let’s talk about that rock star developer on your team of six (and yeah, I get it, nobody actually likes this term, but everybody knows what it means…). Let’s say this rock star is shouldering 70% of the entire team’s progress. If you analyze developer output and attempt to use this as a means of motivation, all you’re really doing is demotivating your other 5 developers with a constant beating because they’re not on the same pace as your rock star. No amount of accounting, portraying, and badgering the 5 is going to turn them into rock stars.

You are better off focusing on the positive reinforcement (i.e. coaching, mentoring, code reviews as a learning experience), and just plain relentlessly targeting getting stuff done. There is only one process I know of that consistently raises (or weeds) out under-performing developers, and that is constant, critical, and constructively fair feedback delivered through code review sessions. Recognize that the rock star, if properly managed and coached, will raise the tide waters and thus all of the boats in the harbor. The rock stars that are team players are the ones you want to keep around as long as you can, but the prima donnas (a.k.a. maverick developers) usually do more harm to the team than good. These developers should be removed from the team and assigned to clearly siloed projects to neutralize their negative impact on a team’s performance, yet still contribute solid value to the business.

If developer metrics doesn’t cut to the chase, what does? To date, the most efficient way I’ve found is to focus on breaking down effort by features to deliver and use a point system (not hours). Then ask each developer to score a user story or task’s difficulty based on the point system. Lastly, make sure every bit of work done is backed by a story that has such an estimate on it. If you do this, then every developer’s estimate is relative to his point of view, yet consistently applied and you can chart two things for your team: velocity and volatility.

If you don’t have a steady velocity and low volatility, examine your processes, stories, and individual developer scoring and determine where the inconsistency is introduced. From there, refine your processes to produce stories easier to estimate on or coach your developer one-on-one to improve his estimating skills.

Focusing on features to deliver allows you to have a consistent dialog in both directions: business need and technical how-to. If a developer scores a task complexity too high, we decompose it further and estimate again.

At the end of the day, the client just wants to know the feature was built, delivered, and works as intended and your team’s already moved on to the next challenge. So a software team that is oriented around what features it is working on is very naturally aligned with the business needs that are driving the development.

Getting it Done

To get it done, you have to know what you are building. To know what you’re building, you have to collaborate with your client and all other stakeholders, including the developers on your team, who will do the work.

There are two ways to get stuff done:

Brute force it. Whatever the client requests, build it and deliver. Unfortunately, this is the most expensive way, too and you may find yourself with a big ball of mud before you know it. Build the simplest solution first and every time. You can only do that by relentlessly narrowing down and fleshing out the scope until what’s needed, and only what’s needed, is defined, estimated and easily understood by everyone involved.

When we first take on a new client, it is a monumental challenge. Why? Because the client and our team doesn’t yet understand each other. Our vocabulary is not established. Our discussions are prone to meandering with many tangents. Fundamentally, we don’t yet know what our clients really want and our client doesn’t know what we’re fully capable of doing. In short, there’s a formative period in which we’re discovering each other.

This is also the period of time that can tend to exhaust my new clients with so many questions he or she wasn’t quite prepared to answer. But what’s crucial here is that I slowly, but surely build up a very clear mental model of what the client is trying to accomplish, at both the macro and micro level. By fully understanding my client’s business at all levels, I set my team up for redefining or recasting many related feature requests into fewer, yet very synergistic and cohesive feature requests. Not only that, I find that the client is suddenly very clear on what he wants to accomplish and almost always knows the order in which he wants things delivered. We have begun to establish a mutual trust and that does wonders towards a smoothly running development effort. The art of listening to your client involves listening as much for what isn’t said as much as what is said. Often a client doesn’t vocalize something because he’s afraid of the scope and cost to build, afraid he’ll be judged as a fool with his head in the clouds, or simply doesn’t know it’s possible, yet he leaves clues all around that there’s a bigger task or desire behind the stated requests.

If you’re keen and pick up the clues and put the puzzle together for your client, then you’ll almost definitely find the project and individual scopes self-organizing and that, in turn, drives you straight to the most comprehensive, yet simplest approach to solving the challenges at hand. All too often, software projects get into trouble halfway through because there wasn’t enough understanding of the big picture at the outset and the developers set building a solution that effectively paints the team into an inescapable corner revealed by the next phase of planning where new revelations previously unvoiced completely does not jive with the initial approach just built. Another common trap business visionaries fall into when describing a solution they want is to describe grandiose systems with tons of features nobody really wants or will use once built. The discussion gets so lost in the inconsequential, that the way from point A to B is muddled until much, much later, and after thousands of dollars spent.

Keep questioning and rephrasing the stated scenarios until both parties completely understand how every part of the system is going to fit together. Only then does the discussion turn to decomposing into user stories and estimates and developers assigned to the tasks. The user story decomposition also serves as final drill-down into the details of the solution and may lead you back to the drawing board for another round of hashing out details.

Have the Right Tools

When it comes to choosing the tools the team uses, look no further than Joel Spolsky’s “The Joel Test: 12 Steps to Better Code” If you have these principles down, then the rest is just deciding the specific tools, the mix of emphasis, and refining how your team works together.

At the time of this writing, we manage approximately three dozen separate projects with the help of just a handful of developers. Needless to say, its enough to keep me busy year around. 90% of my time is spent on the software development effort, which, in my definition, includes: coding, researching, tinkering, learning, code reviews, and mentoring the developers on my team. If I’m looking at code, reading about code, discussing code, or cranking code, I’m developing and I consider software development the most productive and enjoyable use of my time, skill, and talent. The other 10% is spent on collaboration with the client whereby we discuss new features and scope (a.k.a. budget), estimated delivery time-frames, documenting new features and, lastly, accounting, invoicing, and paying taxes.

We relentlessly work to shed every once of non-productive effort we possibly can and continue to refine our team’s processes over time. Every great software team does this. No one process can work for every team. There are great aspects to each of the various processes that have come along over the years, but the one thing I rarely hear discussed when talking about processes is how to truly maximize developer output - the very thing almost always discussed with developer-oriented or code metrics.

Processes do pay some homage to developer productivity, but by and large, they are concerned with solving the failure to ship software on time and within budget. Some processes are meant to drive down the number of defects introduced. Some are meant to improve accuracy of delivery schedules. Yet others are designed to reduce effort of documentation and writing specifications.

How We Are Delivering Software Today

The process we are using today is meant to maximize our ability to meet the client’s needs in our role as an external co-product owner. When a client comes calling, he almost always is bringing a new feature request, and he needs it done in a timely fashion to coincide with a change he’s making in his business. We have to be able to work a client into our daily grind at a moment’s notice. That means we have to figure out where we’re ahead with some clients and where we can pull resources and then negotiate an agreeable deadline. Some projects sit dormant for 2 or 3 months and then light up for a few hours or weeks of new feature implementations before going quiet again. That means our ramp-up and ramp-down on each project has to be super efficient as well.

The challenge is consistently picking up a project exactly where it was left off and continuing to build on it. For the better part of my career, it was nigh impossible to directly manage more than 2 or 3 projects effectively. Every project was dramatically different. A different programming language. A different architecture. A different DBMS. A different operating system. And so on. When Ruby on Rails first hit the scenes with it’s “convention over configuration” mantra, I began to see the light. After a few years of building Rails applications, I realized that I had very little trouble returning to projects I built 4 or 5 years ago and continuing to add new features to it. At some point, it struck me that the key to my own productivity was the consistency brought from the simple concept of convention over configuration. So the first thing I changed some five years ago was to shed all clients that were not Ruby on Rails based projects.

Going all in on Rails was a key turning point. I didn’t care if it was Rails 1 or Rails 5. Every project had the same basic architecture. Some got very elaborate with meta-programming techniques or other decidedly non-Rails approaches baked in, but by and large, they were Rails apps and that meant something very significant to building our team’s workflow: We could automate nearly everything that had nothing directly to do with programming.

Here are the core tenants of our development methodology:

Slack is command central. All communication with the development team is done in the correct slack channel for the client we’re servicing. Email, instant messaging, skype, etc. have all been shunned for discussing the projects. Screenshots, bug fixes, new feature requests, mockups - everything’s in slack. No more time is wasted trying to find some important tidbit that came across in some other medium and we’ve collectively forgotten where. If a client opts not to participate in the slack channel, then emails with details about the project are consistently dropped into the respective channels so the crucial info is searchable in slack. Generally speaking, emails with the client are limited as much as possible, favoring slack or pivotal tracker for discussing project details.

Testing is automated. All features are backed by both unit tests and integration tests as appropriate. Every developer proves the correctness of his contributions with test specs. I code review, verify specs and push back anything that doesn’t meet the standard. Our team uses rspec for most Rails effort. We don’t use cucumber, but we do incorporate capybara, poltergeist, phantomjs, and selenium for automated GUI testing of various critical components (logins, user registration, etc.). Another reason we don’t use cucumber is because our clients don’t write the feature scenarios and don’t read the ones we wrote, so no effort is wasted here. On the other hand, clients do test the application’s behavior through it’s interface, so we generally waste no time writing automated tests for front-end aspects unless we’re prone to breaking some specific feature.

Deployment is automated. Every app under our management can be fully deployed (and rolled back) with a single command. Passwords are completely removed from the picture. SSH keys are always used in place of passwords to access all the application servers. This may sound odd to mention here, but it is key to not losing time tracking down old passwords and risk being locked out of a system when it is critical to have that access. Every developer can deploy. They may start with only staging/pre-production deploys, but everyone deploys. One person doesn’t need to be the middleman for the entire team when each developer is getting their features out the door.

All project code is hosted in git and new features that may break working branches are started and completed in separate feature branches while simple enhancements are directly committed against the main “develop” branch. Feature branches can have breaking code. Working branches may not. Everything is code reviewed, refined, then finally merged to main working branches for deployment, whether to staging or production.

Every git commit references the user story ID in Pivotal Tracker which trigger the git hooks that update the user stories on the developer’s behalf, setting different statuses based on the branch committed or merged. One less thing for developers to do.

Every commit is recorded in the slack channel for the project, also triggered by git-hooks installed for each repository.

Every deployment is recorded in the slack channel for the project and automated through the slackistrano gem.

Every project carries the exception-notification gem which is configured to notify the slack channel that an error occurred in the application. At some point, we’ll likely implement automatically adding a bug ticket to Pivotal Tracker, but for now, the developer that takes the bug claims it in the slack channel then creates the bug in tracker. The developer that claims the bug, fixes and deploys as soon as the issue is fixed.

Every feature request is decomposed to stories of no more than 4 hours effort at the outset unless the client is comfortable with a certain margin of error in the scheduling estimates. In which case, we decompose just enough to estimate to that level of confidence before starting any major new effort. However, once we start building, we continue to decompose the backlog down to no more than 4 hours each before any work actually starts and collaboration with the client is advanced through this effort.

The time spent on every project is fully tracked via Harvest Time Tracking. Time tracking is relatively painless and a force of habit to the developers. Like git commits, time entries reference Tracker stories. This isn’t for automation so much as for informational purposes enabling the client to tie hourly invoicing details back to work done in Tracker.

One thing not really covered in the above is what to do about projects and efforts that span many months. Projects that span this much effort still remain fairly challenging to estimate accurately and schedule reliably, so what I generally try to do is segregate the efforts into “must have,” “nice to have,” and “like to have if everything else is done” tiers. The “must have” features must be doable in three months or less. Think of this as the MVP (minimal viable product), even if bolting on to an already existing application.

We work hard to define the full scope and narrow it down until it is comfortably deliverable in the three month capped time-frame. Everything else is estimated enough to have a rough idea with further discussion on delivering postponed until the first phase is completed. If we’re ahead of schedule, we’ll either go-live ahead of schedule so we can start again, or stay on schedule and broaden the scope to the “nice to have” features.

Tracker becomes extremely important here, because everything included in the MVP lands in the backlog while everything else lands in the icebox. In effect, the scope’s fully agreed upon in a indisputable fashion, thus eliminating scope creep from the client and misconceptions about what’s included and what’s not in an upcoming release cycle.

Closing Thoughts

The thing I love about software development is that the challenges are always fresh and new and the technology field is constantly evolving. In a mere 70 years’ time, we’ve gone from full-basement filled computers programmable only by pulling wires, flipping switches, or feeding punch cards to powerful portable computers that easily eclipses the power of those behemoth machines. The art and science of writing software has evolved tremendously and continues to build at an accelerating pace. Perhaps the most wonderful aspect of software development is that anyone with the desire can pick up a computer and start programming today. But when it comes to delivering on a professional level, it requires nothing more than imagination coupled with discipline and perseverance. Nowhere in humankind’s history has a simple 1 or 0 binary decision at lightning pace changed society as a whole in so short a time.

In ten years time, I guarantee you I will have a far better way to build software than as described above, but today, it working extremely well, and it will necessarily continue to evolve as the years come and go. Your methodologies and processes should, too.