Sunday, February 21, 2010

How I learned to stop worrying and love Unit Testing

I admit it. Throughout my whole career at Microsoft, even as a Dev Lead, I was not a true believer in Unit Testing. That's not to say I didn't write unit tests or require my team to write tests. But I didn't believe that the benefits reaped from unit testing were sufficiently valuable given the time it took to write them (for me, about equal to the time to implement the product code itself).

Now, post-Microsoft, I am a true believer. A zealot even. I can't imagine a world in which I write code that has a ton of unit tests covering it.

So what changed? My eyes have been opened to a development world in which real testing infrastructure exists. In my former role, what I used was a testing framework known as Tux, which ships with Windows CE. It was enhanced for Windows Mobile and given a usable GUI. The result was something like JUnit, eg, a simple framework for defining test groups and specifying setup/teardown functions. The GUI was very much like the NUnit GUI.

So far, so good. There's nothing wrong with this setup. However, a test-running framework is necessary but not sufficient for unit testing. The missing piece was a mocking infrastructure.

One of the most frustrating things about working for Microsoft (and I'm sure the same is true of other big software firms) was that everything, and I do mean everything, had to be developed in-house. For legal reasons we couldn't even look at solutions available in the open source community. The predictable result is that a massive amount of effort is expended to duplicate functionality that already exists elsewhere. In many cases the reality of product schedules and resource constraints mean that we simply must do without certain functionality entirely. This was the case with mocking. Developers were left to create their own mocks manually, or figure out how to write a test without using mocks. I identified the lack of a mocking infrastructure as a major problem, but failed to do anything about it.

Exeunt Gabe stage-left from Microsoft to Kikini and a world of open source.

At Kikini we use JUnit for running tests and a simply beautiful component called Mockito for mocking. I cannot emphasize enough how wonderful Mockito is. Mockito uses Reflection to allow you to mock any class or interface with incredible simplicity:

MyClass myInstance = mock(MyClass.class);

Done. The mocked instance implements all public methods with smart return values, such as false for booleans, empty Collections for Collections, and null for Objects. Specifying a return value for a specific call is trivial:


The semantics are so beautiful that I am certain that readers who have never heard of Mockito or perhaps have never even used a mocking infrastructure can understand what is happening here. When the method myMethod() is invoked on the mock, and the parameter is "expected_parameter", then the String "mocked_result" is returned. The only thing which may not be completely obvious is the eq(), which means that the parameter must .equals() the given value. The default rules still apply so that if a parameter other than "expected_parameter" is given, the default null is returned.

Verifying an interaction took place on a mock is just as trivial:


If the method myMethod() was not invoked with "expected_parameter", an exception is thrown and the test fails. Otherwise, it continues.

Sharp-eyed readers will note that the functionality described so far requires that equals() be properly implemented, and when dealing with external classes this is sometimes not the case. What then? Let's suppose we have an external class UglyExternal, it has a method complexStuff(ComplexParameter param), and ComplexParameter does not implement equals(). Are we out of luck? Nope.

UglyExternal external = mock(UglyExternal.class);
MyClass myInstance = new MyClass(external);
ArgumentCaptor<ComplexParameter> arg = ArgumentCaptor.forClass(ComplexParameter.class);
ComplexParameter actual = arg.getValue();
// perform validation on actual

This is really awesome. We're able to capture the arguments given to mocks and run whatever validation we like on the captured argument.

Now let's get even fancier. Let's say we have an external component that does work as a side-effect of a function call rather than a return value. A common example would be a callback. Let's say we're using an API like this:

public interface ItemListener {
    public void itemAvailable(String item);

public class ExternalClass {
    public void doStuff(ItemListener listener) {
        // do work and call listener.itemAvailable()

Now in the course of doing its job, our class MyClass will provide itself as a callback to ExternalClass. How can we mock the interaction of ExternalClass with MyClass?

ExternalClass external = mock(ExternalClass.class);
doAnswer(new Answer() {
    public Object answer(InvocationOnMock invocation) throws Throwable {
        Object[] args = invocation.getArguments();
        ItemListener listener = (ItemListener)args[0];
        return null;

We use the concept of an Answer, which allows us to write code to mock the behavior of ExternalClass.doStuff(). In this case we've made it so that any time ExternalClass.doStuff() is called, it will invoke ItemListener.itemAvailable("callbackResult1").

There is even more functionality to Mockito, but in the course of writing hundreds of tests in the past 9 months I have never had to employ any more advanced functionality. I would say that only 1% of tests require the fancy Answer mechanism, about 5% require using argument capturing, and the remainder can be done with the simple when/verify functionality.

The truly wonderful thing, and the point of my writing this blog entry, is that a mocking infrastructure like Mockito enables me to write effective unit tests very quickly. I would say that I spend 25% or less of my development time writing tests. Yet with this small time investment I have a product code to test code ratio of 1.15, which means I write almost as much test code as product code.

Even more important, the product code I write is perforce highly componentized and heavily leverages dependency injection and inversion of control, principals which are well-known to improve flexibility and maintainability. With a powerful mocking infrastructure it becomes very easy and in fact natural to write small classes with a focused purpose, as their functionality can be easily mocked (and therefore ignored) when testing higher-level classes. I have always been told that writing for testability can make your product code better, but I never really understood that until I had the right testing infrastructure to take advantage of.

Now, I'm a believer.

Sunday, February 7, 2010

A Taxonomy of Software Developers

After spending years of my previous life at Microsoft as a Dev, Tech Lead, and Dev Lead, I've worked with a broad range of software developers from the US, China, India, and all over the world. I've also been involved in interviewing well over a hundred candidates, and many hiring (and some firing) decisions. From this I've come up a taxonomy describing the characteristics of the various software developers I've encountered, how to spot them, and what to do with them.

Typical Developers

The hallmark of a Typical Developer is a relatively narrow approach to problem solving. When fixing a bug, they concentrate on their immediate task with little regard to the larger project. When they declare the bug fixed, what that means is that the exact repro steps in the bug will no longer repro the issue. However, frequently in fixing the issue described in the bug, they have missed a larger root cause, or have broken something else in the system. This is illustrated in Fig. 1:

In most cases the code a Typical Developer writes is a very small net improvement for the overall project when viewed from a release management perspective. Sometimes the traction is zero if the issue that they created is just as severe as the issue they fixed. Sometimes the traction is slightly positive if the issue they created or the case they missed is easier to fix than the original issue.

When viewed from an engineering management perspective, however, the picture is very different. This is due to the nature of the approach Typical Developers take when actually writing code. A typical bug has the form "under condition X, the project behaves as Y, when it should behave as Z." The Typical Developer is very likely to fix the problem in this way:

// adding parameter isX to handle a special case
void doBehavior(boolean isX) {
  // usually we want to do Y, but in this special case we should do Z.
  if (isX == true) {
  } else {

The Typical Developer simply figures out how to directly apply logic to the code that determines behavior, then make the code behave differently based on that. This is reasonable, but if it's the only way the developer can think of to change behavior, after a while working in the same code it begins to look something like this:

void doBehavior(boolean alternate, String data, File output, Enum enum) {
  if (enum == STATE_A) {
    doBehaviorA(data, alternate);
  } else if (enum == STATE_B && !(alternate || data == null)) {
  } else {
    switch(enum) {
      case STATE_B:
      case STATE_D:
        doBehaviorA(data, !alternate);
        // FALLTHROUGH!
      case STATE_C:
        if (alternate) {
          doBehavior(!alternate, null, null, enum);
        // We should never get here!

When I see code after months of a Typical Developer working on it, this is my reaction:

The Typical Developer will never take a step back and think "Hmm, we're getting a lot of these kinds of issues. Maybe the structure of our code is wrong, and we should refactor it to accommodate all the known requirements and make it easier to changes."

Now the project is in trouble. The team may be able to release the current version (often there is no alternative) after exhaustive manual testing, but the team can never be confident that they fully tested all the scenarios. The first priority after releasing will be to remove all the code written by the Typical Developer and write it from scratch.

Another characteristic of Typical Developers is insufficient testing. Often the code they write will be difficult or impossible to unit test. If unit testing is a requirement, they'll write tests which are just as bad as their code. In other words the tests will be unreliable, require big changes to get passing when a small code change is made, and not test anything important. Furthermore the same narrow approach to development shows through in manual testing. The Typical Developer will follow the steps in the bug when testing their fix, and never stop to think "what other behavior could be impacted by my change?"

Typical Developers are quite willing to chalk up their constant regressions and low quality to factors like "I'm working in legacy code" or "I'm not familiar with this area" or "the tools aren't good enough." Though all of those things may be true, that is the nature of software development, and Typical Developers don't understand how to change their environment for the better.

The root cause behind these failings is most often that the Typical Developer is simply not cut out for real software development. Because the software industry is so deeply in need of talent, no matter how marginal, Typical Developers will always find work. Hiring managers are too willing to fill manpower gaps in order to ship on time. (In fairness, Microsoft managers are pretty good about avoiding this pitfall. However, there are times when it is considered OK to "take a bet" on a marginal candidate.)

A special type of Typical Developer is the brilliant person who simply doesn't care enough. They're in software development because it pays well and they can skate by with putting in 40hrs a week. These Typical Developers are especially annoying because they'll employ their brilliance only when justifying their lazy workarounds, and not on actual design and implementation.

What should managers do with Typical Developers? In most cases manage them out as quickly as they can. Though a Typical Developer may be of use in the final push of releasing a project, in the long run having them working on a project is a net negative. Even if Typical Developers came for free, I wouldn't hire them. It is exceedingly rare for a Typical Developer to become a Good Developer, though in rare circumstances I've seen it happen under the guidance of Great Managers.

Good Developers

Good Developers fix bugs and deliver features on time, tested, and adaptable to future requirements. This is illustrated in Fig. 2:

Once a Good Developer delivers a bugfix or feature, typically that's the last you hear of it. A Good Developer will not fall into the traps that a Typical Developer does. When they see a pattern emerging they identify it and take steps to solve the issue once and for all. They are not afraid of refactoring. They'll come into your office and say "Hey, it's not sustainable to do all these one-off fixes for this class of issue. I'm going to need a week to re-do the whole thing so we never have to worry about it again." And you say great, please do it!

Good Developers will encounter the same environmental issues Typical Developers do, eg, legacy code, or weak tools. Good Developers will not let this stand. They'll realize that if a tool is not good enough to do a job, then they have to improve the tool or build a new tool. Once they've done that, then they'll get back to work on the original problem.

Good Developers are Good Testers. Their code is written to be testable, and because they are able to take a larger view, they have a good idea of the impact of their changes and how they should be tested. Pride is also a factor here. Good Developers would be embarrassed and shamed if they delivered something that wasn't stable.

From a release management perspective, Good Developers are well liked, though their perceived throughput may not be high since they are spending time making the system as a whole better and not just fixing a bug as fast as they possibly can. Good managers recognize and nurture this. Bad managers push them to put in the quick fix and deal with the engineering consequences in-between releases. Good Developers will protest against this but often acquiesce. A Good Developer in the hands of a Good Manager can turn into a Great Developer.

Managers should work hard to keep Good Developers since they're so hard to find and hire. That does not mean forcing them to remain on the team, as doing so risks turning a Good Developer into the "brilliant" variety of Typical Developer described above. Reward Good Developers well and give them interesting things to work on.

Great Developers

Exceedingly rare, the hallmark of the Great Developer is the ability to solve problems you didn't know you had. This is illustrated in Fig. 3:

When tasked with work, a Great Developer will take a holistic view of their task and the project they're working on along with full cognizance of the priorities upper management has for this release and the next. A Great Developer will understand the impact of a feature while it's still in the spec-writing phase and point out factors the designers, PMs, and managers hadn't thought of.

When designing and implementing a feature, a Great Developer will take the time to design in solutions to problems that Good Developers and Typical Developers have run into, even though they're not obviously connected. A solution from a Great Developer will often change how a number of components work and interact, solving a whole swath of problems at a stroke.

Similar to Good Developers, a Great Developer will never let lack of tools support or unfamiliar code deter them. But they'll also re-engineer the tools and legacy environment to such a degree that they create something valuable not only to themselves but to many others as well.

Unlike Good Developers, a Great Developer can almost never be coerced into compromising long-term quality for expediency. They'll either tell you flat out "no, we need more time, period" or they'll grumble and come in on the weekend to implement the real fix themselves.

Sometimes mistaken for a Great Developer is the Good Developer in Disguise. These Good Developers have recognized the impact on others that a Great Developer has, and seek to emulate that by engaging almost exclusively in side projects related to tools improvement and "developer efficiency" initiatives. The Good Developer in Disguise has no actual time to do their own work, but fools management into believing that they're Great Developers. Truly Great Developers improve their environment as a mere side effect of them doing their own job the way they think it ought to be done.

It goes without saying that Great Developers should be even more jealously guarded than Good Developers, with the same caveat about not turning them into prisoners. The flip side is that Great Developers should not be allowed to go completely off on their own into the wilderness. No doubt they will build something amazing, but it runs the risk of being something amazing that you don't need. Better to give broad, high-level goals and let them do their thing.

Final Note

Although I named Typical Developers "typical," I mean that they're typical in terms of the overall industry. Although there were enough Typical Developers at Microsoft, most fell into the Good Developer category.