100% code coverage might not be enough


I was recently working on a project where code coverage slipped and we needed to put some effort to catch up by writing few unit tests. During the process, we came across an interesting case which proved us that having a 100% code coverage was not  a sufficient criteria to stop writing unit tests.

Here is the method we were trying to cover:

public Status RetrieveInitialStatusForTicketType(RetrieveInitialStatusForTicketTypeRequest request)
        {
            try
            {
                var userProfile = base.CheckAuthorizationAndReturnValidUserProfile(request);

                var statusesWorkflow = StatusesRepository.RetrieveStatusesWorkflowForTicketType(userProfile, request.TicketTypeId);

                if (statusesWorkflow == null || statusesWorkflow.Where(s => s.IsInitialStatus).Count() != 1)
                    throw new Exception("Must have one and only one initial status");

                var initialStatuses = statusesWorkflow.Where(s => s.IsInitialStatus);
                Status status = initialStatuses.First();
                Status returnedStatus = (Status)status.Clone();
                returnedStatus.PossibleTransitions = null;
                return returnedStatus;
            }
            catch (SecurityException) { throw; }
            catch (InvalidTokenException) { throw; }
            catch (Exception e) { throw new InternalErrorException(e); }
        }

The whole method was covered except the line 10:

throw new Exception("Must have one and only one initial status");

and line 20 that catches the exception.

So we added a new unit test that expected an exception and in which we mocked the StatusesRepository (line 07) to return a null value. We ran the code coverage analyzer and, yeah!!! 100% of the code was covered. We thought we were done… but wait a minute!!!!

The unit testing is not about testing per se, but rather about validating the requirements implemented by a method. In other words, all the requirements the method implements must be validated by one or more unit tests.

However, by looking at our code, we clearly see, at line 09, that the returned workflow must contain one and only one status which flag “IsInitialStatus” is set to true. However, none of our tests was checking for this rule to be implemented although the code was 100% covered.

We decided to write a new unit test in which the workflow returned by the StatusesRepository (line 07) contains more than one initial status and another in which the returned workflow contains no status marked as the initial status. In both cases, the unit test expects and exception to be thrown. Only then will our code be really 100% covered functionaly speaking.

Along the way, we have split the condition of the line 09 into two different if statements in an attempt to make the code more comprehensive (lines 09 and 12).


public Status RetrieveInitialStatusForTicketType(RetrieveInitialStatusForTicketTypeRequest request)
        {
            try
            {
                var userProfile = base.CheckAuthorizationAndReturnValidUserProfile(request);

                var statusesWorkflow = StatusesRepository.RetrieveStatusesWorkflowForTicketType(userProfile, request.TicketTypeId);

                if (statusesWorkflow == null)
                    throw new Exception("No statuses workflow is defined for the specified ticket type id.");

                if (statusesWorkflow.Where(s => s.IsInitialStatus).Count() != 1)
                    throw new Exception("Must have one and only one initial status");

                var initialStatuses = statusesWorkflow.Where(s => s.IsInitialStatus);
                Status status = initialStatuses.First();
                Status returnedStatus = (Status)status.Clone();
                returnedStatus.PossibleTransitions = null;
                return returnedStatus;
            }
            catch (SecurityException) { throw; }
            catch (InvalidTokenException) { throw; }
            catch (Exception e) { throw new InternalErrorException(e); }
        }

Splitting the condition into two separate conditions had two advantages:

1 – Each condition correpsonds to a particular business rule. Line 09 means that a ticket type MUST be associated with a workflow and line 12 expresses the fact that a workflow must have only and only one initial status. It it now clearer that we do have two business rules and that the condition workflow == null was not a simple check to avoid null object exceptions.
2 – Since the two conditions are separated, we could throw a more specific exception in each case that explains more precisely the reason why it has been thrown (which is very helpful when debugging an application).

Conclusion

While writing unit tests, being in TDD or in TTCUBWWL (tests to catch up because we were lazy – to write them before the code -), we must pay attention to the fact that a method must be covered by unit tests that validate  each implemented requirement in the method and bear in mind that reaching a 100% code coverage of a method is not always a sufficient criteria to stop writing tests for the method.

My advise for the team was that they should write their tests before the code; first to avoid another cathing up effort but more importantly, if the developer wrote the code using TDD, he would certainly not miss the unit test to check the method behavior when a workflow has more than one initial status. But this is another debate: the efficiency of TDD in writing clean and more comprehensive code.