Build vs Buy For Core Business Tools

Before entering the fray, I need to mention that the argument I put forth here is tailored to a very specific question that happens to have been evaluated a number of times at the company I work for. It relates to the question of whether to buy or build a solution to automate business processes of the company. Moreover, it assumes a competent development team that is currently available. Finally, I am discussing a build vs buy decision in a small company (a hundred or so employees). While this post was motivated by a specific build vs buy decision, I only lay out arguments that are generally applicable here.

Probably the biggest reason management likes the idea of buying software is that it is a quick fix that is available today; they do not have to wait for the solution to be built in-house. This is, however, not entirely correct. The last time we tried to adopt a pre-built solution, it was six months after the purchase date that the first user began to use the software. This is because even though the software is available right away, it takes time for people (including IT) to learn the new system, adapt processes to accommodate the new system (more on this later) and most importantly to trust the system so they abandon their old process.

Time to adoption may be long, and if it is also the case (as it often is) that only a small subsection of the built software is really needed by the business (like a ticketing system), it may be possible and easier on the company to have an agile team release the software slowly to allow the business to adapt instead of switching all at once to a pre-made solution. It may even turn out that the time to build with concurrent adoption is equal to the time to adopt the 3rd-party system.

This leads directly to the question of how much development work will be required in both cases. There is no way anyone can tell me that a pre-built solution will not have to be customized once it is purchased. In fact, a significant amount of customization has been done on every solution we have purchased. Because this software is foreign, this may mean buying customization from the vendor (with all the lack of control that entails) or if you are lucky, customization by your own development team. In the latter case, this customization is not as cheap as customization or general development of the same complexity on an in-house solution because the developers did not infuse the foreign software with their philosophy and quality requirements. And again, the customization cost of the new software may not be significantly different to the development cost of the subsection of functionality that is really required by the business in an in-house solution.

A major reason I am a proponent of agile methodologies is that the business I work for changes requirements almost weekly, depending on the department. This can cause major problems with a pre-built solution. It could even mean constant customization of someone else's product. The flexibility of pre-built solutions is definitely questionable. This means that more often than not, the business ends up adapting to the software, and not the other way around. This leads to the long adoption time I mentioned above. This is even more of an issue if the software relates to the core of the business because the usually over-generalised software is telling the company how they should do business (how to handle support calls, how to have new customers apply, how to pay sales agents, how to sell, etc).

There is also the cost of maintenance to consider. At between 10%-20% of the cost of the software per year, this is not insignificant. The same argument about customization given above applies to maintenance as well. Developers will be more efficient maintaining their own system than someone else's, if that is even a possibility. Sometimes, you are dependent on the vendor. Even assuming they are reliable, they may not be very responsive.

Finally, and perhaps most importantly, you may lose your development team by purchasing software. The best developers do not want to do maintenance; they want to do development. If they are maintaining a purchased solution, you better hope it is high quality and built well, in a modern language (did I just cut out 85% of off-the-shelf software?), because if not, you will have a hard time attracting good developers.

For us, it seemed a no-brainer. We would end up customizing the thing anyway, it would still take 6 months before it would be in use and maintenance would still be a problem. Considering that it may take about two months to rebuild the functionality required into our already-built enterprise management system, I cannot understand why anyone would consider buying an off-the-shelf solution. Yet if we had not reminded the executives of these considerations, I may have been working on a filthy perl application, and probably looking for a new job.

Unit Test Pep Talk

The last two major features we developed at work were built largely without writing unit tests, let alone with a test-driven mentality. In fact, I would occasionally explicitly mention that unit tests should be written for such and such an issue, and it would turn out that the tests were not written. This led, predictably, to a much buggier release than normal, with patches being pushed to production every day for over a week.

Talking this over with my colleague, I felt that this was because while the whole team aknowledges that unit tests are theoretically a good idea, they feel that sometimes it is better to move faster and write less tests. So we decided to run down the reasons why our team unit tests using test-driven development in our iteration de-briefing. Here is what we said. Some of this will certainly not be novel, but bear in mind that these are the practical benefits that we see everyday that cause us to require unit tests in the software we write.

First, the obvious. Writing unit tests reduces bugs and speeds development by actually running your code quickly and checking it against the expected result automatically. I actually saw members of our team doing this manually: running the software and seeing if the code busted.

Probably the most important thing for me is that writing unit tests allow you to refactor safely. Without a solid suite of unit tests, it is impossible to go in to a module with a cleaver and be sure that you are not breaking anything.

Finally, and this was certainly not obvious to our team, the unit tests form a contract for what the code should be doing. If I am wondering what a method's behavior should be (as opposed to what it is), I go and look at the unit tests. They will tell me what is required of that method, and what is not required. I use this all the time for critical system configuration issues. For example, if a value must be 2 on system startup for the sytem to operate correctly, I add a unit test so that if ever anyone decides to change that value, the unit tests will tell them that is not allowed.

Moving on to test-driven development, we mentioned two reasons. The first and most important is that practicing test-driven development ensures adequate test-case coverage. By requiring a failing test before writing code, you ensure that every branch of your code is tested for a desired response. Similar to my first point about unit testing in general, this is simply the automation of something we do anyway: come up with the expected outcome of our code and then write the solution until we arrive at the expected outcome.

More subtly, test-driven development will improve the design of your code. Code that is tightly coupled, or not cohesive is much harder to test than the alternative. By writing tests first, you envision how you would like to use the library, instead of forcing yourself to use whatever interface you came up with after the fact. Because you want to isolate the code you are testing, and minimize the amount of test code you write, test-drive developement encourages a modular, re-usable design.

I feel that stressing the practical, immediate benefits of test-driven development is the best way to convince those who may believe, but do not know that automated testing makes their lives easier. It is so easy to write poor unit tests, and so hard to monitor, it is clear this is preferable to forcing the practice, even if it is official policy.

Velocity Crisis: Quality of Code

Recently, we have had a crisis of velocity at work. Our velocity has more than halved in the last six iterations. This post is not about that crisis; it is about one aspect of the introspection the crisis has called for.

We recently lost two not so short-term contractors on a team of seven. Initially, this was blamed for the velocity decrease. However, with a new hire, we still don't expect our velocity to get much higher than half the maximum of what it was with both of the contractors. I believe one of the major contributors to the current decrease is a previous decrease in the quality of our code. With the contractors, we were in a major push for new functionality, and it is clear that the quality of that code is such that now we have an increasingly difficult time adding new functionality. Not only were we not refactoring enough, but even the virgin code could have used a clean-up.

Which brings me to the subject of this post: sustainable development. Refactoring is the critical element in making sustainable development happen. It does require taking the long view, however. We reduce our velocity today, to ensure we can keep up the pace tomorrow. I used to believe in counting refactoring as part of velocity, but am now firmly opposed. The reason is that refactoring cannot be planned like functionality. It must be done at all times. It is not optional, and is not post-ponable. Around 90% of tickets created for refactorings that we did not have time to complete immediately have never been completed. They are either still waiting to be scheduled, or were included incidentally in a refactoring done that was required immediately.

One of the most difficult decisions to make when confronted with new functionality (especially if it has already been committed to the repository) is to reject the change and insist that a refactoring be done to make the change simpler and cleaner. It is clear, however, that this must be done to avoid a serious meltdown in velocity.

I am, of course, confirming what all the agile professionals have said all along. I knew that refactoring was important, but until now, I never considered it critical.

Velocity Crisis: Intra-Iteration Work Distribution

Recently, we have had a crisis of velocity at work. Our velocity has more than halved in the last six iterations. This post is not about that crisis; it is about one aspect of the introspection the crisis has called for.

Of late, we found that we have been pushing massive amounts of tickets because we overestimated what we could get done. Ee suspected, however, that it was more than just optimism and a constantly decreasing velocity. We suspected that part of the problem was not recognizing early that we were overcommitted. We found ourselves too often in situations where work needed to be pushed to the next iteration, but that our options were limited because we were at the end of our iteration.

We began looking at the points completed per day of our iteration to get a handle on the lack of visibility into iteration progress. We found that a graph of our points completed per day looked like a ramp. This, of course, explained why we were pushing tickets so late in the iteration; most of the work was being completed at the end of the iteration. It was normal to be only 20% complete when we were more than halfway through an iteration, so there was no reason to worry. I do not know what the ideal curve would look like, but probably something like a bell. Certainly something more constant than a ramp.

Beyond making a concious effort to complete tickets earlier in the iteration or raise warnings to our colleagues if we are not, we have actually reorganized our iteration. We have increased the time for testing, which gives us an extra day of buffer room at the end of each iteration. We have moved stakeholder meetings to the end of the iteration. This way, the seniors are not completing tickets, but planning for the next iteration towards the end of the current iteration. Not only does this mean we cannot complete tickets at the end of the iteration, but it also means that as soon as the build is approved, developers (and critically, senior developers) can focus and start work on the next iteration.

This seems to be working. Our velocity has begun to increase again (thanks to the new hire, and more senior involvement in direct coding), and the curve is flattening out. It is now much easier to predict whether we will complete what we say we will, usually by the end of the first week of our two week iterations.

Avoiding Storing Common One-To-Many Relationships

Occasionally, one comes across common classes of objects that you would like to be able to link to a number of different types of domain objects through a one-to-many relationship. For example, in a Customer Relationship Manager tool, you may want to have comments on many different types of entitities (customers, applications, disputes, etc). Same thing for tickets, attachments and other common structures.

Linking up these objects in the domain model is trivial: a list here, an ArrayList there. The real pain comes when you want to persist these relationships. Conventional wisdom holds you would create a link table for each of the entities you want to link to your common entity. The problem is that this is tedious and time consuming because not only do you have to create the tables, you also then need to modify your loading and saving procedures for the domain object. Because this is a one-many relationship, you will be getting multiple rows back per object, with all the muck that brings.

An alternative approach which we have begun to use at work would be to instead store the link to the domain object in the other table (the one side of the relationship). This way, there are no new tables to create. Of course, the immediate problem is that you cannot have a foreign key from one table to many tables. But imagine if instead of storing a database foreign key in the column, you instead store a "programming foreign key". In other words instead of a database link, store a programming link. In python, this would be the `repr` of an object. In this context, we could store a call to a finder function in this column, and evaluate it on the way out.

Storing a finder call in this way allows the common object to link to any number of different types of domain object. The domain objects just need to iterate over the collection of common objects to find those that link to themselves. When you add a new domain object, all you need to do is add a method that searches this extent. No database modification, no saving or loading code to manipulate.

Using some other tricks from python (though this is generally applicable), you could also use the `str` of the object to have a generic display of the common object's link to the domain object. If the protocol you are using (like python's `repr`) is generic enough, you could literally store anything in the column. If we take the comments example, the comment could have been left by a `User`, a string username or a list of `User`s.

There are downsides to this approach, of course. The biggest one is that ad-hoc queries on those common objects becomes more difficult. The good news is those common objects are not normally the subject of ad-hoc queries because they are not interesting domain objects. Additionally, if your system has the facility for ad-hoc queries in a programming language (no, not SQL), then this is a non-issue (depending on who is doing the querying). Secondly, you cannot enforce referential integrity in such a system at the database level. Again, though, I have not run into referential integrity issues on these kinds of common objects. Integrity is usually important for pieces of the domain that must fit together properly or system failure may occur. Finally, performance may also be impacted depending on how intense the finder function is, though these should be fast if you have designed your system well.

In the end, it is a trade-off between ease of addition of new domain objects and ease of querying and data integrity. For me, the database is just a storage system, so I will take advantage of the programming language I am working in, even if I lose some functionality at the data level.

The Other Source Control

Writing software without source control these days is insanity. Having worked in professional environments both with and without source control, I know the pain of not having it, and the ease with which it can be introduced. Now, I never write anything that is not under source control, even if I am working alone. It is so free and the benefits so clear that I don't even question it anymore.

I think that almost everybody (except maybe Linus Torvalds) agrees that source control is critical to software development. For all its other shortcomings, the Joel Test has source control right at the top of the list.

What has strikes me, however, is how little concern is normally given to source control of the database. Especially if one is not using an In-Memory Model, your database is critical to the correct functioning of the code. Yet I rarely hear about shops that control change to the database in the same way they control changes to the code.

I'm not talking about changes to the data housed by the database (analogous to instance changes in the running code), but changes to the structure of the database. I can remember countless times where all of a sudden, our application would stop working. It turns out someone applied a database change directly on production. Other cases include asking "what changed in the database between version 1.1 and 1.2 of the system", "what do we need to do to the database when we release this version" and "what do I need to do to run a previous version of the software"? These questions are difficult to answer accurately without all change to the database being controlled just like all other changes to the code.

At work, we have spent a substantial amount of time working to bring the database in line with solid change control for almost a year. We are finally at the point where the code is able to detect the version of the database it is running on, and is able to either upgrade the version or roll it back to a previous version depending on what version of the code is currently running. The only changes that are applied are those that have been checked-in to the code repository as being required to run on the next release of the software.

We have found that controlling change in this way provides all the same benefits source control for our code provides. And in the end, doesn't this make sense? Doesn't it defy logic that part of our code should be controlled, while part of it (db schema) should not be? Now our database acts just like our code: changes are logged historically, changes are fully tested, changes are applied once per release on production, and the only changes made are those that are fully automated.

Like code source control, I will never look back.

Love Your self In Python

When I was using Java, I always insisted on making the this reference explicit when referring to an attribute or method within the same class. I think it always makes the code clearer because there is never any doubt about where an identifier is coming from; either it is a temp or argument or there will be a this in front of it, in which case it is an attribute or method.

The habit of placing this in Java code even where it was not required certainly eased my transition to Python. In Python, self is the explicit first parameter to every method, and must be used whenever accessing methods or attributes of the object. I have often heard people complain about this, but it is one of the things I love about Python.

In addition to the clarity described above, the explicit self as the first parameter to all methods unifies functions and methods in a way that is not done in other languages. Once you start thinking about methods as functions that take an object as their first argument, functions and methods become very similar. This encourages a dual interface that is used often in Python. This dual interface is one where you can use either a function or an object's method to do the same thing (like Python's re module).

For example, I find myself often writing code like this:

def get_table_sql(name, db):
  return 'CREATE TABLE %s (%s)' % (name, get_column_sql(name))

class Table(DbObject):
  def get_sql(self):
    return get_table_sql(self.name, self.db)

Now, nothing prevents you from doing this in other languages, but I find that the explicit self goes a long way towards encouraging programmers to think about functions and methods in similar ways. And at the end of the day, your design is based more upon your frame of mind than the technological limitations of the language you are using.