It is just so cool to use new stuff. Knowing that you are on the cutting edge of technology is one of the most important drives for those of us who do research, and a strong attractor for tech junkies of all sorts. Plus, no one wants to spend months working on a new project and three days before it rolls out learn that it would have been quicker and cheaper to develop, and more reliable to use if only new component X had been part of it.
All of this is important to the argument that every developer should remain conscious of what is new and exciting in technology. No one is likely to keep track of all of it, but there is a clear need to explore and experiment whenever there is an opportunity. This is a well known argument, and hopefully not one I need to convince you of. However, it comes with a caveat.
To understand, consider this story.
Interviewing at GrammaTech
A little over 3 years ago, I interviewed and ultimately ended up working at GrammaTech. Coming into the interview, I had a long-lasting and enjoyable association with the Boost collaboration as an active volunteer and review wizard. I am generally very supportive of Boost, and have made broad use of some of the libraries available in Boost. It probably comes as no surprise that I was asked Boost-related questions as part of my interview.
One such question was about the Boost Spirit parser library. I was told in my interview that GrammaTech was using Flex and Bison for lexing and parsing in source code analysis tools, and they asked me whether they should move to Spirit.
Now, the goal of Spirit is to allow a programmer to design and implement parsers in line in C++ code. More recently, the library has split into three pieces, calling the parser component Qi, and also adding a lexer called Lex, and a generator library for output of program information called Karma. This is a very interesting package that I enjoy programming with in various projects of my own. It is also powerful enough that it has been used to implement a full, standard compliant, and efficient C preprocessor, Wave, as well as many other libraries and programs that have proven useful around the world. In performance tests, it has produced binaries that perform as well as or better than those produced by other parser construction tools, and even hand-tuned code.
Given my background and already-professed liking for the library, you might think that you would know my answer.
I replied with a question. I asked my interviewer whether the Flex/Bison tool chain was meeting their needs, and whether they foresaw needs in the future where Flex/Bison would become an impediment. (At this point in the interview, the mood of the person I was talking to brightened considerably.)
The Peril of Expanding Dependencies
Why not suggest moving to a library I like, that provides a combination of features and performance that is different from other available options and in at least some cases preferable? It’s all about the dependencies.
People outside the software realm often think of a program or service as a single monolithic entity. The program is written, from bare Earth, by the developers and delivered to the customer. Of course, that is almost never true. For example, the Chromium browser I’m using on my computer has a large tree of dependencies (Chromium Dependencies). This isn’t just Chromium — Firefox has its own vast web of dependencies.
Almost every package of commercial or open source software has extensive dependencies.
The arguments for why to do this are well known, and although there are some who argue against this style of development, it is largely the standard in the industry. Using an already developed, widely used, and thoroughly tested library is a savings in many cases. As long as the library is close enough to the needs of the package being developed, flexible enough that any differences can be worked around readily, and actively maintained, using a library or package that comes from outside the project reduces design time, implementation time, and maintenance burden.
The important point is that bringing in a library or external package for a good reason reduces these costs. It does not eliminate them. There are costs for finding the library, testing it, developing engineering skill in using it, and developing interfaces that will allow for clear separation of library concerns from the concerns of the package being developed. Replacing an existing external library or package with a new one has increased testing and implementation burdens, since the new library needs to be at least as good as the old one was on many inputs. There is also an existing interface that may not be well suited to the new library and require modification. Such modification can sometimes stretch through vast reaches of the code base.
In the context of the interview question: I like and understand Spirit, but that doesn’t mean anyone else at the company does. If I want to add Spirit to our tool set and make it a dependency for existing code, I have to justify the involved costs. If I can’t justify the costs, I shouldn’t make the change.
A partial view of the dependencies for the Debian Firefox browser package. No transitive dependencies are included. Generated using the cxnet package of graph building functions for python.
If it ain’t Broke
This is the problem with the latest and greatest. It may be better than what is in the project at the moment, but is it enough better?
New packages and libraries come out every day. The Boost collaboration alone releases new versions four times a year. Most of those new versions feature new libraries, and all include fixes and improvements to existing libraries. Some Boost libraries have been superseded by changes to the standard libraries and their use can be replaced, or possibly a library from a different source is a better fit to the needs of the project. This situation makes it clear that even trying to keep up with the newest from Boost is a important investment in a project.
Most major development projects include dozens or hundreds of dependencies, not just one or two. The problem of tracking and inserting the latest releases as dependencies then becomes impossibly large. Combined with exploring possible replacements for existing dependencies the problem only grows.
Tracking and vetting the latest and greatest is unscalable in a development environment. Expecting to use no external dependencies is also not practical in most development projects. This leaves only one practical path forward.
There must be a trigger that causes developers to look for an external solution to project problems. The trigger has to be a decision that the current attempts to solve the problem will not be sufficient. More simply put – “If it ain’t broke, don’t fix it.” (This is clearly an attempt to insert my Appalachian childhood into my blog entry. I plan to make such an attempt each time I write an entry. I will not promise this will be my only characteristic oddity.)
When to Add Dependencies
The moral of the story, of course, isn’t that dependencies are bad, or that new dependencies are bad. It is that unneeded or uncontrolled dependencies are bad.
Designing, writing, testing, and validating software is hard. Changes in design and implementation over time are inevitable. So, each change, and each new inclusion, has to be done in a considered fashion. Adding a new dependency or a new feature adds new possibilities for failure, skills required for maintenance, and needs for testing. Don’t take these costs on lightly. Understand why the value of the new inclusion outweighs the costs, and if it doesn’t outweigh the costs, don’t make the change.
