Listening to yesterday’s The Daily Show (sorry, I am in London now), I watched Aneesh Chopra speak with John Stewart about the challenges government faces with “legacy systems”, “legacy databases” and “20 year efforts” and watched John’s eyes glaze over.
John, let me see if I can help you understand why Government is not like Campaigns – and use a New Jersey metaphor for you.
Let’s Handmake a Car!
Let’s say you have a 1969 Dodge Charger – it is up on mounts, and you have been wanting to refurbish it so your kid could actually know what it feels like to have true American muscle under their feet. But this Charger has been running….okay….for the past few years, and you realize it needs to a new gas line, a new steering column, and heck – even a new engine if you are going to have this puppy run like it used to.
Well, you could go find some automotive shops that specialize in finding those parts – and if these parts happen to have different connectors (like a hard hose versus a slinky hose), then you need to find the better part that can allow all of the systems within the car to actually work together.
You could also decide that, in the past 55 years, the technology has improved so much that you could conceivably gut the entire system and retrofit the body with a new engine, steering column, and so on. Especially since these parts are so easy to come by from the same auto parts stores.
But here is the rub: what if I told you that to get ANY of the parts mentioned above, you had to make them YOURSELF? And I mean actually handcraft the parts yourself with whatever machine you had available (like a CNC milling machine or a Dremel).
Now lets add one more dimension to this: you are going to either gut the whole thing and start fresh, and you are not quite certain you won’t make a mistake -OR- you are going to try to fix parts of the system and hope that each new part works with the older part. Either way, there are likely to be errors fitting the parts together.
You Don’t Get the Chef to Remove Your Spleen
Now, lets add one more complexity: have each part of the car made – by hand – by someone OTHER than you. And the only way to communicate is via the connections between your parts. You might suggest ways for your other coworkers to make their part, but for the most part, you only have control over how those parts fit with your part. And, remember, who is the person in charge of arbitrating the parts? Oh wait, is it you or the other person making the parts come together?
And one last twist: instead of mechanics, you have some of your older family relatives come in as well as your kids friends come to make these parts as well. None of them have any experience in machining parts or figuring out which bolt goes with which nut, but you put them on the task because they are available to work!
And what kind of Charger do you think will occur after all of this?
What if I pull out THIS wire?
When we talk about the VA Systems and why has it taken so long to fix the system itself, you and I have no idea how complex the system might be. The number of parts in a 1969 Dodge Charger is orders of magnitude smaller (like 1000x fewer) than the number of lines of code, the number of processes, the number of database tables and the number of old pieces of code that someone hacked in to make the system work for some bureaucrat’s report that HAD to be done because the Assistant Director of the Understudy of Artificial Limbs needed to know how many hooks were used during some arbitrary time frame.
Programmers go into legacy code (meaning code that is no longer being actively maintained) and have to discover why someone wrote some code one way, while not having the understanding of how the code in other parts of the system may impact it. It is like fixing the pipe in the gas line but not knowing that someone places a fuel sensor on the line and if you change the hose, the sensor disappears and thus the car won’t start.
And code is like that EVERYWHERE. That is why most coders are frustrated and scared of code written by others – other than themselves.
Greenfield versus Brownfield
When political campaigns get started, they are starting from scratch usually. You pull in the basic database of personal details (unlike the VA databases which may have multiple sets of sub databases) and you put it into a shiny new database. You write brand-new code and leverage off of new systems that allow you to check if you might break something if you modify a part of the system and you document what you are doing with tools that track and understand what coding this system is about.
That is what we call a GREENFIELD project. It is clean, it can be specified, the coders have a chance to make something new and fresh and the outcomes are comparatively simple since the number of permutations for various outcomes are still manageable/small.
The VA Systems are far from GREENFIELD. They are BROWNFIELD – and we mean really dirty – varying in code, systems, databases, interfaces (think different hoses or bolts), just a minefield of trying to understand what is happening and not knowing if a change in one part of the system will not break another.
Bureaucracy in government is about not breaking anything, especially since politically it can be a nightmare. Bureaucracy in coding is essentially a process not for the faint of heart, and requires focus — and serious time.
To give you an example: most startup companies I work with usually need to rewrite their platform (think the Healthcare.gov codebase) because the first version has so many hacks and crossed wires and such that they need something that will “scale” (read: grow with the user base without breaking). Every person who wants to revamp the system says “three to eighteen months” — and that is for a new site/application.
Now look at the VA Systems and tell me what you think those coders face?
Four years? Seven years? And add to that a bureaucracy concerned about not losing their jobs?
Twenty years sounds optimistic.
Why not revamp the system and address the current needs, and not keep holding onto the past?
Watching the interview, I was even more skeptical about the “20 year efforts” than Jon seemed to be. It seems to me that the problem VA Systems needs to solve is quite simple – store existing veteran information in a database, then provide an API for entering new data and for computing eligibility and various benefits.
This seems like a basic piece of infrastructure that every web-based startup needs, and a small team of 3-5 programmers could implement all the requisite functionality as a greenfield project in a year (accounting for a rigorous testing and review process). You’d also need to scrub all the existing databases, which doesn’t seem terribly difficult either, even if you had to write custom code to convert each of dozens of existing databases.
After all, we’re not talking about an algorithmically challenging problem – there’s no machine learning here, no noisy sensors, no NP-hard problems to approximate. It’s just a database with a nice interface, and I don’t understand why it would be so difficult to rebuild the whole system from scratch. What am I missing?
Ivan – I agree with your premise, but the real challenge tends to be all of the existing business logic that the system has. And since it can be quite a nightmare for all of the logic, it would be quite the challenge UNLESS the VA and/or government gets appropriate buy-in and agree to have some features evolve over time.