When I got into the business of data management, it struck me how many similarities these implementations share with standard software development projects. What struck me even more is how many organizations do not apply lessons learned from software development to data management system implementations.
I wish I knew why, and I have some ideas, but I do not know for sure. It does seem that many of the people involved in data management come from data or business backgrounds. They know their craft, their business, and their data extremely well, but few have spent time in software development. Then they are tasked with running a large data management implementation, which is more similar to a software development project than a database, warehouse, or BI project.
Innovations from mainstream software development over the last decade can and should be applied to data management system implementations. HotQuant has testing
and requirements management tools in this vein, and many more techniques and tooling exist. But this post focuses on a higher level strategy. This has nothing to do with the technology or development process. The right strategy simply:
- delivers value to the business quickly
- builds momentum for you and your team
- exposes the progress and value to stakeholders
The following three strategies go from worst to best, generally. Your mileage may vary, because every situation is unique.
Boil the Ocean
I know “Boil the Ocean” is a cliche phrase, but this is the approach I see more than anything else. I feel like I need to address it. Most people I talk to know there must be a better way, but they cannot fight the inertia. You can recognize this approach by long negotiations with global vendors, who bring in a team of 10-30 people, then things get chaotic very quickly. If you see multiple committees making decisions slowly, multi-year roadmaps, “resources" (a fancy but cold term for people), and more architecture or process documents than actual code deployed to production—you are probably in a Boil the Ocean project.
These start with big, big promises and big, big architectures, and all-encompassing plans. Six months later you are still building “infrastructure.”
As I reread the above, it sounds pretty negative. That is only because the results are lackluster at best, even if the piece parts are high quality. Usually, most of the plans, architecture, and technology are all quality. The people involved are pros. The problem is your business stakeholders do not have the stomach for a multi-million dollar annual run rate with no tangible results in the first 6-12 months. At some point—soon—they have to see the results. No matter how good the infrastructure is, no matter how solid the plan is, no matter how important it is to lay the foundation—if it is not in production, delivering well understood business value, it is as if it didn’t happen.
What do you do if you are in the beginning or middle of this? Assuming a reset or scale back is out of the question, the only thing I have seen that works is to build a contingent that sidesteps (not ignores!) the long-term plan. They then figure out how to deliver something valuable—no matter how small—to the business. Use this success to communicate the value the team is producing. Rinse and repeat until you have a body of successes that clearly show momentum.
Fill the Container
Otherwise know as If You Build It, They Will Come. I borrow the phrase Fill the Container from a dear client who will remain nameless here. And honestly, I was right there with them in this mentality. The good thing is, they were able to pull out of this before it went too far.
Fill the Container is an analogy of filling the master data hub as the main focus of the project, at least at first. This is easy to step into, especially when the data governance initiative starts with individuals with a clear vision of the long term goals and value. Have bad data? Fix the bad data! Finding data that probably is useful to other target systems in the organization is not too difficult. Compared to finding target system owners and convincing them to get on board with a central data hub—finding and sourcing data is a cakewalk.
So even though technically you are making progress, until the data is in use, it is not delivering value to the business. Do not bide your time perfecting the system, making the data better and better, waiting for the right big-ticket target system to be ready to use your platform.
Just like a minor league player working up to the bigs, or a startup winning bigger customers as they prove themselves with smaller ones, sometimes you will not get to start with the big, important systems. If this is the case, keep the focus on finding target systems who need clean, consistent, quality data. As you build a portfolio of satisfied customers, this gives credibility to your platform. Then the critical, bigger systems will be more willing to listen.
Of course, focusing on the most critical target systems would bring the most value to the business. And everyone knows they suffer from poor data quality and access. But many times, they are slow to jump on board with your data management system. Why? Well, if we already know they need the data—maybe it’s you. You need to prove yourself first. So rather than focusing on Filling the Container with data that may be useful to the big systems in the future, focus on wins with target systems and teams that need your data today.
And when you do get that big, important, high-profile system interested—their requirements will probably not match your roadmap. This will be incredibly frustrating, because all your well thought out plans will have to change. Roll with it. What seems like your biggest headache is actually your biggest opportunity in disguise. If you can find a way to turn that high-profile system’s champion into your biggest advocate; if you can find a way to get that critical target system to start to rely your data management system—you have sealed your data management system’s place in the enterprise.
Being Target Driven
is somewhat a continuation of the preceding paragraphs, but called out distinctly now. We are all operating in a cone of uncertainty
, and none of us can truthfully say we have all the answers from the beginning. Which data attributes and requirements will be the ones that are most important to the target systems will be as much discovered as they are planned for.
Being Target Driven
is similar to JIT manufacturing
. It is a pull vs. push mentality. In other words, rather than Fill the Container
(“pushing” data into the hub in hopes that it will be consumed), the target systems’ needs defined the priorities. They effectively “pull” the right
requirements through your development process. The Goal
by Eliyahu Goldratt is an easy read with some deep points on this.
By putting all your team’s energy into delivering only the data your target systems explicitly need, and only at a level of quality that is acceptable to them (rather than perfect), your efforts will not get lost in the morass of “infrastructure” or “foundation” or “architecture”—whatever word your team uses to defend themselves when challenged by someone questioning a perceived lack of progress.
Target Driven sounds simple, and it is. But somehow, in most big data management projects, this simplicity gets lost. And better than simple, Target Driven is easy to communicate, to help your team and stakeholders understand your strategy.
Bonus: Two-Speed Delivery
The big consultancies
two-speed IT architecture as a way to have part of your team focusing on long-term security, stability, compliance, etc; while another part focuses on agility and responding to emerging business needs. I like the idea, but most of the information I have found is pretty fluffy. But, this may have a place in any of your teams. I have seen one client use it successfully.
Some people point out to me that my solution to Fill the Container and my emphasis on Target Driven strategy has a weakness of being reactive and not planning for the long term. Fair point. Maybe splitting the team in a two-speed delivery model can help address that weakness.
This whole post is kind of one dimensional, and there are a lot of factors to consider that will drive your team’s focus and process. These are the most general patterns I observe, and this post is an effort to help give shape to your thinking as you lead your data management implementation.