7 Reasons why Data Projects are challenging (and how to avoid failure)
It’s no secret that companies have been putting more focus on activating their data in the last couple of years. Digital transformation projects are popping up left and right and enterprises are increasingly investing in their AI strategy. With so many data science, machine learning, and AI -oriented initiatives, one could expect a new technological revolution right around the corner, right? Then, why are companies failing in their efforts to embrace this new paradigm?
A recent VentureBeat article revealed that 87% of data science projects never make it into production, while just in 2016, Gartner estimated that 60% of Big Data projects fail. Those are staggering numbers.
The problems are very much real, but what is interesting is that not all of them are technological. On the contrary, some of the biggest challenges come from the more “organic” part of the company: its people.
Let’s go into more details about these struggles and how to overcome them.
Some organizations just aren’t ready yet. And the biggest setback is their mindset. A true digital transformation requires a change of the very nature of the organization. It’s not enough to simply build or buy the new tools. They also have to be deeply integrated and properly used in order to be efficient. In some cases, the processes need to be adjusted to include this new digital reality.
Change can be a slow process but is aided by consistently involving all stakeholders involved. Don’t hesitate to include the most critical persons. Asking them how their work can be better and listening to their opinion, creates engagement and more buy-in.
A good way to involve all stakeholders is to organize (a limited number of) workshops. These workshops need a pre-set structure, clear and tangible goals by formulating questions that need to have an answer by the end of the workshop. Take also into account that when innovations start as a playground, they require a plan to be adopted by, and rolled out in the organization.
Becoming data-driven requires a bit more of a risk-taking attitude than organizations are accustomed to. Working with data involves experimentation and it’s important that data professionals have the freedom to do this. But progress must also be evaluated, and projects stopped when they don’t seem as fruitful as anticipated. So ‘Fail fast, learn fast(er)’!
Unclear business requirements
Because digital transformation leads to a profound organizational change, companies need to be willing to look internally and decide where they are and where they want to go in the future. Reaching the right business requirements for innovation is mostly about understanding the business value of the project.
Although it makes sense on a strategic level, how do you actually make it granular and formulate an action plan? You have to bridge the gap from business value to business requirements to functional/technical requirements. Often the problem is that even though business stakeholders have a good high-level overview (where they want to go), they rarely envision each step of the way getting there. They also tend to change their mind which makes it hard for technical people with a process mindset to wrap their heads around it. This is why working in an Agile way and being flexible about what needs to be done is vital for such projects.
At Anchormen, we have incorporated this process as part of our philosophy and approach, which we call ‘Grow Live’. We always start small and work closely with the client. This lean approach revolves around flexibility and short sprints with extensive feedback moments, allowing us to test results quickly and improve efficiently. Priorities are continuously adjusted, and new insights incorporated easily.
Disappointment in previous innovative and/or data and AI projects
Failure to yield results with past innovation projects leads to disappointment in the organization. This disappointment can even lead to more risk-aversion and push back when new initiatives are proposed. People that have been part of a failing projects before, might feel demotivated to do the work again.
This can be devastating for a company that strives to innovate. But you have to remember that failure is part of the transformation process. As mentioned in the first point, there will be some projects which are just not feasible and have to be stopped early. But that doesn’t mean the next one won’t be a success. As the saying goes, “the only real mistake is the one from which we learn nothing.”
Lack of (internal) expertise and resources to run a data project
The demand for business analysts, data scientists, AI specialists, and machine learning engineers is booming – recruiting new employees in this field is hard for most companies. Also, it takes time to train people and give them the right expertise to do the job. Especially if the organization itself doesn’t have the required knowledge to train this new talent to begin with. But then how can you find people to connect the business requirements with data solutions and insights?
If you are looking to accelerate your data project, you might be better off taking a more hybrid approach: hire external experts who can manage and work on your project and business cases, while simultaneously have your employees trained (on the job) and give them the expertise and experience to take over once the solution is ready. This ensures a smooth progress and the development of people which are involved with your company from day one of the digital transition.
Data Warehouse is running out of capacity
The Data Warehouse is the central data platform within most organizations and thus, the typical starting point when becoming more data-driven. However, it is important to understand that the technology itself and organization around it are optimized for reporting on structured data which is tightly governed. In addition, it can easily run out of capacity due to the tidal wave of data incoming daily nowadays and the required processing capacity to run everything and turn it into insight.
When the business side of the company demands more, better, and quicker insights and analysis, the traditional Data Warehouse tends to fall behind. Other technology is needed to store and process data in different ways to complement the warehouse.
This is where new data solutions come in. Cloud-based, flexible data platforms that connect various data sources to allow for predictive and advanced analysis. With Anchormen Data Hub, we help companies to accelerate their data strategy, get business insights and achieve meaningful results in the shortest period of time. Leveraging our data expertise and a solid cloud foundation, the hub combines structured, unstructured and real-time data and information from traditional data warehouses, business intelligence and various other data sources, like e-commerce and ERP-systems to allow various business cases.
Data Migration and connecting data sources is difficult
Companies are organized in departments, with their own systems, applications and reports, mostly unconnected, not in sync and sometimes out of the scope of the IT department. For Data projects to succeed it is crucial to connect the (real-time) data and information sources. We can be very honest about this, it’s extremely challenging without the proper experience and insight in data structure, sources, and interactions.
Although investing in ESB/ETL tooling helps minimize the problem initially, it opens a totally different can of worms. It usually comes with a hefty price tag and an annoying vendor lock-in. Innovation projects often require more functionality, speed, and flexibility than standard tools can deliver. In those cases, companies might find it more beneficial to go with something that is best of both worlds – an operational data hub that works as a data warehouse but has an ESB-like profile.
Danger of Shadow IT
Data Science, AI and ML projects have a dependency on the IT department and require IT involvement. In reality, the waiting line at the IT department can be long and tedious. When it’s taking too long for the business, they lean towards solutions that can be implemented without involvement of IT. That is how “Shadow IT” is born.
In our experience it is best to keep IT involved and get a buy-in from them. In the end it will be the IT department that will have to manage any data applications in production, so it is good to incorporate their requirements and wishes as well. During the experimentation and PoC phase of data projects it may be OK to work with ‘shadowy IT’ (as long as security, privacy, governance, etc. are taken care of). But this shadowy IT must be migrated to the regular IT as soon as possible and a continuous dialog about this is required to make it happen.
We have to admit that this list is not exhaustive, we could have come up with a couple more points. But as a matter of fact, many companies overcome these challenges and manage to successfully activate their data (as you can see from our success stories 😉), so we shouldn’t be too negative about it.
Failure is a part of life. And like mentioned before in this blog, innovation involves embracing risk. What we have learned over the years though, is that a planned, but agile approach, like Grow Live, that involves all stakeholders and aims to create quick results as proof of value, is a great way to keep on innovating!
Are you interested to hear how Anchormen has created success for our clients? Feel free to contact us!