The Big Data Divide

Understanding the Data Sprawl

In October of 2017, Joe Caserta, Caserta Founder and President, led a 3-day educational series: Agile Data Warehousing, ETL, and Big Data Workshops. On the third day of the course Joe explained a common business challenge known as the Data Sprawl. 

The following is an excerpt from the recording.

The Data is Ready 

The innovation is starting and the data is definitely here, this is not something in the future, it’s happening in real time. The business understands that they need to keep progressing and they need to keep evolving.

The IT Roadblock

IT invests so much time and effort into doing things the old way. They develop business processes that forces us to do things the old way. This reinforces a sort of technical debt and requires all processes be done the old way. In order to change, it’s kind of like turning the titanic, it’s very slow. But the business can’t wait, right?

The Business Timeline 

So the business says you know what, hey there’s AWS out there, there’s google cloud out there, so I’m just going to swipe my credit card and do it myself. Or I’m just going to buy a server and put it under my desk and do it myself. IT is just too rigid, not agile enough and can’t really help me meet my needs. By the time they get me my answers, it’s too late, my competitors have already moved past us.

The Data Sprawl

So there’s a problem, and we’re calling this ‘data sprawl.’ People are just taking the data they need, making copies of it, and putting it somewhere else, to do analytics. This is done by the business. Your division doesn’t want to be dependent on IT, because IT takes too long, IT has to have all things considered, they cant just do what you want immediately. IT has to think about the impact of everyone else and that slows you down. So we have to do things in a way where it doesn’t slow us down. We have to have an enterprise solution and we have to have some governance but without it slowing us down, otherwise this is never going to stop.

The Irony of it all

We, as IT folks, are very rigid and very governed, in our every step of the way when we do a solution. But we’re actually forcing the users to take the data and put it somewhere else completely ungoverned. So we’re trying to solve the problem, but we’re actually causing the problem. We need to come up with a better way to do business.

The Statistics

Currently, 43% of all organizations are doing this. They’re saying this is not working anymore, we have to change the way we do things. While, 37% are planning to make changes. So in total, that’s 80% of all companies that are either doing or planning to change the way they do business. Moving from regular IT infrastructure to more of an agile environment – usually including a data lake and moving to the cloud.