State of Big Data Markets

bigdatasummary

I recently had the pleasure of presenting the “State of Big Data Markets” to a non-technical audience. Big data might be the biggest hyped term of 20132014, but the interesting part for VividCortex and many others in the industry might not be as obvious.

Big data disrupts traditional technology vendors and creates an emerging world order where legacy technologies and business models fall short. Sure, like any industry, legacy players will become entrenched rent collectors, but the innovators are a new breed of software and services, built on top of increasingly commoditized hardware, with costs that are miniature compared to just months ago.

But how do we get to these conclusions?

bigdatadefinition

Digitalia

For lack of a better term, ‘digitalia’ means the increasing digitization of our world. Companies, from Google, to your local convenience store, have an increasing obsession with collecting data about you. You even have an increasing obsessions with digitizing your life. It’s almost as if it’s part of human evolution to be digital.

bigdataproblem

Humans are the bottleneck

Rapid innovation and growth in data has created a challenges for humans. Humans are managing more systems, data, and technology per capita than anytime in history. Even on a personal level, imagine how many tools you use per day to make your work efficient. Now imagine that the people who build and manage those tools are growing at a rate slightly faster than the economy as a whole. But, data is going to grow 50x in the next 5 years.

This results in a gap between humans and the amount of data they must manage. Empirically, go ask your local database administrator, or system administrator how much free time they have. Ask them how often PagerDuty goes off. Ask them how many developer code pushes they need to manage.

It’s not pretty out there, and it’s getting worse for people on the systems side.

bigdatamarket

Open Source is the hidden beast

If you read the industry reports and follow the growth of the companies on this chart, you’d think that the legacy vendors have the technology side of things on lockdown. Because open-source solutions don’t generate the same revenue that legacy licensing does, you’d think that they struggle.

The reality: everywhere you look, in enterprise, in nearly every startup in the last 5 years, open source is abundant and flourishing. From tools, to database technology, to automation, to storage systems, it’s there, and in a big way. It’s not going away.

The final thing to mention about open source, is the license scales very nicely. Need to add 100 more machines? Well, you don’t have to drop $10k per database license if you are using hadoop. That’s a clear financial win.

bigdatatech

A more accurate database technology landscape

Oracle, Microsoft, and IBM will continue to be strong players, probably forever, but on the streets, developers are using Mongo, MySQL, Hadoop, and a long list of other open source tools to manage their data systems.

It’s only a matter of time until vendors build cost-effective solutions around big data technology and those services become a clear competitor (both cost and functionality) to the tools and services the legacy vendors provide.

bigdataenabler

The Perfect Recipe for Big data

Many things have come together at the right time to make this all a possibility. It’s important to note that new business opportunities exist in each of these categories.

bigdataenabler

The Economics of Tools & VividCortex

We think about this stuff everyday at VividCortex. We are playing in the tools space. Our big bet is that humans need more help managing increasingly complex systems.

We believe the industry trends support this thesis and that’s why we are tackling one of the most underserved sectors of this emerging world order.

Thoughts? Criticism? Concerns?