The Last Mile of Big Data

A lot has been said on Big Data as it continues to propagate into more functions, industries and markets in the business world. Hadoop, Hive, BigQuery and other technologies dominate, as well as the promises for profitable growth, competitive advantage and success for businesses that embrace this evolution, or some would say revolution.

FROM ‘HOW’ TO ‘SO WHAT?’

Technology has traditionally reigned supreme in Big Data initiatives. And for good reason. It is due to technological advancements that we can now capture, store, curate and analyse large quantities of data. However, we slowly see a shift downstream, to the analysis and reporting layer where data scientists and self serving tools reside. I believe this is a shift in the right direction as it moves the focus from the “how” to the “so what”.

In many organization that would be considered the last mile of big data – identifying the patterns that generate the insights to drive the business decisions. In theory, they’re right. In practice, there’s one substantial element missing – change.

TRANSFORMATION

As Bill Schmarzo, CTO at EMC, writes in his book ”Understanding How Data Powers Big Business”, one of the more significant impacts of big data is the organizational change or transformation necessary to support and exploit the big data opportunity. This is reinforced by some in the industry who consider the last mile of big data as the step to address the organization’s specific business situation, culture and goals.

YOUR UNIQUE HIGGS BOSON

This is different in every case and is highly context sensitive. It is the final alignment of the big data analytics processes, frameworks and best-practices with the unique business situations that make up the last mile. By reaching that rendezvous, you are taking a journey to determine what makes your organization tick and what your business is actually made of, towards its unique “Higgs Boson”. Then you have something very powerful, allowing the business to dive into deeper levels of leadership, culture and decision making.

BIG COMPANIES

My experience thus far serves me as evidence to that. While in Skype, we moved very quickly from driving a Big Data tech-heavy initiative to a business transformation one where culture and habits played an equal role in our ultimate success. This did not write-off the importance of being thorough and detailed when architecting the pipelines, feeds and technologies applied across the data value chain. All these elements needed to work holistically in order to create value and materialize on the objectives of the change.

I’ve had similar experiences when expanding our transformation across parts of Microsoft. As with Skype, we needed to tap into and change the daily habits of the managers and employees in order to get them on the Big Data bandwagon. That included developing an entire ecosystem of tangible and intangible elements thread together, similar to software and hardware, to create a new, lean and insights-driven business rhythm embedded in their daily work.

And in today’s large scale Google, I look at simplifying the way we make our decision making, prioritizing customer experience above all and ensuring consistency where needed across regions, products and functions. Balancing between these business principles and reaching the unique note optimized for your organization is key to ensure the right people get the right data at the right time (and make the right decision).

CROSSING THE LAST MILE

There are many elements that you need to get right in order to promote your organization to shift towards more data based decision making. In some cases, a few of these elements are beyond your control. However, discounting “acts of god” and the specific circumstances and personalities in each organization, there are a few fundamental principles that I found to be universally paramount.

From ensuring commitment, sponsorship and patience from the top to designing flexible solutions that can change when the need arises. You always need be on the lookout for anything that can jeopardize the success of the change, like minor and isolated disagreement of one individual to full-on divisional dissent. By starting small and building a web of trustful and credible relationships in the business you can ensure you have the necessary anchors in place when going “all in”.

Depending on the maturity of the organization and how far along you are in the data journey, you may encounter some of the instances above or of the following:

Habit Change

We all know that changing habits is a tricky endeavor, whether in a professional or personal arena. If we try and look at our own personal experience, we may find that when we do succeed in changing a habit or two, it usually comes down to our perseverance and our ability to stay the course. These two principles should guide you in conquering the last mile of big data by relentlessly communicating and promoting new and better ways of working. They should not be mistakenly associated with stubbornness and lack of context or empathy.

Put your trust in the Data

The first step before transforming the business into a data-driven one is to ensure the data is what you think it is. What I mean is that there needs to be a thorough review of the entire data stream, from telemetry implementation to metric definition and data aggregation, to ensure the data is properly curated and can be trusted. It only takes a few instances of misleading information to make employees lose trust and send the data bandwagon into a lengthy and costly diversion.

Collaborate with the business

Big data is unlike most IT initiatives in that it requires a close and on-going collaboration with the business stakeholders to ensure that whatever is being developed and delivered is relevant and material to the business. If you think of it, the business is there, accessible and ready to adopt. Bringing the business under your “data tent” ensures higher level of engagement in the midst of the initiative and more adoption and usage at the end of it.

Data-Driven or Driven-Data

It’s trendy, relevant and right to be data-driven. But you may sometimes encounter, particularly in more mature organizations, that the notion of data-driven is being slightly exploited to fit a specific narrative or agenda. So instead of letting facts and data determine the course of action, we see data being massaged and manipulated to justify a desired one. And thus, very quickly, we can end up with a narrative-driven data culture which is hardly the point.

Information overload

The topic of information overload has been widely studied by academics from neuroscientists to economists. Economist Herbert Simon once said, “A wealth of information creates a poverty of attention”. This goes back to the limits in our innate human ability, our sensory and cognitive faculties, to process this data torrent. So while Big Data has the potential of providing additional levers to fast track our business, if we’re not careful, it can also flood us with unnecessary noise, preventing us from focusing on our core competence and on what really matters.

With Big Data Comes Big Responsibility

Last but certainly not least is privacy. While going through a transformation, things can sometimes get overlooked and cracks can form in the overall solution which may expose user identifiable data to your employees. It is the responsibility of the team to ensure the solution is stress-tested for these issues and collaborating with the legal department while the initiative is ongoing is one way to go.

So, just like in the telecommunication world, where the last mile refers to the last leg of delivering the content to the customer, Big Data initiatives should also adopt this customer centric approach and focus on the last mile of the data journey. As without it, all else has no meaning.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s