It's a word I’m sure we've all come across by now, even if you’re not in the industry. It's been spoken at every conference I've been to (though that may have been due to the fact that I've only been to one and it was about Big Data), I've heard it bantered around in the office, meetups, startup events and believe it or not, once at a bathroom stall.
So what can I surmise about this enigmatic term who's non-techie definition alludes even the staunchest supporter and has every sales person raving on about it like it's some sort of bread that's been sliced and packaged for the first time. How about this: It's a bit of a buzzword in the industry even to the extent that even recruitment agencies are starting to specialise in it...
So what is it that you actually need to know about Big Data besides the fact that data is growing at an exponential rate? Here are three things I learned about Big Data coming from a geeky but non-technical background.
1. BEYOND THE THREE V’S
When I first heard that Big Data can be broadly (and we’re being liberal with that word here) defined by the three V’s, I was as confused as a cat with a spanner. Volume, velocity and variety… Do they all need to be present to be considered Big Data like social media data (text, photo, video) despite today’s application predominantly revolves around the text based data? But what about Call Detail Records (CDR) in Telcos which details your phone usage (call, videos, text, data) as well as geographical location, signal strength in relation to which cell towers, etc. I’m pretty sure they all collected in one format but does it still constitute as variety? Or is it still Big Data because of the variety of sources? Beyond that I noticed that not much was said about the link between the raw data, storage technologies (data systems), analytics software (application) and the visualisation software (application) for which helps frame what Big Data is all about.
The image above which I handily screenshot(ed?) for my research purposes helps portray said relationship and the flow in which data is transformed from data into actionable insights for the business. Big Data is not actually about the way in which we characterise the data but rather the outcomes which is derived from it’s analysis.
2. IT IS A SUBSET OF A LARGER FUNCTION
When I started looking into Big Data and Data Analytics I soon found that Business Intelligence (BI) was a highly related field which I thought was a little odd because data analytics deals with the future while BI deals with the past and present. On storage technology side of things, it certainly made sense as the data still needed to be cleansed, manipulated, stored and accessed but the way in which the data was analysed was completely different. After speaking to many businesses about their data science/analytics and business intelligence functions, I noticed a pattern in what they were all trying to achieve. Big data analytics, data analytics and business intelligence - they are all different types of analytics but by and large they are all trying to achieve the same goal: using data to justify or help guide certain business decisions.
3. ITS APPLICATION IS LIMITED TO A NUMBER OF INDUSTRIES BUT THAT NUMBER IS GROWING
At this stage I knew what Big Data was, what it was trying to achieve and now I wanted to know about the ‘How’. I knew that data analytics and predictive analytics was not new. Neither was the problem of having data sets larger than our current technological capabilities. The statistical techniques we use to analyse our data have not changed for the last 20 or 30 years but the key difference now is that the technology or rather the affordability of technology has caught such that we are now able to gather, store and analyse these “Big” data sets.
IN REAL TERMS, WHEN WE’RE TALKING ABOUT BIG DATA WE’RE ACTUALLY REFERRING TO THINGS LIKE:
Retail (POS data, NOT CRM marketing analytics)
Telco (Call Detail Records, Geolocation data)
Digital (Real Time Bidding [RTB]/Social Media/eCommerce)
Finance (Automated trading systems)
Data from data driven companies (LinkedIn/Amazon recommendations, FB Social Graph)
Healthcare – ehealth, MRIs
Manufacturing (Machine to Machine (M2M), Internet of Things (IoT))
As you can see, Big Data is present in quite a number of industries and the application of analytics on that data has not been fully realised. Despite the fact that the term ‘Big Data’ is a bit of a buzzword, it looks like that it’s starting to find its place in the mainstream as people start to realise its full potential.