How are the latest trends increasing demand for Data Scientists?
February 9th, 2023
X
Latest news
How big is Big Data?
One of biggest assumptions people make about Big Data is that it’s just about the amount of data. It’s called Big Data so surely it’s about size right? Of course we know that’s not the case, but what exactly is Big Data?
As recruiters in the Information Management market we hear the Big Data buzzword daily. Big Data is big news, but not just in our industry. It’s everywhere! We regularly speak to Big Data gurus; we’ve found people to work for them or found jobs for them. And outside of that people are always asking us “What is Big Data?!”
In 2012 Gartner updated its definition of Big Data as follows: ‘Big Data are high volume, high velocity and/or variety of information assets that require new forms of processing to enable enhanced decision making, insight, and discovery and process optimization’. But what does that mean in layman’s terms? John Coppins of Kognitio explains; “I actually quite like the Gartner representation of big data consisting of Volume; how much data. Variety; the types of data. For example structured and unstructured data, such as free format text, pictures, video and sounds. Velocity; how quickly that data is coming at you.”
I was mulling this over whilst stuck in traffic on my way to work the other day. Anyone in the Knutsford area will know the chaos that is being caused by all the road works and temporary traffic lights in the town at the moment. I’d called my boss to let him know I was running a few minutes late and updated my Facebook status to have a moan about how much time I spend stuck in traffic these days. I made the decision to go the long way round in future and thought about the thousands of other people doing similar things at that moment. Not just in Knutsford but all over the UK. The impact must be huge!
So what’s this got to with Big Data? I wondered how Big Data is used to track the problems and thought it would be great way to demonstrate what Big Data is for the non techies of the world like myself.
According to the BBC, a survey of councils in Britain has found that there are at least 19,000 sets of roadworks under way right now. The research suggests if we combine the roadworks reported by more than 80 local councils, they would stretch from Land’s End to John O’Groats! Then imagine the number of people affected in cars, buses and other forms of transport every day. As well as making people like me a little bit late for work, it will have the knock on effect of people changing their route, buying more fuel and more breakdowns and accidents. Not to mention the shops and businesses affected if people avoid the area! If we wanted to track the impact it would involve not just vast quantities of data (volume) but data from various sources, in lots of different formats, (variety) and the data would be changing by the minute (velocity)
The thing that makes Big Data complex is much of the data is ‘un-structured data’. Un-structured data is hard to track but can be real gold mine of information. Some typical examples of un-structured data would be emails, text messages, recorded phone calls to customer service agents, photos, videos, social media, sensor data and customer surveys.
To track the impact of roadworks and traffic problems would include data from many of these sources. Sensor data which is information culled from road cameras, satellites and other recording devices that tracks behaviour such as traffic flow would be a huge source of information.
Add to that the thousands of phone calls, letters, emails, tweets from people and businesses we’ve got a ‘big’ Big Data case!
Of course most of KDR’s clients and candidates are involved in much more complex big data projects and using the insights generated to make business critical decisions. Sometimes it’s useful to take things back to basics and I hope that translating it into layman’s terms helps you to think about how to talk to non-technical stakeholders when discussing Big Data.
If you are embarking on a Big Data project and are looking for more information, I read a useful whitepaper titled ‘Achieving Business Value with Big Data’ written by ‘the father of data warehousing’ Bill Inmon. You can register to receive a copy on his website here.
How do you explain Big Data to people outside of the IT department?
KDR Recruitment is the home of the best data, technology and analytics jobs. For more big data news and views follow KDR on LinkedIn.
