Data Science or BI? Part 1 from Information Management
By Chris Bongard
19th December 2011

The idea was to assemble a data set from information readily accessible on the web that would be suitable for interactive visualization and predictive modeling. My searches took me to website of the Current Population Survey, a joint data initiative between the bureaus of Census and Labor Statistics. There I discovered the annual CPS March Supplement files which contain a treasure of information on demographics, education, occupation and income of the U.S. population at annual points in time. So I decided to put together a soup-to-nuts analytics demo using the CPS data.

Starting with raw files from 2003 through 2010, I wrote a Ruby program to process each and assemble a final comma-delimited data set of over 575,000 individual records with attributes such as age, sex, race, education, occupation and income. I then piped those records into both the R statistical package and the visualization tool Omniscope from Visokio. In my presentation, I displayed relationships between age, sex and education with income using slick Omniscope trellis visuals, and then fit R machine learning models and lattice graphics to show “predicted” income as a function of age, education and health status. Since mine was the last presentation of the day, I'm not sure whether the smiling faces indicated enthusiasm for data science or excitement over the pending happy hour festivities.

After the presentation, one of my partners proposed that we consider using the CPS data as a basis for a formal OpenBI demo that would serve as training for new BI staff. But rather than Ruby to assemble the data, he mused, how about using Pentaho Data Integration (PDI)? Rather than a CSV file as the ultimate data store, why not use VectorWise or LucidDB as an analytical database? Rather than storing the data in a denormalized, flattened format that's needed for visualization and statistics, why not deploy a star schema design instead?

While we're at it, why not create an OLAP cube from the database for drilling and slicing and dicing into average income by age, education, race and health status? And why not use the database and PDI to source Omniscope and R?  Finally, why not deploy Revolution Analytics DeployR to integrate the R predictive models with Pentaho reports? In short, why not use BI foundational technologies to support the data science tasks?

My partner's suggestions made sense and got me thinking about the distinctions between data science and BI. Back in the Spring, I wrote a series of articles on DS for Information Management. One delineation I noted then, attributable to statistician and R user group leader Mike Driscoll, argues that statistical science and data manipulation are central to the conduct of data science. As a third critical emphasis, he cites visualization. For Driscoll, it's statistics for studying data, data “munging” = hacking for suffering with dataand visualization for storytelling with data.

Based on Driscoll's definition, I'd say my CPS demo clearly qualifies as DS – I munge the data using Ruby; I storytell with the data using Omniscope; and I study the data using R. And yet my partner sees the same set of tasks through a BI lens: ETL, relational database, OLAP and statistical models. I think both of us are right. The CPS work can simultaneously be seen as both BI and data science.

Since I posted those blogs six months ago, there's been an explosion of new articles purporting to define data science, several of which give me considerable heartburn. Next week I'll give my take on the similarities and differences between BI and data science. Warning: my point of departure is that the two are more similar than they are different – and that each can learn from the other to the ultimate benefit of  “competing on analytics.”

Read at source: Steve Miller, Information Management

Comments

Currently there are no comments. Be the first to post one!

Post Comment

*
*
*

Why chatbots matter to your business

The way we communicate is changing, especially for businesses. Many companies are now using social media to interact and talk directly with their customers, and the rise of Artificial Intelligence (AI) and chatbots is only helping this. In 2011 Gartner... Read More

How NOT to write your CV

Your CV is your selling point, it is the first interaction between you and your future employer. Having a good CV can make all the difference to getting your foot in the door at a company. There are plenty of... Read More

Big Data – Driving F1 forward

We’re coming up to the end of the 2017 season. The championship battle might be over but there is still plenty of work for the teams to do as they get ready for the 2018 title fight (here’s to another... Read More

Bringing company culture to life

We’re always telling our candidates they need to sell themselves when at an interview , but the same can be said to our clients. While you are looking for the perfect candidate, the candidate you are interviewing is looking for the... Read More

Where should we send our newsletter?

Close