The Emergence of Data Scientists

Image courtesy of Sergej Khakimullin/Shutterstock.com

Every year around this time, I like to do a tour of companies, mostly startups, in Silicon Valley.  It’s a chance for me to get annual pulse on new trends in “the valley”.

Silicon Valley is like no other place.  When I grew up there many years ago it was a much different place.  At the time, IBM was the largest employer where I lived in San Jose and their campus included a hotel, sports complex, orchards, and a nine hole golf course (to entertain and impress their increasingly present Japanese visitors). Although, many of the orchards still remain, the IBM campus has nearly disappeared over the last 15 years in favor of housing complexes and strip malls.  

Today, Silicon Valley is littered with thousands of high tech companies throughout every corner of the Bay Area.  I now live on the East Coast but it always fascinates me to drive up Highway 101 and witness the new names on the building and updated trends conveyed on billboards. One thing that is constant is that although startups come and go at an increasingly fast pace, defining technology trends emerge frequently that shape the future.

Three years ago, it was all about Ruby on Rails, a model-driven application framework for rapid development.  Over the past 2 years, it’s been all about mobile.  Mobile developers continue to be in huge demand and some of the best can command annual salaries in excess of $250K. Over the past few months, I’ve noticed the emergence of “data scientists“. In fact, at one of the startups I visited last week, nearly 25% of their employees were data scientists.  For software companies, the role of the data scientist is to take increasing amounts of public and private data, identify correlations in the data, and incorporate those findings into an ever evolving algorithm in the company’s product.  Often the end result isn’t necessarily a product that the company is trying to sell but embedded to sell other product or services.

Data scientists aren’t new to Silicon Valley or technology-focused companies.  Companies like Google were based on the premise of aggregating large amount of data and presumably have been employing hundreds of data scientists for some time. Other companies such as Amazon, Nike and LinkedIn are now aggregating vast amounts of data to sell products and services.

LinkedIn, in fact, is quickly becoming less of a social network and more of a data intelligence company.  Don’t think for a second that they aren’t taking all of that information you are giving them and using it to sell other stuff.  Today, it is very safe to say LinkedIn knows more about most companies’ employees than their employers themselves. One could argue they are deceiving users by enticing them with functionality that captures even more personal data not just about themselves but now their connections.  Just look at the recently launched Endorsements.  As I look at my own endorsements, I see a lot of valuable data that LinkedIn can now use both for and against me (ie. show highly targeted job opportunities = good, have skills that are defined subjectively vs. objectively = bad).

I guess we must all get used to the power of data because Moore’s law is now very applicable to data growth. I just wish more of my connections would appreciate my “Awesomeness” skills.