Peter Macdiarmid/Getty Images for Somerset House
show image

Sooraj Shah

Contributing Editor

Sooraj Shah is Contributing Editor of New Statesman Tech with a focus on C-level IT leader interviews. He is also a freelance technology journalist.

Should we stop talking about the data science shortage?

For the last five years, the common consensus has been that data science skills are the most sought after on the planet. Every report seems to reaffirm this belief: LinkedIn said the role was was the most promising in 2019, while Indeed showed a 29 per cent increase in demand for data scientists year-on-year between January 2018 and 2019.

There is a huge demand for data scientists, and there is not enough supply. So yes, there is a shortage, and indeed in my conversations with chief information officers (CIOs) and other IT leaders, it is clear that along with cyber security experts, it is one of the hardest positions to recruit for.

However, there are three key reasons why this shortage is actually not the biggest issue for CIOs (or businesses) when it comes to data science.

1. Misunderstanding what the role is

Despite such a surge in demand for data scientists, there still seems to be a lack of knowhow about exactly what they do, or what skills they have. There is a big difference between a data analyst and a data scientist, for example, but the roles are often thought of as the same. A data analyst is still a critical role in many businesses, but the data science skill set is far deeper, incorporating a combination of mathematical, statistical, computer science and modelling skills – as well as the analytical skills more associated with a data analyst. This means a data scientist can create models that can help to predict what will happen in the future – asking questions around the unknown, whereas a data analyst can provide insights on existing data from a number of different perspectives.

Without knowing what a data scientist really is, what is an organisation meant to do when they hire one?

2. CIOs aren’t helping data scientists enough anyway

…which brings me onto my next point. CIOs and their C-level counterparts can’t presume that hiring a data scientist will solve all of their data issues. There needs to be a clear plan in place: can the data scientists access all of the data they need? Is there an overarching data strategy which means that the data is flowing throughout the organisation? Are there internal or external barriers? Has all of the board bought into the idea of using data to this extent? There are so many hurdles to overcome before a data scientist is useful. That doesn’t mean that these need to be in place before hiring a data scientist, in fact it would make more sense to have one there to help make the strategy work, and tell the CIO or chief data officer, exactly what they’d need.

3. Striving too high – mixture of domain experience

One of the reasons businesses might not be able to fill a data scientist vacancy is because there are not enough candidates with the domain experience necessary. I have written at length about the need for a data scientist that knows the industry – whether it be in the legal or finance sector – but the truth is when demand is low, organisations have to think outside of the box. In fact, when I spoke to Mike Bugembe, the former chief analytics officer at charity JustGiving, he said if he had to choose between getting the best algorithm writer who doesn’t know anything about the domain or to get a mediocre one who knows a good amount about the domain, the choice would be easy.

“I’m an advocate to say: get the best algorithm writer – make sure you have the best domain experts in house, and make sure they work together,” he told me.

In summary, there’s a shortage of data scientists – but it’s not the main reason companies are failing with their data strategies or business initiatives. It’s time enterprises considered how they can better recruit, use and work with data scientists.