This is what you need for the different jobs in big data
As we've noted before, not all data jobs are the same. People talk about big data and data science as if they're catch-all phrases requiring a particular skillset. They're not: data jobs are a broad spectrum and the person who suits one job may not suit another.
If you're wondering where you fit in the data universe, quantitative assessment firm Correlation One has developed a helpful mapping methodology explaining precisely what each role involves and the skills required. We've added its charts to the bottom of this page, but here's our short verbal explanation to accompany them.
What's a data scientist?
A data scientist is a kind of catch-all description for someone who finds, processes and analyzes data in order to extract meaningful results that can predict the future. As a data scientist, you'll need to be able to devise ways to collect data, and you'll also need to be able to interpret that data that's delivered. You'll also need to be particularly good at developing and testing hypotheses.
What's a data engineer?
As a data engineer, you're going to be much more about 'data wrangling' or preparing the data for use and then devising ways of presenting and analyzing that data.
You'll need to be familiar with SQL and database programs and you'll also need to know how to write algorithms that can parse the data. You'll need to know Extract, Transform and Load (ETL) processes, and you'll need to understand how to structure data in hierarchies and create visualization tools that allow users to access and interpret the data more easily.
What's a quantitative researcher?
If you're a quantitative researcher, you're mostly going to be about investigating the data (including through visualisations and hypothesis testing, again) and using the data to make predictions about the future. To make predictions you'll need to understand quantitative modelling techniques and model validation, including the dangers of under and over-fitting your model to the data available.
What's a machine learning researcher?
Machine learning researchers have a whole category of their own according to Correlation One. Their main purpose is automating the use of data to make predictions about the future.
What's a business intelligence (BI) analyst?
BI analysts work with end users in the business to look at what historic data says about a business's performance. - BI is less about predicting the future and more about analyzing what the past says about the present.
If you want to work in BI you'll need to be excellent at sourcing, describing and synthesizing data. You'll need to be able to develop and test hypotheses and to communicate them to the business you work for. And you'll need to know a lot about that business (which isn't always imperative for pure data roles) for your conclusions to be meaningful.
What's a database developer?
If you're a database developer you're going to be at the forefront of data storage and retrieval. You'll need excellent knowledge of SQL and systems like Kubernetes.
What's a product manager?
Lastly, a product manager works closely with end users (eg, portfolio managers in a hedge fund) to devise a data-led product that will be valuable to the business. In the words of hedge fund Two Sigma, "product managers aim to create products that customers love.” – They’re, “the conduit between the users, the business, and the engineers.” They’re also, “held accountable for the end-to-end user scenarios.”
If you want to be a product manager you're therefore going to need to know a lot about the business you're working for. You'll also need to understand how to generate, interpret and communicate hypotheses resulting from the data, and how to source data in the first place.
Most data jobs require a bit of everything
The charts below from Correlation One reflect what's required for different jobs in Big Data. Dark blocks (number 5 on the color code) are the skills that are imperative for each role; light blocks are the skills that help but aren't essential.
Most jobs require a few specialist skills, plus a bit of everything. And as Correlation One points out, different employers use job titles to refer to different things. - A data scientist at one place might be called a business analyst at another.
The charts below are used together. The first chart shows what each data job entails. The second breaks down the skills required for each task.
Have a confidential story, tip, or comment you’d like to share? Contact: sbutcher@efinancialcareers.com in the first instance. Whatsapp/Signal/Telegram also available. Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)
