It's time to learn Kafka for data jobs investment banks

If you want to work in a data engineering role with an investment bank, it's not enough to be an expert in Python any more: you also need experience with the sorts of platforms that allow banks to handle data in real time. 

One platform is particularly popular: Apache Kafka, the open-sourced event streaming platform that allows users to sort and store large streams of data in the cloud. As banks' demand for data scientists grows, so their need for Kafka expertise is expanding too.

Kafka's use in financial services is not new. Bloomberg presented on the use of Kafka in its derivative market data group at Kafka's summit in 2018. Dutch Bank ING presented the same year. British bank Nationwide has been boasting about its 'Kafka speed layer' since 2019. 

Developed by LinkedIn and open-sourced in 2011, however, many of Kafka's most vocal users have been big tech firms like LinkedIn itself, as well as Tinder, Netflix and IBM. But behind the scenes, investment banks have also been boarding the Kafka wagon.

When Bloomberg presented its reasons for choosing Kafka in 2018, it said the platform appealed because of its ultra low latency, super-scalability, zero data loss, fault tolerance and large storage capacity. All of this applies equally to banks and to hedge funds. 

Accordingly, Goldman Sachs and JPMorgan alone are currently advertising around 100 and 500 jobs respectively for which Kafka is a prerequisite. Goldman is using Kafka with everything from its data lake to strats roles and is in the process of building a brand new inventory management system, written in Java and using Kafka for event sequencing.  

As banks' use of data and need for data scientists expands, their need for Kafka expertise can be expected to grow correspondingly. Now might be the time to get ahead of the trend.

Have a confidential story, tip, or comment you’d like to share? Contact: sbutcher@efinancialcareers.com in the first instance. Whatsapp/Signal/Telegram also available.

Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)

Photo by Markus Spiske on Unsplash