Working as a front office technologist in a bank can be a nightmare

A career in banking technology can seem like a safe and secure path. - Less volatile than hedge funds with their hire and fire mentality, and much less likely to fail than fintech start-ups who rely almost solely on VC money. At a bank, however, you are typically dealing with enormous amounts of money. The revenue of a single trading desk or investment banking team is often far greater than the total cash-flow at a well-established FinTech. And when things go wrong, which they inevitably will, the losses can be eye-watering and sometimes career ending.

Everyone knows the infamous 2012 Knight Capital case, where $461m of losses stemmed from sending erroneous orders out to the market. This was caused by, among other things, an individual not copying new code to just one of eight production servers. A similar incident happened at Goldman Sachs in 2013 - the SEC report into it gives a fascinating insight into just how lax controls were in SecDB at the time. I have some inside knowledge of this incident - and four engineers were fired in pretty short order. Controls in SecDB were tightened up - whereby code could no longer be released without peer review. But just one year later, the old CTO Steve Scopellite was gone and the strats organisation had to all intents and purposes taken over technology. So, the message here is that even if you’re not directly responsible for something going wrong, it can still have massive consequences on careers.

Personally, I’ve released code with bugs that has cost a trading desk six figures through mis-pricing. The client in question probably couldn’t believe their luck. The first time you’re going through an incident like this it can be frightening, and therefore I’ve detailed below the steps you can take to manage your reputation and image. We all develop code with bugs - it’s unavoidable and particularly likely in a fast-paced environment where there are expectations around productivity. But you are also at the mercy of the wider change and release processes.

So, when you make a mistake in your banking technology job - which you absolutely will, there are a few things you need to do. Based on my own long experience in the industry, this is the playbook you need to follow.

1. Own up straight away. Don’t hide anything, you’ll only make it worse. The focus should be on escalating and getting help.

2. Try and lead the response. Fixing the issue in the timeliest manner possible, dropping everything else until this is done. Framing is important here - yes, you or your wider team caused the problem, but if you are instrumental in fixing it then that gets you some points back.

3. Over the course of the next few weeks, analyse and refactor the test suite of the application to try and avoid this type of error happening again. Metrics are useful here - if you introduce user interface tests, or significantly increase test coverage, then those are numbers you should give to the business and tech management.

4. If there was a control failure, lobby for a change in the process and implement it if possible.

5. If the business has lost millions, crack open your CV and start ringing around recruiters. Even if you don’t get fired, you have just lost so much career capital that it won’t be worth staying.

Joe Piccolo is the pseudonym of a technologist working at a leading U.S. bank

Have a confidential story, tip, or comment you’d like to share? Contact: sbutcher@efinancialcareers.com in the first instance. Whatsapp/Signal/Telegram also available.

Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)