Big Data Engineer

  • Negotiable
  • New York, NY, USA
  • Permanent, Full time
  • GMS Advisors
  • 15 Dec 17 2017-12-15

Big Data Engineer on new initiative to build a large consolidated audit trail system.

This company is a market structure technology provider focused on delivering high performance trading technologies and big data solutions to financial market participants.


Their technology handles approximately 5% of the daily volume in U.S. equities & 10% of Canadian equities on the back of their ultra low latency HFT offering and their advanced gateway technology for Broker-Dealers. They also provide exchange / matching engine technology, and currently run two U.S. equities dark pools. The third pillar of their business is a Financial Big Data platform which powers the SEC’s MIDAS data and analytics system and the recently awarded Consolidated Audit Trail system.


They are looking for a self-motivated contributor who is passionate about supporting and building high-quality tools and systems. This person will help develop a new product in the domain of big data analytics on financial data. The role involves combining cutting-edge open source and proprietary technology to support back-end systems written in Apache Spark, Python, C++ and front-end systems using React. The role will also help improve supporting tools and infrastructure, create fast build systems, and improve workflows, with a persistent emphasis on pushing the limits of scalability and performance.




•   Develop automation infrastructure for deploying multiple environments

•   Using dynamic programming languages, such as Python and Linux command Language to architect, implement and integrate data analytics tools for financial markets

•   Proactively assisting other developers in diagnosing and solving issues with development and build environments

•   Establishing and improving infrastructure in support of production, QA, and development systems

•   Implementation of system monitoring infrastructure and tools, including dashboards

•   Monitoring systems for issues in real-time, responding to them as they occur, and working proactively to minimize future issues

•   Level 3 support




•   5+ years of experience building large scale big data applications

•   Proactive with a high attention to detail and ability to manage multiple projects simultaneously

•   Strong working knowledge of the Linux operating system, including basic system administration and a high degree of comfort working from the command line

•   Experience with version control software and SDLC principles

•   Strong programming skills, including experience with development in dynamic programming languages, strong knowledge of Python

•   Experience with Open Source software including Puppet, Jenkins, Nagios, GNU make, cmake, Redmine, OpenLDAP, FreeRADIUS, BIND, collectd, Graphite, Grafana, Hypertable, and PostgreSQL

•   Experience with containers, orchestration or CI/CD pipelines is a bonus

•   Financial experience is not required