Big Data Engineering Lead
Big Data Engineering Lead
This is a lead engineering role within the team responsible for BGC Partners' Big Data platform requiring a candidate with strong Scala, Spark and Java skills. The successful candidate will report to the Head of Big Data and Messaging in London and be responsible for working with senior developers to take BGC's Big Data platform to the next level, supporting technology and business use-cases involving both on-premise and Cloud implementations. This is a new role within the London Big Data team requiring close collaboration with the business-focused lead developer who owns the data plant ETL framework. Responsibilities:
- Steer the development of the data plant through completion of its current phase leading to the development of a next generation platform to serve the expanding needs of the business.
- Integrate Data Warehousing and RDBMS solutions with the Big Data platform to enhance accessibility and analytical capabilities of the global platform.
- Work closely with senior developers to further the goals of the platform leadership team, including hands-on application of architecture, design, development and testing competencies.
- Ensure systems are resilient, highly availability and conformant to Business Continuity requirements.
- Produce estimates for platform enhancements.
- Creation/updating of system functional and reference documentation
- To identify any technical requirements for new products
- To work with all the business and technology departments to ensure all business and technical requirements are met
- To identify and manage any integration work and analyse, design and build any such projects
- To provide substantial input into the system architecture
- To liaise with other development teams as necessary to implement cross-team projects
- Be alert to Conduct Risk issues, specifically the risk of harm to client interests, market integrity and/or competition in financial markets due to inappropriate practices or behaviours across the firm
- Experience in leading hands-on development of Big Data platforms covering the following languages and technologies: Scala, Java, Hortonworks, Hive, Spark, SparkSQL, Kafka, RDBMS, ex PostGres, SQL Server, Oracle
- Excellent relational and NoSQL data modelling skills.
- Able to demonstrate effective engineering leadership building at-scale real-time data platforms with high availability and efficient distribution/accessibility of data sets to stakeholders.
- Communication skills required to navigate the needs of stakeholders whilst also being adept at dealing with application development issues.
- Experience of software development in a financial services environment
- Willingness to keep up to date with latest technology trends and proactively identify appropriate areas into which they can be applied
- Computing Degree
- Leading the integration and divesting of data services resulting from merger, acquisition and spin-off events.
- Addressing efficient data storage and querying against very large stores, distributed across a global organisation.
- Ansible and Docker
- Migrating on-premise platforms to cloud architectures.
- AWS: Data Pipeline, S3, EMR, Lamda, DynamoDB, RedShift.
- High-performance messaging middleware platforms, eg Tibco or Solace.
Applications for this role open on 11/11/19
Applications for this role close on 11/12/19
Salary level: competitive
BGC Partners is an equal opportunities employer