Data Pipeline Developer Data Pipeline Developer …

BGC Partners
in London, England, United Kingdom
Permanent, Full time
Be the first to apply
BGC Partners
in London, England, United Kingdom
Permanent, Full time
Be the first to apply
Data Pipeline Developer
Looking for a Senior Developer to work on the firm's Big Data analytics platform, with a specialism in Java or Scala required. The successful candidate will work in the data team to build out the existing platform to support additional technology and business use-cases involving both on-premise and Cloud implementations. The platform delivers data solutions across BGC Group including use-cases for: market data analytics, regulatory reporting, surveillance and revenue analysis.

  • To work in partnership with the business/business analysts to identify key requirements for implementation
  • To identify any technical requirements for new products
  • To work with all the business and technology departments to ensure all business and technical requirements are met
  • To identify and manage any integration work
  • To analyse, design and build any such projects
  • To provide input into the current and on-going system architecture
  • To liaise with other development teams as necessary to implement cross-team projects
  • Be alert to Conduct Risk issues, specifically the risk of harm to client interests, market integrity and/or competition in financial markets due to inappropriate practices or behaviours across the firm

Required Experience
Minimum of 3 years development experience working with Big Data platforms covering the following technologies:
  • Java or Scala on Linux. Docker beneficial
  • Hortonworks, Hive
  • Kafka, KafkaConnect, KafkaStreaming
  • Spark, SparkSQL, Yarn, Ansible

  • Substantial database experience: Relational and NoSQL data modelling.
  • Experience of software development in a financial services environment advantageous
  • Willingness to keep up to date with latest technology trends and proactively identify appropriate areas into which they can be applied
  • Solid Computing Degree

Desired Experience:
  • Cassandra / Datastax, ElasticSearch, LogStash, Kibana
  • Experience in addressing efficient data storage and querying against very large stores
  • Experience in accommodating key requirements of MIFID II in designs/implementations
  • AWS: Data Pipeline, S3, EMR, Lamda, DynamoDB, RedShift
  • Experience in the use of high performance messaging middleware such as Tibco RV or Solace

Personal attributes:
  • Team player with excellent inter-personal skills and confident communication skills.
  • Able to effectively disseminate knowledge and experience to less experienced team members.
  • Must be able to deal with our customers effectively, ie development teams working across the globe using Messaging and Data products, alongside external vendors who use the products.
  • Must be able to deal with and adapt to change extremely effectively.
  • Must be proactive in generating ideas and effective at developing solutions that are balanced, proportionate and effective. It is critical that this is achieved in collaboration with the global team.
  • Ability to work and function under pressure, handle multiple tasks, and shifting priorities
  • "Self-starter" always looking to improve quality of process and deliverables and keen to take a lead role in that process