Metrics and Data Analytics Development Lead
- New York, NY, USA
- Permanent, Full time
- Morgan Stanley USA
- 13 Dec 18
Metrics and Data Analytics Development Lead
You will be leading the technical development effort for the Data Ingestion & Controls program.
In this role, your goal will be to build and own a comprehensive Data Control & Surveillance toolset that spans across LCG systems. This focus will be on developing Data Ingestion utilities, tools, and controls in both the Big Data and relational DB environments.
The candidate will be working on existing and new initiatives within the Metrics and Data Analytics application area with primary focus on standardizing processes and controls for ingesting data on our Big Data ("Caspian") platform. Initiatives include customizations for evolving business needs as well as architectural and infrastructure improvements. Ultimately the health of our data can be quantified with a Risk Assessment Scorecard by source, data element, or data concept. This can range from upstream/downstream processing of data, ETL, file handling, storage, entitlements, data masking, and lineage.
This role will be a Senior Developer/Lead position with hands-on data architect/development background. The candidate will be responsible for managing offshore resources as well as working in a horizontal organization leading cross-team resources. This role will establish best practices, achieving operational excellence and delivering projects on time.
LCG provides technology solutions in support of the identification and oversight of the Firm's Legal, Compliance and related franchise risks. This includes functions such as anti-money laundering, trade surveillance, fraud, employee registration, outside business interests, employee trading, as well as the Firm's regulatory requirements.
You will be accountable to the Legal & Compliance division (LCD), Global Financial Crimes, Operational Risk, Technology and Information Risk, and Fraud Operations business clients as well as the Technology teams that share the data across the LCG Super Department.
The Data team works in close partnership with the LCG department teams as a horizontal group aligned in accordance to their strategic data needs. In support of this, the Data team is responsible for the strategic efforts of a comprehensive relational database platform (e.g., LCD DataMart, LC Sales Practice, and RACER) as well as the Big Data/Analytic initiatives (e.g., Caspian) and data engineering tools. In addition, the team works closely with their upstream data providers and their technology teams in order to govern the RTB efforts for our clients.
· Lead and develop the Data Ingestion & Controls program. Data ingestion is comprised of ETL workflows (Informatica/BDE), Database development (DB2), Big Data development (Cloudera/Scala/Hadoop), batch stream processing (UNIX/Autosys), as well as dashboards (Tableau, Other).
· Develop tools for proactive and predictive support of LCG Data Governance program. Responsible for ensuring that the architecture, framework and standards are enforced.
· Leverage Cloudera Manager and Cloudera Navigator to integrate with Informatica BDE/Intelligent Data platform to drive Risk Assessment Scorecard process.
· Create data ingestion tools and job tracker daemons to determine deficiencies and controls (i.e. where new ETL objects are introduced but lacking Collibra elements, PanDQ rules, LCG Data Catalogue entries, DevOps controls/FileHandler jobs)
· Create real-time tool trackers and daemons to determine where there's holes/gaps on the platform in the areas of security, entitlements, storage, processing efficiencies, table optimization (i.e. incremental loads vs cache)
· Leverage Cloudera Data Encryption or determine masking guidelines for PII data. Create processes to protect the HDFS and data environments from data breaches (i.e. copy from PROD to non-PROD)
· Bachelor's degree in Computer Science, Software Engineering, Information Technology, or related field required.
· Overall 8+ year's technology experience desired. At least 5 years of experience as a developer with experience leading projects. Financial industry preferred.
· At least 5 years of T-SQL experience with the ability to write queries to perform data analysis. A preference is given to DB2 v10+ databases.
· Strong Experience in ETL Informatica/BDE, Cloudera Manager, Cloudera Navigator, Scala/Hadoop software libraries, Hue, HIVE, Impala, SQuirrel, DBArtisan software tools.
· Experience with BI/Data visualization tools - Tableau is preferred.
· UNIX and Autosys knowledge is preferred. Any JAVA experience is a plus.
· Strong collaboration and communication skills. Must be a team leader and able to shift gears based on business demand and competing priorities. Must be well organized and able to tailor communication based on the audience and level (clients versus technical staff versus technical management).