Senior/Lead Big Data Engineer
You will be part of the Data Engineering & Operations team.
You will be responsible for designing, building, automating and maintaining data pipelines used for data management and governance initiatives and activities. The Job
- Collaborate with data management and governance team and IT to analyse and translate requirements to design end-to-end data pipelines that supports data management processes and data quality assessment and monitoring.
- Responsible for building, deploying and managing these data pipelines and workflows. It involves data ingestion, metadata harvesting and data cataloguing, data discovery and profiling, data transformation and integration to Enterprise data governance (EDG) platform and reporting tool (Tableau).
- Responsible for developing and deploying Data Quality & Reference data dashboards using Tableau for management reporting and governance monitoring.
- Ensure best practices and standard development protocols are applied. Perform periodic review to ensure the performance of data pipelines remain efficient and sustainable.
- Develop , perform and coordinate testing by various stakeholders to ensure effective and quality deployment of data pipelines and dashboards.
- Perform ongoing monitoring of data pipelines and work with stakeholders to troubleshoot and performance tune.
- Create and maintain accurate and complete documentation of the EDG platform and the pipelines developed to ensure its currency.
- Maintain awareness of industry trends on regulatory compliance, emerging threats and appropriate steps to mitigate the risks; and
- Highlight any potential concerns/risks and proactively shares best risk management practices.
- Any other ad-hoc duties as assigned.