Senior Data Engineer JOB

Job Description

Job Title: Senior Data Engineer

Sector: Strategic Affairs

Department: Business Intelligence Department

Section: Enterprise Data Management

JOB PURPOSE

The Senior Data Engineer role will be responsible for strengthening the Data Management practice within the department. The role will help deliver large and small data warehouse, Big Data, and Data Lake projects to bring up the data management maturity curve in the organization.

Also, the role will propose and enhance data exchange protocols, fix templates for data exchange, propose ETL/ELT and data ingest/modeling/storage best practices, focus on automation, and overall strengthen the enterprise data practice in DCT.

Key Accountabilities

Data Transformation Needs

Design and build the data warehousing systems to store the data safely and securely as per the desired specifications defined by the organization.

Analyse and identify the organization’s data storage requirements and implement the best approach to data storage

Manage and oversee the process of extracting, transforming, and loading datasets into the data warehouse. To be responsible for gathering and processing data into a unified format and standard, and subsequently, loading this data into the warehouse.

Manage data projects from start to finish, including prioritization, resource allocation, and risk mitigation

Consult IT & BI teams to get a big-picture idea of the organization’s data transformation needs in order to design appropriate data solutions to effectively meet business needs. 

Improve the engineering efficiency: Extensible, reusable, scalable, updateable, maintainable, virtualized traceable data and code would be driven by data engineering innovation and better resource utilization.

Smartly engineering DCT data at scale, as one unified asset, to unlock the value of our unique collection of data and predictions in real time.

Design, build, and maintain batch or real-time data pipelines in production. 

Automate data workflows such as data ingestion, aggregation, and ETL/ELT processing. 

Develop ETL/ELT (extract, transform, load) processes to help extract and manipulate data from multiple sources. 

Build, maintain, and deploy data products for analytics and data science teams on cloud platforms (e.g. AWS, Azure)

Data Integration Process

Present data warehousing options based on cloud, hybrid, and on-premise storage needs.

Conduct preliminary testing of the warehousing environment before the data extraction process.

Extract organization data and transfer it into the new warehousing environment as per the target data warehouse.

Test the new storage system once all the data has been transferred in order to ensure that the system is working efficiently.

Troubleshoot any issues and inefficiencies that may arise and provide support in implementing appropriate solutions.

Maintenance & Support

Support the team to identify opportunities for increasing the scale and automation of data collection to increase overall efficiencies.

Support the team in ensuring easy database access and overall database security.  

Continuously provide maintenance and support to all deliverables of the Enterprise Data Management team.

Debug and rectify any problems that arise with the data storage systems

Monitor data systems performance and implement optimization strategies.

Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.

Maintain and update knowledge of the latest technologies, tools, and methodologies

Reports and Documentation

Create documentation and prepare reports in relation to data integration and data management, as required.

Policies, Processes, and Procedures

Follow all relevant section policies, processes, procedures, and instructions so that work is carried out in a controlled and consistent manner.

Collaboration

Collaborate with internal and external stakeholders on matters related to the section in order to facilitate the flow of information to build awareness in those areas.

Coach data scientists and analysts on best practices

Day-to-Day Operations

Follow the day-to-day operations set by the Line Manager in the section to ensure continuity of work and the delivery of effective and high-quality outputs.

Report on a regular basis to the Line Manager on operational activities, challenges, hurdles and methods of resolution or mitigation etc. as required to keep the Line Manager informed and updated about the unit’s activities.

EHS (Environment, Health, and Safety)

Comply with all relevant EHS guidelines, policies, and procedures, by reporting incidents and hazards in a timely manner, and reducing consumption of natural resources to support protecting the environment and ensure a healthy and safe work environment.

Change Management

Support the creation of a culture susceptible to change management through a ‘hands-on’ and ‘can-do’ approach to DCT’s new business opportunities, participating in the development of new initiatives, meeting planned targets, and demonstrating preferred high-performance behaviours.

COMMUNICATION & WORKING RELATIONSHIPS

Internal

External

Staff within all sectors/departments in DCT

Data providers & data subscription providers

Technology vendors

Other government organization

QUALIFICATIONS, EXPERIENCE, COMPETENCIES

Qualification (e.g. Academic Qualification, Certifications, Licenses)

Bachelor and Master’s degree in Computer Science, Information Technology, Information Systems, Business Intelligence, Machine Learning, Engineering, or another equivalent field

Certification in Data Warehousing & Informatica is highly preferred

Certified Azure Data Engineer is highly preferred

Experience

At least 8+ years of experience in data warehousing & API management, or an equivalent experience within similar roles.

Strong programming experience in Python, SQL, Scala, and/or Go is required.

Advanced experience with SQL database design

Advanced technical expertise with data models, data mining, and segmentation techniques

Experience implementing low latency public APIs (REST architecture, Fast API preferred).

Advanced experience with Azure Data cloud-based platform

Advanced experience in building and optimizing ‘big data’ data pipelines, architectures, and data sets.

Advanced experience in performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

Strong analytic skills related to working with unstructured datasets.

Advanced experience in building processes supporting data transformation, data structures, metadata, dependency, and workload management.

Advanced experience in manipulating, processing, and extracting value from large disconnected datasets.

Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.

Strong project management and organizational skills.

Strong data modeling, data mining, and data warehousing and/or analytics experience

Excellent attention to detail – Ability to manage multiple projects and meet demanding deadlines

Experience in supporting and working with cross-functional teams in a dynamic environment.

Key skills

Functional Competencies

Marketing data reporting & analytics

Preferred

Tourism data reporting & analytics

Preferred

Domain knowledge of a CRM function

Preferred

Working in Microsoft Azure Cloud Platform and data tools

Must

End to end API Management from sourcing data from APIs to building data exports in APIs

Must

Analytical skills and problem solving abilities are required to interpret data flows

Must

Drive enterprise data decision making processes and facilitate discussion to address conflicting viewpoints with the goal of arriving at mutually satisfactory agreements

Preferred

Should demonstrate a good understanding of the enterprise direction, processes, rules, requirements, and deficiencies and work around these to implement Data Governance

Preferred

Core Competencies – Subject Matter Expertise

Worked as an ETL/ELT developer in a project that involved sourcing data through the integration of APIs from Social Media platforms such as Facebook, Twitter, Google Analytics, etc.

Must

Hands-on ETL/ELT developer on at least 3 large to medium scale end to end Data warehouse implementation projects

Must

Enterprise Data Warehouse, Big Data, Data Lake, Data Modelling, data taxonomy, Data Lineage, Reference and Master data management

Must