• Design, build and maintain data processing ETL pipelines & data products in a multi-cloud, multi-region, distributed processing context choosing the right technologies.
• Drive data investigations to deliver a resolution of technical, procedural, and operational issues.
• Solve complex business problems by utilizing disciplined development methodology, producing scalable, flexible, efficient and supportable solutions using appropriate technologies.
Skills:
Data engineering patterns: Sound knowledge of the different data engineering patterns to determine what to use and when
Data architecture: Understanding of data architecture and frameworks like Data mesh, data fabric etc.
Big Data Technologies: Proficiency in big data technologies such as data lake, EMR, Glue for analysing large datasets.
Programming Languages: Proficiency in programming languages commonly used in data engineering, such as Python, Java, or Scala. with OOP expertise
Data Warehousing: Strong knowledge of data warehousing solutions preferably Amazon Snowflake.
ETL/ELT Tools: Familiarity with orchestration tools
Database Systems: Expertise in both relational and NoSQL databases.
Data Modelling: Skill in designing efficient data models, both for OLAP and OLTP systems.
Streaming Data: Knowledge of streaming data technologies.
Version Control: Experience with version control systems : Git.
Containerization and Orchestration: Understanding of containerization technologies (Docker) and container orchestration platforms (Kubernetes).
Data Security: Knowledge of data encryption, access control, and compliance with data privacy regulations.
Monitoring and Logging: Proficiency in setting up monitoring and logging solutions for data pipelines using different tools
