Leads the design, development, and maintenance of scalable ETL/ELT pipelines and data products across distributed, multi-cloud environments. Defines data engineering architectures, resolves complex production issues, and designs efficient data solutions. Provides technical leadership and mentorship while collaborating with architects, engineers, and business stakeholders. Requires expertise in big data, AWS services, data warehousing, databases, data modeling, orchestration, streaming, security, observability, version control, containers, and modern development practices.
Job Title Lead Data Engineer Location (s) Cochin Years of Experience 5-10yrs Job Description
Role & Responsibilities Technical Qualifications Python + AWS EMR + AWS Glue + Snowflake + ETL/ELT + Data Engineering + Data Architecture + Big Data + SQL
Lead Data Engineer
What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Role & Responsibilities
- Lead Data Engineer
- What You’ll Do
- Lead the design, development, and maintenance of scalable ETL/ELT pipelines and data products in multi-cloud, multi-region, distributed environments.
- Define and apply appropriate data engineering patterns and architectures based on business and technical requirements.
- Drive technical investigations and resolve complex data, operational, and production issues.
- Design scalable, flexible, efficient, and supportable solutions using appropriate technologies and disciplined development practices.
- Provide technical leadership, guidance, and mentorship to data engineering teams.
- Collaborate with architects, engineering teams, and business stakeholders to deliver robust data solutions.
- Key Skills
- Data Engineering & Architecture: Strong understanding of data engineering patterns and modern architectures such as Data Mesh, Data Fabric, Data Lake, and Data Warehouse.
- Big Data & Cloud: Proficiency in Amazon EMR, AWS Glue, Data Lake, and technologies for processing large datasets.
- Programming: Strong proficiency in Python, Java, or Scala, with solid OOP expertise.
- Data Warehousing: Strong knowledge of data warehousing solutions, preferably Snowflake.
- ETL/ELT & Orchestration: Hands-on experience with ETL/ELT pipelines and data orchestration tools.
- Databases: Strong expertise in relational and NoSQL databases.
- Data Modelling: Experience designing efficient OLAP and OLTP data models.
- Streaming: Knowledge of streaming data technologies and architectures.
- Version Control: Experience with Git and modern development practices.
- Containers & Orchestration: Understanding of Docker and Kubernetes.
- Data Security: Knowledge of encryption, access control, data privacy, and compliance.
- Monitoring & Logging: Experience implementing monitoring, logging, alerting, and observability for data pipelines.
Similar Jobs
Retail • Manufacturing
Own Azure Data Factory pipelines, Salesforce and SAP integrations, KPI logic, dashboards, semantic definitions, validation rules, and reconciliations for trade promotion and sales data. Investigate discrepancies, monitor feeds, support platform changes, review AI-generated financial figures, and improve data quality. Serve as the India-hours escalation point while collaborating globally, guiding junior colleagues, and partnering with commercial, finance, IT, and vendor teams.
Top Skills:
Azure Data FactoryCrm AnalyticsPbcsS/4HanaSalesforceSalesforce Consumer Goods CloudSalesforce Trade Promotion ManagementSAPSQL
Aerospace • Security • Energy • Industrial
Designs and maintains scalable batch and streaming data pipelines in Databricks using Spark, PySpark, SQL, Delta Lake, and medallion architecture. Builds data models, automates workflows, develops Power BI reports, and provides statistical analysis, KPI tracking, financial and operational reporting, and business insights. The role coordinates with engineering, product operations, finance, and program leaders, supports automation and AI agent initiatives, mentors junior colleagues, and helps improve program performance and business efficiency.
Top Skills:
Ai AgentsSparkAuto LoaderDatabricksDatabricks JobsDatabricks WorkflowsDelta LakeLakeflow Declarative PipelinesMedallion ArchitectureExcelMicrosoft OutlookMicrosoft PowerpointMicrosoft ProjectPower BIPysparkPythonSpark SqlSQL
Aerospace
Designs and maintains scalable batch and streaming data pipelines in Databricks using Spark, PySpark, SQL, Delta Lake, and medallion architecture. Builds data models, automates workflows, develops Power BI reports, and delivers operational, financial, cost, quality, and schedule analytics. Partners with engineering, product operations, project management, and business leaders to support planning and performance decisions. The role also mentors junior colleagues, oversees work, performs statistical analysis, and drives process automation and efficiency improvements.
Top Skills:
Ai AgentsSparkAPIsAuto LoaderAutomated TestingDatabricksDatabricks JobsDatabricks WorkflowsDelta LakeLakeflow Declarative PipelinesMedallion ArchitectureExcelMicrosoft OutlookMicrosoft PowerpointMicrosoft ProjectPower BIPysparkPythonSpark SqlSQL
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.



