Lead a team of three data engineers supporting a large-scale analytics platform. Manage and optimize ETL pipelines, data transformations, monitoring, reliability, production incidents, and SLA-driven enhancements. Collaborate with business advisory teams and maintain operational runbooks and documentation. Participate in a 24x7 P1 incident rotation, including weekend and night on-call coverage, and work from the Mumbai client site as required.
About the Role:
We are looking for a Data Engineering Lead to join our AMS team supporting a large scale analytics platform for a major Oil & Gas enterprise. You will lead the data engineering track, managing ETL pipelines, data transformations, and ensuring platform reliability.
Key Technologies:
• AWS Glue, AWS Lambda, Amazon Redshift, Aurora PostgreSQL
• MSK Kafka, AWS SQS, AWS Transfer Family
• Apache Airflow (orchestration)
• SQL, Python
What You'll Do:
• Lead a team of 3 data engineers in maintaining & enhancing large-scale data assets
• Own ETL pipeline performance, monitoring & optimization
• Handle production incidents (24x7 rotation for P1 issues)
• Drive enhancements within defined SLA timelines
• Collaborate with client business advisory team
• Create & maintain runbooks and documentation
Requirements:
• 8+ years in Data Engineering with leadership experience
• Expert in AWS data services (Glue, Redshift, Aurora)
• Strong SQL and Python skills
• Experience managing production data pipelines at scale
• Ability to work from Mumbai (client site as required)
• Available for weekend/night on-call rotation
Nice to Have: - Experience in Oil & Gas domain / large enterprise analytics
Similar Jobs
Cloud • Information Technology • Consulting
Design and scale data infrastructure for agentic AI, including RAG pipelines, vector databases, semantic layers, knowledge graphs, and automated data-quality guardrails. Build and maintain data pipelines, ETL processes, and CI/CD infrastructure while resolving source-data issues and protecting against noise, bias, and PII. The role also requires expertise with ELK, Python, relational and NoSQL databases, data modeling, and collaboration with technical and business stakeholders.
Top Skills:
AWSCi/CdDb2ElasticsearchElkETLGitGCPKibanaKnowledge GraphsLogstashAzureMilvusMongoDBNosql DatabasesPineconePostgresPythonRagRelational DatabasesVector EmbeddingsVisual StudioWeaviate
Artificial Intelligence • Consulting
Owns technical direction for enterprise agentic AI solutions across application, cloud, data, and AI layers. Leads architecture decisions, deployment standards, security reviews, reusable infrastructure, CI/CD practices, and production monitoring. Mentors Forward Deployed Engineers toward independent delivery, guides design and code reviews, prototypes high-risk components, and resolves critical blockers. The role emphasizes technical leadership, scalable deployment capability, cloud cost discipline, agent evaluation, guardrails, and operational reliability rather than sustained feature development.
Top Skills:
AiopsAPIsCi/CdDatabricksDevOpsInfrastructure As CodeJavaScriptAzureNoSQLPysparkPythonSQLTypescript
Information Technology • Consulting
Lead end-to-end delivery of enterprise data engineering workstreams, designing and operating scalable batch and streaming pipelines on AWS, Databricks, and Spark. Provide technical governance, design oversight, stakeholder leadership, and mentoring. Build transformations with dbt, orchestrate workflows with Airflow, and ensure systems are secure, tested, observable, reliable, and cost-efficient. Troubleshoot production issues, optimize performance and data quality, and contribute to estimation and commercial discussions.
Top Skills:
Apache AirflowSparkAWSCi/CdData Quality FrameworksDatabricksDbtJavaMonitoring And Observability ToolingPostgresPythonRelational DatabasesSQLSQL ServerTesting Frameworks
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
