Design, build, and optimize end-to-end data pipelines for large structured and unstructured data. Implement near-real-time ETL, data validation, monitoring, and performance optimization. Collaborate with stakeholders, document designs and workflows, and provide technical guidance to the team.
Location: Pune
Responsibilities include:- Design, implement, and optimize end-to-end data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data.
- Develop data pipelines to extract and transform data in near real-time using cloud-native technologies.
- Implement data validation and quality checks to ensure accuracy and consistency.
- Monitor system performance, troubleshoot issues, and implement optimizations to enhance reliability and efficiency.
- Collaborate with business users, analysts, and other stakeholders to understand data requirements and deliver tailored solutions.
- Document technical designs, workflows, and best practices to facilitate knowledge sharing and maintain system documentation.
- Provide technical guidance and support to team members and stakeholders as needed.
- 8+ years of work experience.
- Proficiency in writing complex SQL queries on MPP systems (Snowflake/Redshift).
- Experience in Databricks and Delta tables.
- Data engineering experience with Spark/Scala/Python.
- Experience in Microsoft Azure stack (Azure Storage Accounts, Data Factory, and Databricks).
- Experience in Azure DevOps and CI/CD pipelines.
- Working knowledge of Python.
- Comfortable participating in 2-week sprint development cycles.
Similar Jobs
Agency • Information Technology
Design, build, optimize, and maintain high-performance Spark-based data pipelines using Scala/Java and Hive on Hadoop/CDP. Own full project lifecycle, enforce coding best practices, troubleshoot Spark/Hive/YARN performance, and collaborate with stakeholders to deliver scalable data solutions.
Top Skills:
SparkCloudera Data Platform (Cdp)HadoopHiveJavaScalaYarn
Cloud • Information Technology • Productivity • Software • Automation
Provide advanced technical support for Boomi integrations: troubleshoot APIs, protocols, and logs; use tools like Postman, Wireshark, and Charles Proxy; work with Java/Groovy/JavaScript, Kubernetes, and enterprise systems (Salesforce, NetSuite, Hadoop); leverage AI for diagnostics and collaborate with customers and engineering teams to resolve complex integration issues.
Top Skills:
AIAi TechnologiesBoomi AtomsphereCharles ProxyEltETLGroovyHadoopHttp/HttpsJavaJavaScriptKubernetesLinuxNetSuiteOauth 2.0PostmanRancher DesktopReactRestSalesforceSftpSoapSsl/TlsTcp/IpWindowsWiresharkWsdl
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Lead clinical trial disclosure strategy and operations for Pfizer-sponsored interventional trials. Ensure timely, compliant posting of protocols, SAPs, CSRs, and clinical summaries per EMA Policy 70 and other regulations. Manage vendors, develop processes and technical solutions, represent Medical Writing on governance committees, and maintain regulatory knowledge and best practices to drive quality and consistency across disclosures.
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
