Design, develop, and optimize large-scale PySpark ETL/ELT pipelines on Databricks. Implement Delta Lake lakehouse architectures, tune Spark jobs, write complex SQL, monitor production, enable CI/CD, mentor junior engineers, and collaborate with stakeholders on scalable data solutions.
About the Job
About Aligned Automation
At Aligned Automation, we live by our "Better Together" philosophy to build a better world. As a strategic service provider to Fortune 500 companies, we help digitize enterprise operations and drive impactful business strategies. Our purpose goes beyond projects—we strive to deliver meaningful, sustainable change that shapes a more optimistic and equitable future.
Our culture is deeply rooted in our 4Cs—Care, Courage, Curiosity, and Collaboration—ensuring that each employee is empowered to grow, innovate, and thrive in an inclusive workplace.
Senior Data Engineer – PySpark
Experience: 7–8 YearsLocation: Pune (Work from office)
Job Summary
We are seeking a highly skilled Senior PySpark Data Engineer with 7–8 years of experience in designing, developing, and optimizing large-scale data engineering solutions. The ideal candidate should have extensive experience with PySpark, Python, SQL, Databricks, cloud platforms, and modern data architectures. The role requires hands-on development, solution design, client interaction, and mentoring junior team members.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
- Build high-performance data pipelines to process large volumes of structured and semi-structured data.
- Develop reusable frameworks for data ingestion, transformation, validation, and monitoring.
- Optimize Spark jobs by tuning partitions, joins, caching, and memory configurations.
- Design and implement Data Lake/Lakehouse architectures using Delta Lake.
- Write complex SQL queries, stored procedures, CTEs, and window functions.
- Collaborate with business stakeholders, architects, and data analysts to understand business requirements.
- Participate in architecture discussions and recommend scalable data solutions.
- Perform code reviews and enforce coding standards and best practices.
- Monitor production pipelines, troubleshoot issues, and implement performance improvements.
- Work with DevOps teams to implement CI/CD for data pipelines.
- Mentor junior engineers and provide technical leadership.
- Estimate effort, prepare technical documentation, and participate in Agile ceremonies.
Programming
- Python (Advanced)
- PySpark (Advanced)
- SQL (Advanced)
- Apache Spark
- Spark SQL
- Delta Lake
- Parquet
- iceberg
- IOMETE
- SQL Server
- PostgreSQL
- Azure SQL
- Azure
- AWS
- Databricks
- Apache Airflow
- Git
- Azure DevOps / GitHub
- CI/CD Pipelines
Similar Jobs
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Perform mechanical design, combustion and emissions analysis for aerospace engines, APUs, and advanced technology programs. Conduct CFD, thermal, and FEA analyses using engineering software; develop engineering tools and methods; support compliance, problem solving, and technical resolution. The role also involves combustion modeling, turbomachinery and gas turbine combustor analysis, and collaboration on project, process, and product requirements.
Top Skills:
Ai/MlCC++CfdFeaFortranJavaSiemens NxSiemens Star-Ccm+Synopsys FluentSynopsys WorkbenchThermal Analysis
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Performs fund accounting for global hedge fund, private equity, and debt fund clients. Responsibilities include managing complex fund structures, investor capital activities, trading and non-trading transactions, NAV workbooks, allocations, management fees, carried interest, performance ratios, investor reporting, and financial instruments. The role also involves client interaction, resolving accounting and operational issues, and working across global teams on UK shifts.
Artificial Intelligence • Consumer Web • Edtech • Enterprise Web • HR Tech • Social Impact • Generative AI
Build and scale production-grade agentic AI infrastructure and platform capabilities. Lead architecture, technical design, reliability, scalability, data privacy, and end-to-end delivery. Collaborate with product and innovation teams to define customer solutions, mentor engineers, improve developer experience, and resolve complex technical issues. The role requires backend engineering, cloud infrastructure, APIs, databases, automated testing, infrastructure as code, and CI/CD expertise.
Top Skills:
AWSAzureCi/CdDockerGCPGithub ActionsGoJavaJenkinsKafkaKubernetesLangfuseLanggraphMcpNode.jsNoSQLPythonRagRest ApisSQLTerraform
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
