Design, develop, and optimize large-scale PySpark ETL/ELT pipelines on Databricks. Implement Delta Lake lakehouse architectures, tune Spark jobs, write complex SQL, monitor production, enable CI/CD, mentor junior engineers, and collaborate with stakeholders on scalable data solutions.
About the Job
About Aligned Automation
At Aligned Automation, we live by our "Better Together" philosophy to build a better world. As a strategic service provider to Fortune 500 companies, we help digitize enterprise operations and drive impactful business strategies. Our purpose goes beyond projects—we strive to deliver meaningful, sustainable change that shapes a more optimistic and equitable future.
Our culture is deeply rooted in our 4Cs—Care, Courage, Curiosity, and Collaboration—ensuring that each employee is empowered to grow, innovate, and thrive in an inclusive workplace.
Senior Data Engineer – PySpark
Experience: 7–8 YearsLocation: Pune (Work from office)
Job Summary
We are seeking a highly skilled Senior PySpark Data Engineer with 7–8 years of experience in designing, developing, and optimizing large-scale data engineering solutions. The ideal candidate should have extensive experience with PySpark, Python, SQL, Databricks, cloud platforms, and modern data architectures. The role requires hands-on development, solution design, client interaction, and mentoring junior team members.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using PySpark.
- Build high-performance data pipelines to process large volumes of structured and semi-structured data.
- Develop reusable frameworks for data ingestion, transformation, validation, and monitoring.
- Optimize Spark jobs by tuning partitions, joins, caching, and memory configurations.
- Design and implement Data Lake/Lakehouse architectures using Delta Lake.
- Write complex SQL queries, stored procedures, CTEs, and window functions.
- Collaborate with business stakeholders, architects, and data analysts to understand business requirements.
- Participate in architecture discussions and recommend scalable data solutions.
- Perform code reviews and enforce coding standards and best practices.
- Monitor production pipelines, troubleshoot issues, and implement performance improvements.
- Work with DevOps teams to implement CI/CD for data pipelines.
- Mentor junior engineers and provide technical leadership.
- Estimate effort, prepare technical documentation, and participate in Agile ceremonies.
Programming
- Python (Advanced)
- PySpark (Advanced)
- SQL (Advanced)
- Apache Spark
- Spark SQL
- Delta Lake
- Parquet
- iceberg
- IOMETE
- SQL Server
- PostgreSQL
- Azure SQL
- Azure
- AWS
- Databricks
- Apache Airflow
- Git
- Azure DevOps / GitHub
- CI/CD Pipelines
Similar Jobs
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Leads the design, development, integration, testing, and delivery of cardholder services applications. Provides hands-on technical leadership across Java and Spring Boot services, React or Angular applications, APIs, databases, event-driven integrations, and cloud platforms. Drives architecture decisions, technical evaluations, scalability, security, resiliency, CI/CD, production support, and engineering best practices while mentoring engineers and collaborating across Product, Engineering, Architecture, and QE teams.
Top Skills:
AngularApi GatewaysCi/CdCloud FoundryContainersGitHadoopJ2EeJavaKafkaKubernetesMicroservicesOracleOzonePlaywrightReactRest ApisSeleniumSpringSpring BootSQLTddWeb Services
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Provides Tier 1 and Tier 2 Jira Cloud support, including user onboarding, issue resolution, provisioning, migrations, demonstrations, and escalations. Partners with engineering and vendors to troubleshoot configurations, integrations, plugins, permissions, and data synchronization. Coordinates change planning, schema updates, data cleanup, security compliance, and documentation across global programs. Requires strong technical program management, stakeholder communication, Agile tooling expertise, and experience with enterprise migrations and dependency tracking.
Top Skills:
Aha!AtlassianConfluenceDomoEazybiJira CloudRallyRemedyXray
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Develops, tests, maintains, and troubleshoots enterprise data pipelines and analytical data solutions using Alteryx, Microsoft SQL Server, and Microsoft Fabric. The role focuses on data integration, transformation, quality, security, documentation, monitoring, operational resiliency, and cloud-native engineering patterns. It also involves collaborating with technical and business stakeholders to translate requirements into reliable, maintainable data products.
Top Skills:
AlteryxData FactoryData PipelinesLakehouseMicrosoft FabricMicrosoft Sql ServerOnelakePythonSQL
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
