Lead the design and optimization of enterprise-scale data pipelines and architectures, ensuring high performance and collaboration with teams while implementing data governance frameworks.
Lead the design, development, and optimization of enterprise-scale data pipelines and architectures to drive data-driven decision-making. Architect efficient data tables and schemas to enhance query performance, ensuring seamless integration with cross-functional teams
Key Responsibilities - Data Pipeline
- Design and implement high-performance data pipelines using ETL processes, batch/streaming frameworks (e.g., Apache Spark, Airflow), and cloud platforms (AWS).
- Optimize data ingestion, transformation, and storage workflows to meet scalability, reliability, and latency requirements.
Key Responsibilities - Data Architecture & Query Efficiency
- Architect database schemas and dimensional models (e.g., star/snowflake) to maximize query performance and reduce latency.
- Implement indexing strategies, partitioning, and materialized views to optimize data retrieval.
Key Responsibilities - Collaboration & Governance
- Partner with data scientists, product managers, and engineers to align data infrastructure with business needs.
- Establish data governance frameworks, ensuring compliance with security, privacy, and quality standards.
- Mentor junior engineers and foster best practices in data engineering
Key Responsibilities - Innovation & Leadership
- Stay ahead of industry trends (e.g., AI/ML integration, real-time analytics) and advocate for emerging technologies.
- Lead technical strategy for data infrastructure, balancing innovation with operational stability.
Technical Expertise
- 8+ years of experience in data engineering, with a focus on ETL, data modelling, and cloud-based architectures.
- Proficiency in SQL, Python, and tools like Spark and Snowflake.
- Strong knowledge of database design, dimensional modeling, and query optimization.
Leadership & Collaboration
- Proven ability to lead teams, mentor engineers, and drive technical initiatives.
- Experience translating business requirements into scalable technical solutions.
Preferred qualifications
- Expertise in machine learning model deployment or real-time analytics.
- Familiarity with DevOps practices (CI/CD, Docker, Kubernetes).
Our Core Values
- Game Changers- A game changer is someone who looks at problems with an open mind and shares new ideas with team members, regularly reassesses existing plans and attaches a realistic timeline to goals, makes profitable, productive, and innovative contributions, and actively pursues improvements to Pattern’s processes and outcomes.
- Data Fanatics- A data fanatic is someone who recognizes problems and seeks to understand them through data, draws unbiased conclusions based on data that lead to actionable solutions, and continues to track the effects of the solutions using data.
- Partner Obsessed- An individual who is partner obsessed clearly explains the status of projects to partners and relies on constructive feedback, actively listens to partner’s expectations, and delivers results that exceed them, prioritizes the needs of your partners, and takes the time to create a personable experience for those interacting with Pattern.
- Team of Doers- Someone who is a part of a team of doers uplifts team members and recognizes their specific contributions, takes initiative to help in any circumstance, actively contributes to supporting improvements, and holds themselves accountable to the team as well as to partners.
Pattern is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
Top Skills
Airflow
Spark
AWS
Docker
Kubernetes
Python
Snowflake
SQL
Similar Jobs
Fintech • Financial Services
The Principal Data Engineer leads the design and implementation of data solutions on AWS, mentors teams, and aligns data initiatives with organizational goals by utilizing advanced security analytics and AI workflows.
Top Skills:
Apache AirflowAWSDockerJenkinsKubernetesPysparkPythonSparkSQLTerraform
AdTech
Lead a team of data engineers, oversee data projects, ensure data integrity, and implement ELT processes to support data-driven decision-making.
Top Skills:
AWSDbtPostgresPythonSQL
Angel or VC Firm
Lead and mentor a team of data engineers, oversee data projects, ensure data quality, and collaborate with cross-functional teams.
Top Skills:
AWSDbtPostgresPythonSQL
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
