The Senior Data Engineer will design and implement scalable ETL/data pipelines, manage data migrations to Snowflake, and oversee data processing for high-volume datasets while ensuring security and performance optimization.
Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making.
As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.
If your passion for data engineering solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering! 🌍🔥
Responsibilities:
- Lead the design and implementation of scalable ETL/data pipelines using Python for data processing.
- Ensure efficient data processing for high-volume clickstream, demographics, and business data.
- Lead the strategic planning, execution, and optimization of large-scale data migration initiatives to Snowflake, ensuring data integrity, security, and minimal business disruption.
- Use Snowflake components/tools - Snowpipe, SnowSQL, SnowPark etc
- Guide the team in adopting best practices for data pipeline development, code quality, and performance optimization.
- Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security.
- Support data storage and retrieval workflows using S3 and SQL-based storage solutions.
- Provide architectural guidance for cloud-native data solutions and infrastructure design.
- Oversee legacy framework maintenance, identify improvement areas, and propose comprehensive cloud migration or modernization plans.
- Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints.
- Develop and implement robust monitoring solutions to track system health, performance, and data pipeline accuracy.
- Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues.
- Lead efforts in incident response, root cause analysis, and post-mortem processes for complex data engineering challenges.
- Document workflows, troubleshooting procedures, and code for system transparency and continuity.
- Provide mentoring and training to team members on best practices and technical skills.
- Foster a culture of continuous learning, knowledge sharing, and technical excellence within the data engineering team.
Qualification
- Experience: 6 years of experience in data engineering or a related technical field, with at least 4 years in a Snowflake projects.
- Cloud Data Warehousing: Proven expertise with Snowflake data warehousing, including schema design, efficient data loading (e.g., Snowpipe, COPY into), performance tuning, and robust access control mechanisms.
- Programming & Scripting: Strong programming skills in Python for automation and data workflows.
- Data Processing & Storage: Expertise in managing SQL databases for data storage and query optimization.
- Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines.
- Experience with the AWS Cloud and data engineering services
Top Skills
Aws Ec2
Aws Emr
Aws Rds
Aws S3
Python
Snowflake
SQL
Similar Jobs
Artificial Intelligence • Cloud • Sales • Security • Software • Cybersecurity • Data Privacy
The Site Reliability Engineer will design, develop, and improve the reliability of SailPoint's IdentityNow services, focusing on automation, incident management, and mentoring teams in observability best practices.
Top Skills:
AWSGoGrafanaHoneycombJavaKibanaKubernetesPrometheusPythonTerraform
Artificial Intelligence • Cloud • Sales • Security • Software • Cybersecurity • Data Privacy
The Senior Software Engineer will develop Java and Golang SDKs, produce design documents, manage code quality, and lead projects while collaborating with stakeholders.
Top Skills:
AWSDockerEksGoGrafanaJavaJIRAKafkaKibanaNoSQLPrometheusRedisRestful ApisSpring BootSQL
Artificial Intelligence • Cloud • Sales • Security • Software • Cybersecurity • Data Privacy
The Senior Resource Manager will lead staffing, allocation, and utilization strategies, ensuring alignment with project needs and optimizing resource management for the professional services team.
Top Skills:
Certinia PsaResource Planning ToolsSharepointSmartsheet
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
