Senior Advisor - Big Data, Analytics and ML

 We are seeking a highly skilled and experienced Senior Advisor – Data Engineering with 10 plus years of experience to lead the design, development, and implementation of end-to-end software and data engineering solutions. In this role, you will be at the forefront of building scalable, high-performance systems for data processing, storage, and analytics while maintaining best practices in software engineering.  

As a Senior Advisor, you will guide the technical direction of both software and data projects, mentor junior engineers, and ensure that data engineering solutions are efficient, reliable, and align with business goals. You will work across cloud technologies (primarily Google Cloud Platform) and be responsible for building data pipelines, orchestrating workflows, and leveraging cloud-native tools for modern data solutions.  

Key Responsibilities:  

Solution Design & Architecture: 

• Lead the design and architecture of complex software and data engineering solutions, ensuring they are scalable, reliable, and secure. 
• Build and maintain end-to-end data pipelines for efficient data ingestion, processing, and transformation in cloud environments (primarily Google Cloud). 
• Design and implement cloud-based data solutions using Cloud Run, Cloud Functions, Big Query, Datastore, GCS, Dataplex, Pub/Sub, Cloud Scheduler and Composer. 
• Ensure data quality, consistency, and integrity across data sources and throughout the ETL pipeline. 
• Implement data governance best practices, ensuring compliance, security, and data privacy in the processing of sensitive information.  

Software Engineering & Development: 

• Contribute to the development of scalable, maintainable, and efficient software applications in Python, Scala, Spark. 
• Write code to integrate various data systems and ensure seamless data flow between storage, processing, and analysis layers. 
• Develop and optimize SQL queries, Stored Procedure for Big Query to support efficient data retrieval and analytics. 

Mentorship & Team Guidance: 

• Mentor junior software and data engineers, providing guidance on technical aspects (e.g., data pipeline design, cloud technologies, OOP principles). 
• Provide technical leadership in both software engineering and data engineering, helping to define best practices and standards for the team. 

Collaboration & Communication: 

• Work closely with cross-functional teams to define requirements and deliver high-quality solutions. 
• Communicate technical solutions clearly and concisely 

Code Quality & Best Practices: 

• Promote a culture of clean, maintainable code and ensure best practices for software and data engineering are followed. 
• Implement design patterns and architectural principles in both software and data engineering projects to ensure long-term maintainability. 

Continuous Improvement: 

• Stay up to date with emerging trends in both software/data engineering to bring new tools, technologies, and approaches to the team. 
• Advocate for continuous improvement in the development and deployment processes, particularly around data management, cloud services, and automation.  

Required Skills & Qualifications:  

Proven Expertise in: 

• Object-Oriented Programming (OOP) and software design principles. 
• Python for building cloud-native applications, data processing, and automation. 
• Extensive experience with Google Cloud Platform (GCP), particularly with BigQuery, Cloud Functions, Cloud Run, Datastore, GCS, Pub/Sub, Cloud Scheduler, Dataplex, and Composer. 
• Strong expertise in data engineering, including building ETL pipelines, data warehousing, and data storage solutions. 
• Proficiency in PostgreSQL, MS-SQL, and designing relational database solutions. 
• Familiarity with Apache Airflow for workflow orchestration and managing data pipelines. 
• Ability to design scalable data architectures and work with both structured and unstructured data. 

Architectural Skills: 

• Deep understanding of design patterns and their application to both software engineering and data engineering solutions. 
• Experience designing and implementing large-scale data systems and distributed architectures. 

Mentorship & Leadership: 

• Proven experience mentoring and guiding junior engineers  

Nice to Have:  

• Experience with Apache Spark and Scala for large-scale data processing. 
• Exposure with Azure cloud services.

Education:

• Bachelor's in computers science or related.