Senior Data Engineer (Flink/Spark Streaming, Java)

This is a unique opportunity to build something from scratch but have the backing of an organization that has the muscle to take it to market quickly, with a
very satisfied customer base.

Responsibilities:

• Spearhead the design and implementation of ELT processes, especially focused on extracting data from and loading data into various endpoints, including RDBMS, NoSQL databases and data-warehouses.

• Develop and maintain scalable data pipelines for both stream and batch processing leveraging JVM based languages and frameworks.

• Collaborate with cross-functional teams to understand diverse data sources and environment contexts, ensuring seamless integration into our data ecosystem.

• Utilize AWS service-stack wherever possible to implement lean design solutions for data storage, data integration and data streaming problems.

• Develop and maintain workflow orchestration using tools like Apache Airflow.

• Stay abreast of emerging technologies in the data engineering space, proactively incorporating them into our ETL processes.

• Thrive in an environment with ambiguity, demonstrating adaptability and problem-solving skills.

Qualifications:

• BS in computer science or a related field.

• 5+ years of experience in data engineering or related field.

• Demonstrated system-design experience orchestrating ELT processes targeting data

• Hands-on experience with at least one streaming or batch processing framework, such as Flink or Spark.

• Hands-on experience with containerization platforms such as Docker and container orchestration tools like Kubernetes.

• Proficiency in AWS service stack.

• Familiarity with workflow orchestration tools such as Airflow.

• Experience with DBT, Kafka, Jenkins and Snowflake.

• Experience leveraging tools such as Kustomize, Helm and Terraform for implementing infrastructure as code.

• Strong interest in staying ahead of new technologies in the data engineering space.

• Comfortable working in ambiguous team-situations, showcasing adaptability and drive in solving novel problems in the data-engineering space.

Preferred

• Experience with AWS

• Experience with CICD

• Experience instrumenting code for gathering production performance metrics

• Experience in working with a Data Catalog tool ( Ex: Atlan / Alation)

What success looks like in the role

Within the first 30 days you will:

• Onboard and set up your development and test environment and get access to the data platform.

• Meet your team and key stakeholders and complete new-hire onboarding.

• Build understanding of the company, the team’s mission, and your role, and identify the team’s top priorities.

• Complete your first ticket and learn core workflows (e.g. version control and configuration).

• Complete relevant product and platform training.

• Shadow on-call and monitoring to see how the team supports production.

By 60 days:

• Contribute regularly by working on well-scoped tickets and documenting how similar work is done.

• Use team development practices (e.g. CI/CD, code review) and give or receive feedback on pull requests.

• Complete required security and secure-coding training.

• Build familiarity with the team’s data models, business logic, dashboards, and runbooks.

• Participate in secondary on-call and monitoring.

By 90 days:

• Take on more ownership: higher-point tickets, primary on-call, and helping improve runbooks and operations.

• Learn observability, data, and streaming concepts used by the team and document key components and their interactions.

• Share your experience (e.g. internal blog or project summary) and continue learning deployment and orchestration concepts relevant to the platform.

• Collaborate with support and engineering management to resolve escalations and improve how the team operates.