Reference Data Engineer

POSITION SUMMARY

We are seeking an Assistant Vice President (AVP), Software Engineering to join our Operations Systems team. This role blends senior individual contribution with emerging technical leadership: you will design and deliver robust backend services and data pipelines while helping guide the technical direction of your team. The ideal candidate is a strong Python developer with hands-on experience in distributed data processing (Spark), infrastructure-as-code (Terraform), and deep data engineering fundamentals spanning data modeling, ETL, and data distribution. You will own significant workstreams end-to-end, raise the engineering bar through code reviews and design input, and mentor junior engineers as you continue to grow toward a lead or vice president level.

KEY RESPONSIBILITIES
Design, build, and own scalable backend services and data pipelines primarily in Python.
Develop and optimize large-scale data processing workflows using Apache Spark for batch and near-real-time use cases.
Lead data modeling and schema design decisions that support reliable analytics and downstream consumption.
Architect and maintain robust ETL/ELT workflows with strong data quality, lineage, and observability.
Design and implement data distribution patterns to serve data efficiently across internal systems and consumers.
Provision and manage cloud infrastructure using Terraform and infrastructure-as-code best practices.
Own delivery of significant workstreams end-to-end, from technical design through production and operations.
Drive engineering quality through design reviews, code reviews, and clear technical standards.
Mentor and support the growth of analysts and junior engineers on the team.
Partner with product, data, and platform leaders to shape roadmap and translate requirements into scalable solutions.
Lead root-cause analysis for complex production issues and drive continuous improvements in reliability and performance.

QUALIFICATIONS
Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
4–7 years of professional software engineering experience with a strong backend focus.
Expert-level proficiency in Python for building production services and data pipelines.
Strong hands-on experience with Apache Spark for distributed data processing at scale.
Proficiency with Terraform for infrastructure-as-code and cloud resource management.
Deep data engineering fundamentals, including data modeling, ETL/ELT design, and data distribution patterns.
Solid command of relational and non-relational data stores and SQL.
Experience with version control (Git), CI/CD pipelines, and modern software development practices.
Demonstrated ability to own complex workstreams and mentor other engineers.
Strong communication skills and the ability to collaborate across technical and business teams.

PPREFERRED QUALIFICATIONS
Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
Familiarity with workflow orchestration tools (e.g., Airflow, Dagster).
Experience with streaming technologies (e.g., Kafka, Spark Structured Streaming).
Exposure to containerization and orchestration (Docker, Kubernetes).
Prior experience in financial services or another data-intensive, regulated industry.
Experience with Palantir Foundry or similar data integration and analytics platforms.