Principal Software Engineer (Backend & Product Systems)

Job description

What You’ll Be Working On The engineering organization builds the backend systems responsible for ingesting healthcare information, processing documents, coordinating workflows, integrating with external systems, and automating operational tasks. As a Principal Software Engineer, you will help design and improve systems involving: Customer-facing healthcare workflow automation High-volume document ingestion and processing pipelines Distributed backend services and asynchronous workflows Event-driven systems coordinating complex operational processes APIs and integrations with healthcare platforms and external services Electronic health record synchronization Reliable workflow orchestration, retries, and state management Document classification and structured information extraction Patient communication, intake, referral, and scheduling workflows AI-assisted automation and LLM-powered product capabilities Platform scalability, observability, and production reliability Backend architecture supporting continued customer and product growth About the Role Our client is hiring a Principal Software Engineer to help architect, build, and scale the backend systems powering its healthcare workflow automation platform. This is a deeply hands-on individual contributor position for an experienced backend engineer who can make high-impact architectural decisions while continuing to write production software. You will work across distributed systems, workflow orchestration, APIs, integrations, document-processing pipelines, data models, and customer-facing product infrastructure. The ideal candidate has spent their career building and scaling software products rather than focusing primarily on internal infrastructure, DevOps, data platforms, or experimental AI systems. You should be comfortable taking ownership of technically ambiguous problems, evaluating architectural tradeoffs, and designing systems that remain reliable as the product, engineering organization, and customer base grow. This role requires more than strong implementation skills. Our client is looking for someone who can understand how the entire product operates, identify the architectural decisions that matter most, and guide those decisions from design through production. You will have significant influence over the future direction of the platform while remaining closely involved in building it. What You Will Do Architect scalable backend systems supporting customer-facing healthcare products and workflow automation Lead major technical initiatives from early discovery and system design through implementation, launch, and continued production improvement Design distributed services and event-driven workflows that process healthcare documents, data, and operational tasks Build reliable systems for workflow orchestration, asynchronous processing, retries, failure recovery, and state management Improve the scalability, performance, reliability, and maintainability of the existing platform Design and maintain APIs used by customer-facing applications, healthcare integrations, and internal services Develop systems that integrate with electronic health records and other external healthcare platforms Build and improve high-volume document ingestion, classification, processing, and extraction pipelines Evaluate when existing systems should be extended, redesigned, simplified, or replaced Make thoughtful architectural tradeoffs involving product velocity, reliability, scalability, complexity, and long-term maintainability Identify bottlenecks, failure modes, and architectural risks before they affect customers or slow product development Improve observability, monitoring, tracing, and production debugging across backend services and workflows Establish durable engineering patterns for service boundaries, APIs, event processing, data consistency, and failure handling Work with technologies such as Ruby, Python, Go, JavaScript, MySQL, Kubernetes, and Google Cloud Platform Support AI-assisted product capabilities and LLM integrations as components within broader customer-facing workflows Collaborate closely with Engineering, Product, Operations, and Executive Leadership to translate healthcare problems into scalable technical solutions Communicate architecture decisions, technical risks, and engineering tradeoffs clearly to technical and nontechnical stakeholders Provide technical guidance and mentorship to other engineers while remaining deeply hands-on Raise the engineering standard for backend architecture, systems design, code quality, and technical ownership Proactively identify important technical problems and take ownership of solving them with minimal oversight What Our Client Is Looking For Extensive software engineering experience building production backend applications and customer-facing software products Principal, Senior Staff, Staff, or equivalent individual contributor experience Significant ownership of backend architecture across complex production systems Experience personally designing and building distributed systems from the ground up Strong experience scaling software platforms through meaningful customer, transaction, or data growth Experience with event-driven architectures, asynchronous processing, queues, and workflow orchestration Strong understanding of APIs, service boundaries, data models, state management, and distributed system failure modes Experience designing systems for idempotency, retries, fault tolerance, consistency, and observability Strong experience with one or more backend languages such as Ruby, Python, Go, Java, or a comparable language Experience working with relational databases such as MySQL or PostgreSQL Experience with cloud platforms and containerized production environments Experience building and operating services using Kubernetes or similar orchestration technologies Ability to independently evaluate architecture options and clearly explain the reasoning behind technical decisions Experience operating systems after launch and improving them based on real production behavior Strong product instincts and an understanding of how technical decisions affect users and business outcomes Experience working closely with Product and business stakeholders in addition to engineering teams Strong written and verbal communication skills Startup, scale-up, or high-growth technology company experience preferred Ability to work collaboratively in a hybrid New York City environment Principal-Level Expectations You have personally architected and built complex backend systems used by real customers You have owned major technical initiatives from initial design through implementation, launch, and long-term production support You can translate an ambiguous product or business problem into a scalable technical architecture You understand how individual services, workflows, integrations, and data models fit into the broader product platform You have made architectural decisions that materially improved the direction, scalability, or reliability of a software product You can recognize when a system requires a new architectural approach and when a simpler incremental improvement is the better decision You make thoughtful tradeoffs between delivery speed, system complexity, scalability, reliability, and maintainability You can design dependable systems that account for failures, retries, partial processing, duplicate events, and unavailable dependencies You remain deeply hands-on and are comfortable contributing production code alongside other engineers You influence engineering direction through technical judgment, communication, and execution rather than relying on organizational authority You can identify technical risks and opportunities before they are formally assigned You are comfortable operating in an environment where requirements and priorities evolve as the company grows You can communicate complex system designs clearly during architecture discussions and technical reviews You understand that strong architecture should improve both the customer experience and the engineering team’s ability to deliver Modern Backend Engineering Approach Strong software engineering fundamentals and a production-first mindset Product-oriented thinking focused on building capabilities customers use and depend on Pragmatic approach to languages, frameworks, infrastructure, and architectural patterns Ability to balance long-term architecture with the practical needs of a growing company Strong understanding of distributed system design and production failure modes Preference for clear service boundaries, reliable workflows, and understandable operational behavior Experience using automated testing, monitoring, tracing, and observability to improve production systems Ability to create systems that remain debuggable as complexity and volume increase Comfortable working across application code, APIs, databases, cloud infrastructure, and product workflows Willingness to challenge unnecessary complexity and choose the simplest architecture capable of reliably solving the problem Comfortable introducing structure and technical standards without slowing down a high-ownership engineering team Ability to use modern AI-assisted development tools thoughtfully while maintaining strong engineering judgment and code quality Nice to Have Healthcare technology experience Experience building products for healthcare providers, clinical teams, or healthcare operations Experience integrating with electronic health record systems Familiarity with healthcare interoperability standards such as FHIR or HL7 Experience working with HIPAA-regulated data and healthcare security requirements Experience building document-processing or data-extraction pipelines Experience with OCR, document classification, or unstructured data processing Experience building workflow automation platforms Experience with referral, scheduling, intake, patient communication, or clinical administration systems Experience with high-volume API integrations and external system synchronization Experience with Ruby, Python, Go, or JavaScript in production environments Experience with Google Cloud Platform, Kubernetes, and MySQL Experience incorporating LLMs, retrieval systems, or AI-assisted automation into customer-facing products Experience improving an existing production architecture rather than only building greenfield systems Experience at a high-growth SaaS, fintech, healthcare technology, marketplace, communications, or enterprise software company