Azure Senior Data Lead with Azure Data Factory (ADF), Azure Databricks, SQL, Oracle PL/SQL, Python

Lead the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks - bringing a strong blend of software engineering, data engineering, cloud architecture, and performance optimization.

Key Responsibilities
Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
Implement Delta Lake solutions using the Bronze Silver Gold architecture.
Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
Implement data quality, reconciliation, validation, and monitoring frameworks.
Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.
Benchmark converted applications against original Python implementations.
Core Skills
Python (expert), OOP, and advanced Python design patterns
PySpark, Spark SQL, and SQL
Azure Databricks, Azure Data Factory, ADLS Gen2
Apache Spark, Delta Lake, Data Lakehouse architecture, distributed computing
Must-have Skills
Python
Azure Databricks
Azure Data Factory (ADF)
MS SQL
Oracle PL/SQL
Good to Have
PySpark
Certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python
Experience & Expected Outcome
Senior data engineering leader with proven delivery of large-scale Databricks modernization programs.