Senior Azure Data Engineer

Job Summary:

We are looking for an experienced Sr Azure Data Engineer with strong hands-on expertise in PySpark, Azure Synapse Analytics, SQL, Big Data technologies, and modern cloud data engineering practices. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines, distributed data processing frameworks, and enterprise-grade analytics solutions across Microsoft Azure and Big Data platforms.

Responsibilities :

· Design, develop, and maintain scalable data pipelines using PySpark, Spark SQL, Python, SQL, and Azure Synapse Analytics.

· Build and optimize ETL/ELT workflows for large-volume structured, semi-structured, and unstructured data processing.

· Develop and support Azure Synapse pipelines, notebooks, SQL scripts, stored procedures, views, and data transformation logic.

· Work with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Databricks, Azure SQL Database, and related Azure data services.

· Develop and support Big Data solutions using technologies such as Apache Spark, Hadoop, HDFS, Hive, Sqoop, Kafka, HBase, and related distributed data processing components.

· Translate business requirements and functional specifications into detailed technical designs and reusable data engineering components.

· Implement data ingestion, transformation, validation, reconciliation, and publishing processes across enterprise data platforms.

· Perform performance tuning of PySpark and SQL workloads by optimizing joins, partitioning, caching, file sizing, indexing, and query execution plans.

· Optimize Big Data jobs and distributed workloads by tuning Spark configurations, partitions, memory usage, execution plans, storage formats, and cluster resource utilization.

· Develop reusable frameworks for metadata-driven ingestion, incremental loads, error handling, audit logging, and data quality checks.

· Support unit testing, system testing, UAT, deployment, production monitoring, incident resolution, and post-production support.

· Collaborate with architects, business analysts, QA teams, DevOps teams, platform teams, and client stakeholders to deliver robust data solutions.

· Ensure compliance with data governance, security, lineage, audit, access control, and operational standards.

· Mentor junior data engineers, review code, provide technical guidance, and promote engineering best practices.

Required Skills:

· 8 to 15 years of overall IT experience with a strong focus on data engineering, Big Data platforms, distributed data processing, cloud data solutions, and enterprise data processing.

· Minimum 5+ years of hands-on experience in PySpark, Spark SQL, Python, SQL, and data pipeline development.

· Minimum 4+ years of hands-on experience with Big Data ecosystem tools such as Spark, Hadoop, HDFS, Hive, Kafka, Sqoop, or HBase.

· Minimum 3+ years of practical experience working with Azure Synapse Analytics and Azure cloud data services.

· Proven experience in designing, developing, testing, deploying, and supporting production-grade data engineering solutions.

· Ability to independently own technical design, development, defect resolution, deployment, and production support activities.