Key Responsibilities:

Design, develop, and maintain scalable and reliable ETL/ELT pipelines on Azure.
Work closely with data scientists, analysts, and other stakeholders to deliver end-to-end data solutions.
Build and optimize big data pipelines using PySpark and Azure Databricks.
Orchestrate data workflows using Apache Airflow or Azure Data Factory.
Implement and support MLOps frameworks and deployment pipelines.
Monitor, troubleshoot, and ensure data quality and reliability across systems.
Drive architectural decisions for data platform components, ensuring best practices are followed.
Mentor junior data engineers and contribute to knowledge sharing within the team.

Key Skills and Technologies:

Cloud Platforms: Microsoft Azure (Azure Data Lake, Azure Databricks, Azure Synapse, Azure Blob Storage)
Programming Languages: Python (strong), SQL
Big Data Technologies: PySpark, Spark
ETL Tools: Azure Data Factory, custom ETL development
Workflow Orchestration: Apache Airflow
DevOps/MLOps: Azure ML, MLflow, CI/CD pipelines for ML models
Version Control: Git, GitHub/Azure DevOps
Other Tools (Preferred): Terraform, Docker, Kubernetes