Data Engineer / Databricks

Data Engineer (Databricks) is required by a global software company to join its AI and Data team and play a key role in designing, developing, and maintaining data solutions.

Responsibilities:

• Designing, developing, orchestrating, maintaining, and optimizing robust data pipelines and data solutions using Databricks and the Azure ecosystem.
• Building and enhancing structured data models that transform raw data into reliable, business-ready information.
• Implementing data loading, transformation, exploration, and processing solutions.
• Troubleshooting, analysing, and optimizing ETL processes while performing root cause analysis to improve reliability and performance.
• Supporting CI/CD processes and Agile delivery practices using Azure DevOps, Repos, and pipeline automation.
• Creating and maintaining technical documentation covering workflows, data models, pipelines, and processes.

Required experience and skills:

• Extensive experience in a Data Engineering role delivering data pipelines, data warehouses, data lakes, and lakehouse solutions for BI, reporting, and analytics.
• Star Schema, fact and dimension modelling, SCD, and CDC.
• Strong hands-on experience Databricks including Workflows, Delta Live Tables, Delta Sharing, and Unity Catalog.
• Medallion Architecture.
• Apache Spark with DataFrames.
• Experience integrating Databricks with Azure data services, including Azure Data Factory (ADF) and Azure Data Lake Storage Gen2 (ADLS Gen2).
• Advanced SQL and Python.