Data Engineer - Databricks

Data Engineer (Databricks)

Inside IR35 6-Month ContractSector: Public Sector / Central GovernmentLocation: London or Telford (Hybrid - Remote with on-site as required)Security Clearance:Active SC Clearance due to time scales

About the Role

We are seeking an experienced Data Engineer with deep expertise in Databricks to support the development of scalable data platforms within a Central Government environment.

This role is delivery focused and requires someone who has worked in secure, regulated public sector settings. You will be responsible for designing, building, and optimising data pipelines that enable analytics, reporting, and downstream data services across the organisation.

Key Responsibilities

• Design, build, and maintain data pipelines using Databricks
• Develop ETL/ELT processes with PySpark and Spark SQL
• Transform and model structured and semi-structured datasets
• Improve performance, reliability, and cost efficiency of data workloads
• Ensure compliance with data governance, security, and quality standards
• Work collaboratively with architects, analysts, and delivery teams
• Produce clear documentation and contribute to engineering best practice routes

• Active SC Clearance - mandatory and non-negotiable
• Strong hands-on experience with Databricks in production environments
• Expert knowledge of Apache Spark, including PySpark and Spark SQL
• Proficiency with Python for data engineering
• Experience delivering solutions on cloud data platforms (AWS preferred, Azure acceptable)
• Understanding of data lake and lakehouse architectures
• Ability to work autonomously and collaboratively within delivery squads
• Experience working within Central Government departments (highly desirable)

• Azure Data Lake, Synapse, Delta Lake
• Streaming technologies (Kafka, Event Hubs)