| Description: |
Location: London, Potters Bar, Bristol or Isle of Man (Hybrid working options available) Job Summary The Data Engineer is a technical role responsible for designing, developing, and maintaining data pipelines within the IT department. The pipelines will be realised in a modern lake environment and the engineer will collaborate in cross-functional teams to gather requirements and develop the conceptual data models. This role plays a crucial part in driving data-driven decision-making across the organisation, ensuring data availability, quality, and accessibility for various business needs. What you'll do Data Pipeline Development - Design, model, develop and maintain data pipelines to ingest, store, process, and present data.
- Ensure data quality, accuracy, and consistency.
- Collaborate with data architects to ensure data pipelines align with the overall data architecture strategy.
Data Transformation and Integration - Perform data transformation tasks, including data cleansing, enrichment, and aggregation, to prepare data for analytics and reporting.
- Integrate data from structured and unstructured sources, ensuring compatibility and alignment with data models and business requirements.
- Automate data transformation processes to improve efficiency.
Data Quality Assurance - Implement and maintain data quality checks and validation processes to identify and resolve data anomalies and errors.
- Monitor data pipelines for data quality issues and implement data quality improvements.
- Collaborate with business stakeholders to define data quality requirements.
Data Modelling and Schema Design - Collaborate with data architects and data scientists to design and implement data models, schemas, and structures.
- Ensure that data models support business reporting and analytics needs while optimizing query performance.
- Maintain data dictionaries and metadata to document data structures and relationships.
Performance Optimization - Optimize data storage, retrieval, and query performance by implementing indexing, partitioning, and caching strategies.
- Monitor data processing performance and address bottlenecks as they arise.
- Stay updated with best practices in data processing performance tuning.
Documentation and Knowledge Sharing - Create and maintain documentation for data pipelines, data transformation processes, and data integration procedures.
- Foster a culture of knowledge sharing within the data engineering team and across the organization.
Collaboration and Stakeholder Engagement - Collaborate effectively with cross-functional teams, data stakeholders, and business units to understand data requirements and deliver data solutions that meet business needs.
- Communicate technical concepts and data solutions to non-technical stakeholders in a clear and understandable manner.
Must Have Skills/Experience Essential: - Commercial experience of DataBricks, SQL, Python, Power BI
- Experience in data engineering, including designing and developing data pipelines for retrieval / ingestion / presentation / semantics in an Azure environment.
- Effective communication and collaboration skills to work with cross-functional teams and gather data requirements.
- Skills in data modelling (both structured and unstructured data)
Desirable: - Power Automate
- Data optimisation for performance, scalability, and efficiency.
- Experience in a financial corporation
 |
