Senior Data Engineer
nottingham, england, United Kingdom
Are you a seasoned Data Engineer looking to make a significant impact in the renewable energy sector? Look no further! We're seeking a Senior Data Engineer to spearhead our migration to Databricks, one of the most exciting projects in the UK energy market.
Job Title: Senior Data Engineer (Migration to Databricks)
Duration: Initially 3 months
Location: Predominantly remote with occasional onsite travel to nearest site
Contract Type: Outside of IR35
Immediate Start
Job Overview:
As a Senior Data Engineer, you'll play a pivotal role in our data transformation journey. You'll implement scalable data pipelines within Databricks, ensuring adherence to best practices and data governance principles.
Your responsibilities will include:
Building and maintaining efficient data pipelines within Databricks, evolving them as requirements evolve.
Developing and maintaining an enterprise data model within Databricks.
Integrating new data sources through batch and streaming processes while meeting SLAs.
Documenting data pipelines and systems, prioritizing security and monitoring.
Ensuring data quality and reliability through effective processes.
Requirements:
To thrive in this role, you'll need:
Strong Python skills with at least 5 years of experience, including advanced concepts.
Proficiency in SQL, database design, and data architecture.
Experience with Databricks and/or Spark.
Familiarity with data governance, cataloguing, and quality principles.
Expertise in data extraction, joining, and aggregation tasks, especially with big and real-time data.
Ability to perform data cleansing and transformation operations.
Understanding of data storage concepts and logical data structures.
Experience writing production-quality code for data pipelines.
Previous data migration experience is advantageous.
Openness to new technologies and ways of working.
Self-motivation and the ability to troubleshoot and devise solutions independently.
If you are looking for a fast moving role, using your expertise in Data Modelling, Python and Databricks, please apply for your CV to reach me directly.