Lead Data Engineer
Leeds
Job Description Lead Data Engineer:
We need some strong Data engineer profiles… they need good experience with
Pyspark, Python, SQL, ADF and preferably Databricks experience
Job description:
Building new data pipelines and optimizing data flows using the Azure cloud stack.
Building data products from scratch.
Support Business Analysts and Data Architects with discovery and best practices.
Self-directed and comfortable supporting the data needs of multiple teams, systems and products.
What’s required ?
Advanced Azure knowledge and experience working and migrating data products from on-prem to Azure.
Experience building and optimizing ‘big data' data pipelines, architectures and data sets using Py-spark.
Experience building real-time data pipelines using Event-hub, storage queues and Azure stream analysis.
Strong analytic skills related to working with unstructured datasets.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Cloud Big Data Analytics in Azure Synapse Analytics, Azure Analysis Services
Data Ingestion and Storage including Azure Data Factory, Azure Databricks, Azure Data Lake, Kafka and Spark Streaming, Azure EventHub/IoT Hub, and Azure Stream Analytics
Experience with Object-oriented/ object function scripting languages: Python preferred