Data Engineer

Key responsibilities of the department

We are living in interesting times. Multiple technologies, improving exponentially, are converging. Data powers much of the transformative technologies we see today – artificial intelligence, automation and advanced, predictive analytics. These technologies have been recently bundled in a central unit, the Digital Solutions team. This allows us to prioritize better, learn from each other, inspire each other and together get the most out of these innovative technologies. The Digital Solutions team is, among others, responsible for:

• Building a modern data platform on Azure for all our structured, unstructured and streaming data, to drive transformative solutions and facilitate self-service data analytics, reporting and prediction.
• Leveraging machine learning algorithms to analyze large, complex datasets andidentify meaningful patterns that lead to predictive models and actionable recommendations.
• Designing, developing and maintaining our renewed business intelligence solution, working with the latest Microsoft Power BI technology, to be able to turn data into information, information into insight and insight into business decisions.
• Automation of predictable business processes by means of Robotic Process
• Automation, so management can focus on higher-value tasks like innovating, partnering and identifying new opportunities.

Job Description

We have a wide variety of complex data within the company, such as customer data, sensory data from our equipment and data from our business processes. The imperative is to turn this data into a strategic asset, organize smarter through data-driven decisions and develop innovative services. To achieve this, data must be brought together and made accessible. This will be your main challenge!

Your job is to unlock all internal data as well as relevant external data and make it available within the organization. You will have a key role in a greenfield project to setup up a modern data platform in Azure to collect, store and process (big) data and streaming data. You have a lot of freedom and responsibility to define the structure and make technical choices yourself.

Challenges

• This is an exceptional opportunity to do innovative work that means more to you and those we serve. The goal of the team is to design digital solutions that both protect profitable existing operations and assets, while making the transition to a new digital business or digitally enhancing part of it.
• We offer a stimulating working environment in which you contribute to innovative and challenging projects. The challenge of digital capabilities is to keep track of new developments and constantly evaluate which digital technologies might be used to achieve business goals. The team regularly take the time to experiment with new technology to see if it can be used.
• Your biggest challenge will be working with many different data sources, some of which yet to be unlocked, such as:
  • IOT
  • API/EDIs
  • Relational databases
  • Legacy systems
  • Unstructured data
  • External (big) data
• You will deal with raw data that contains human, machine and/or instrument errors. The data might not be validated and contain suspect records. It can be unformatted and can contain codes that are system-specific. The challenge is to process this into a comprehensive data set.

Specific responsibilities of the job

• Building, deploying, monitoring and managing data pipelines into Azure.
• Build and implement efficient delta loads on all these data sources.
• Continuously monitor performance and quality control plans to identify improvements.
• Discover opportunities for data acquisition from other (external) sources.
• Develop data set processes for data modeling, mining and production.
• Employ a variety of (scripting) languages and tools to marry systems together and/or review existing integrations.
• Recommend and/or implement ways to improve data reliability, efficiency and quality of existing MSSQL and Oracle data sources.

Ideal character of the candidate

• You can quickly master new environments and situations. You have guts, a positive attitude and execution power. This allows you to achieve the intended objectives in an ever-changing organization.
• Critical thinking and problem-solving skills are essential for interpreting data.
• You can collect, organize, analyze, and share significant amounts of information with attention to detail and accuracy.
• In order to handle last minute change of priorities, pressure and tight deadlines, you need to be flexible and effective at interpreting multiple complex data sources.
• You have excellent communication and collaboration skills and you know how to influence, negotiate and present information to engage others and get the job done.

Education and experience

• Computer Science or Engineering education.
• Proven working experience as a Data Engineer
• Azure Data Engineer Associate certification or similar is preferred.
• Experience with databases, ETL frameworks, data models, containerization and data formats such as JSON and Parquet
• Transport/Logistics industry experience is a plus.

Knowledge and skills

• Expert knowledge of SQL (MySQL, MSSQL, Oracle SQL, etc.)
• Good knowledge and understanding of Microsoft Azure features, in particular: storage services like Azure Data Factory and Azure Data Lake gen2.
• Knowledge of BI technologies, preferably Microsoft Power BI, and affinity with data science and analytics.
• Knowledge of scripting languages (Python, etc.)
• Knowledge of DevOps principles.
• Excellent command in spoken and written English

What are the motivators?

• Greenfield project
• Freedom to set this up your way
• Great team
• Good atmosphere
• Innovation
• Interesting projects

What are the pain points?

• Imperfect data sets
• Legacy systems
• Ad-hoc changing business priorities