☰

»

Data Engineer -- brisan oglas

Opis delovnega mesta

About Us

At Viaduct, we are developing an end-to-end machine learning platform to empower automakers to build safer, more intelligent, and personalized vehicles. Our platform increases the accessibility and actionability of connected vehicle data for automakers and their partners and end-customers. We are a diverse team motivated to solve the hardest problems in the automotive industry.

Who You Are

You are a thoughtful engineer. You understand the complexities of distributed systems and how to triage and solve issues that arise with them. Scalability is top of mind when designing any system or writing code. You believe building a better ETL system requires close collaboration with the machine learning and data science teams. You avoid reinventing the wheel unless necessary and are excited by opportunities to contribute to the open-source community.

Od kandidatov zahtevamo

Expected Skills

+ 2+ years as a data engineer
+ Proficiency in Python and SQL
+ Hands on experience with Spark or equivalent technologies
+ Working proficiency with a workflow scheduler (Airflow, Prefect, Argo, etc)
+ Experience with distributed file-systems (HDFS, S3, etc)
+ Familiar with the tools in open-source data ecosystem (Apache, CNCF, etc)
+ Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming, etc)

About the Role

Day 5

+ Learn about Viaduct history and mission
+ Get to know every team member
+ Set up your development environment
+ Understand Viaduct ETL pipelines and run your first DAGs
+ Deep dive into the nuances of vehicle data
+ Attend our weekly ML lunch

Day 30

+ Take ownership of ETL pipelines
+ Identify scalability bottlenecks in the existing ETL pipelines
+ Be familiar with the day-to-day work of machine learning engineers and data scientists
+ Learn the architecture of data engineering systems and services

Day 90

+ Be the ETL pipeline expert at Viaduct
+ Improve overall data quality and discoverability
+ Confident in the scalability of Viaduct ETL pipelines
+ Present your work at our weekly ML lunch
+ Comfortable contributing to our engineering infrastructure and systems

Klasifikacija delovnega mesta

Lokacija:
Plačilo:
okvirno 1000 EUR / teden
Delovni čas:
po dogovoru

 

Zahtevana znanja

data engineering
ekspert