Big Data Engineer

Description

We are looking for a Big Data Engineer that will work on the collecting, storing, processing, and analysing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across the company.

Duties and Responsibilities

• Implementing ETL process 
• Maintaining and evolving a lambda architecture 
• Monitoring performance and advising any necessary infrastructure changes 
• Continuous learn and use of ground-breaking technologies 
• Apply advanced troubleshooting techniques 
• Work as a part of a Scrum team 

Time cocktail

• 1/3 working on our coming BigData/Data-Lake cloud based architecture 
• 1/3 writing map-reduce and Spark jobs as well as ETL jobs 
• 1/3 share knowledge with peers and leaders 

Skills

• [Advanced] Coding skill with Java and/or Scala and/or Python, R is a plus 
• [Advanced] Hadoop ecosystem knowledge 
• [Standard] Proficient understanding of distributed computing principles 
• [Standard] Experience with building stream-processing systems, using solutions such as Spark-Streaming or Apache Beam 
• [Standard] Good knowledge of Big Data querying tools, such as Pig, Hive, and Impala 
• [Standard] Knowledge of various ETL techniques and frameworks, such as Flume 
• [Standard] Clouds knowledge (GCP & AWS) [Experience with BigQuery / Dataflow is a real plus] 
• [Standard] Containers knowledge (Docker) 
• [Standard] Distributed computing environments knowledge 
• [Standard] Good understanding of Lambda Architecture, along with its advantages and drawbacks 

Spoken languages and education

• [Mandatory] English, working proficient 
• [Big plus] Italian 

Education and experience

• BS in Computer Science or Engineering or equivalent professional experience 
• 2-5 years working experience in similar roles