Senior Data Engineer

Senior Data Engineer

Job Title: Senior Data Engineer
Contract Type: Permanent
Location: Western Cape
Reference: JN -012020-19543
Contact Name: Jaco Perold
Contact Email:
Job Published: March 12, 2020 13:28

Job Description

This role functions as a core member of an agile team. These professionals are responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly. They enable solutions, by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces.

Design and develop data feeds from an on-premise environment into a datalake environment in an AWS cloud environment
Design and develop programmatic transformations of the solution, by correctly partitioning, formatting and validating the data quality
Design and develop programmatic transformation, combinations and calculations to populate complex datamarts based on feed from the datalake 
Provide operational support to datamart datafeeds and datamarts
Design infrastructure required to develop and operate datalake data feeds
Design infrastructure required to develop and operate datamarts, their user interfaces and the feeds required to populate the datalake.

*Creating data feeds from on-premise to AWS Cloud (2 years)
*Support data feeds in production on break fix basis (2 years)
*Creating data marts using Talend or similar ETL development tool (4 years)
*Manipulating data using python and pyspark (2 years)
*Processing data using the Hadoop paradigm particularly using EMR, AWS’s distribution of Hadoop (2 years)
*Devop for Big Data and Business Intelligence including automated testing and deployment (2 years)

Further technical skills required:
* Talend (1 year)
* AWS: EMR, EC2, S3 (1 year)
* Python (1 year)
* PySpark or Spark (1 year)

Get similar jobs like these by email

By submitting your details you agree to our T&C's