Senior Data Engineer
We are currently looking for a Senior Data Engineer for a permanent with a leading financial services client based in the CBD. The role is responsible for supporting the Data Science team with delivering data solutions into the data science process and productionising data science solutions into the business.
You will join a small close-knit team, who are focused on transformational change with the exciting remit of experimenting with emerging technologies to ‘step-change’ within the business. This is an environment that requires agility and pace, and often lead in the direction of un-ventured territory.
Key Responsibilities include:
- Developing and productionising machine learning models.
- Interrogating source systems to obtain data.
- Designing and developing ETL/ELT processes - using both Hadoop Ecosystem on-prem and on Azure
- Building and maintaining CI/CD processes for ML applications deployment.
- Monitoring the end-to-end processes from data extract, model train, test and deployment.
- Maintaining development standards and code quality
We’re looking for the following experience:
- Must have significant experience as a Data Engineer
- Experience working with Big Data technologies and distributed computing tools (Azure, Databricks, Spark, Hadoop, Hive, MapReduce, Azure Data Factory, Databricks, CosmosDB, Azure Storage Account etc)
- Experience with multiple programming languages such as python, R, pyspark, scala, SQL, etc
- Processing unstructured data
- Excellent analytical and data analysis skills
- CI/CD and/or DevOps experience
- Data science/machine learning knowledge a plus
- Experience with Azure / cloud would be desirable
- Experience with test-driven development, test automation & continuous delivery.
- Experience in building automation technologies.