Data Engineer

About the Role

We are looking for a Data Engineer to develop, maintain and improve the data infrastructure and backend services supporting our RE:Flex platform.

The role involves working with real-time IoT data, cloud infrastructure, databases and data pipelines to ensure reliable data collection, processing, storage and delivery.

You will work closely with backend developers, DevOps engineers, machine learning engineers and other teams across the business, operations and finance. Alongside hands-on development, you will contribute to technical improvements, code reviews, troubleshooting, documentation and knowledge sharing.

Key Responsibilities

Data Engineering and Pipelines

• Develop and maintain Python-based services for collecting and processing data from connected energy devices, including battery systems and meters.
• Build and improve real-time and batch data pipelines using MQTT, Google Cloud Pub/Sub and other messaging technologies.
• Integrate data from external APIs, energy market platforms and other relevant data sources.
• Implement data validation, error handling, retry mechanisms and safe backfilling to improve data quality and reliability.
• Develop and maintain data processing workflows, including aggregation and transformation of time-series data.

Backend Development and Cloud Infrastructure

• Develop and maintain backend APIs and services for accessing real-time and historical platform data.
• Work with databases and storage solutions such as BigQuery, TimescaleDB, PostgreSQL and MongoDB, depending on application requirements.
• Develop, deploy and maintain containerised applications using Docker and Kubernetes.
• Contribute to cloud infrastructure improvements, primarily within Google Cloud Platform.
• Support integrations between data services, backend applications and other platform components.

Data Quality, Monitoring and Reliability

• Monitor data pipelines, database performance and service reliability using tools such as Grafana and cloud monitoring services.
• Investigate data inconsistencies, missing records, processing failures and other production issues.
• Improve data processing performance, scalability and operational stability.
• Implement appropriate data quality checks, logging, monitoring and alerting.
• Support incident investigation, troubleshooting and implementation of technical fixes.

Energy Market and Business Data

• Support data processing and integration for energy market operations, including market prices, schedules, measurements and settlement information.
• Develop and maintain data workflows used for operational reporting, revenue calculations and customer invoicing.