Data Engineer

Data Architecture Melbourne Australia Regular / Full time

Description

About the Role

Vericent is partnering with a leading global technology consulting organisation to recruit an experienced Data Engineer. This is an exciting opportunity to work on enterprise-scale data initiatives, building modern data pipelines and cloud-based data solutions that enable analytics, reporting, and AI-driven outcomes.

In this role, you'll collaborate with cross-functional teams, including data scientists, software engineers, product owners, and business stakeholders, to design, develop, and optimise scalable data integration and transformation solutions. You'll also play a key role in ensuring data quality, reliability, security, and performance across the entire data lifecycle.

Key Responsibilities

• Design, build and maintain batch and near-real-time data pipelines.
• Develop ingestion and transformation processes for structured and semi-structured data.
• Build reusable data components and curated datasets for reporting, analytics and machine learning.
• Develop solutions using SQL, Python, dbt, PySpark or equivalent technologies.
• Integrate data from enterprise applications, APIs, event streams and external sources.
• Optimise data pipelines for reliability, performance and cost.
• Support migration and modernisation of legacy data processes.

Data Modelling and Quality

• Develop logical and physical data models aligned to business requirements.
• Build dimensional, relational and analytical data structures.
• Implement data-quality rules, reconciliations and validation controls.
• Investigate data defects and coordinate remediation with source-system and product teams.
• Maintain data definitions, lineage, metadata and technical documentation.
• Apply appropriate handling for sensitive, customer and commercially restricted data.

Engineering and DevOps Practices

• Develop maintainable, tested and version-controlled data solutions.
• Implement automated testing across ingestion, transformation and data-quality processes.
• Support CI/CD pipelines and controlled deployment across development, test and production.
• Participate in code reviews and apply engineering standards.
• Implement monitoring, alerting and operational logging.
• Reduce manual deployment and support activities through automation.

Production Support and Operations

• Monitor production pipelines, schedules and data-processing workloads.
• Investigate and resolve failed jobs, delayed data and data-quality incidents.
• Participate in incident, problem and change-management processes.
• Perform root-cause analysis and implement permanent remediation.
• Maintain operational runbooks, support procedures and recovery steps.
• Work with Platform Operations to ensure data products meet availability and service-level requirements.

Collaboration and Delivery

• Work with product owners and stakeholders to define data requirements and acceptance criteria.
• Collaborate with data scientists to provide model-ready datasets and features.
• Work with software engineers to support application and API data requirements.
• Contribute to technical design, estimation and delivery planning.
• Communicate data risks, dependencies and design decisions clearly.

Required Skills and Experience

• Advanced SQL skills and strong understanding of data structures and query performance.
• Experience with at least two of the following:
• Snowflake
• dbt
• Databricks and PySpark
• Experience building automated data-ingestion and transformation pipelines.
• Strong understanding of data modelling, warehousing and analytical data patterns.
• Experience implementing data-quality controls and reconciliation processes.
• Experience with Git, code reviews and CI/CD practices.
• Understanding of cloud security, access control and data-protection requirements.
• Experience supporting production data pipelines and resolving operational incidents.
• Strong technical documentation and stakeholder-communication skills.
• Tertiary qualification in computer science, engineering, information systems or a related discipline, or equivalent practical experience.

Key Deliverables and Success Measures

• Reliable and scalable data pipelines and data products.
• Achievement of agreed data availability and processing service levels.
• Reduction in pipeline failures and recurring data-quality issues.
• Accurate, traceable and well-documented datasets.
• Automated testing, deployment, monitoring and recovery processes.
• Improved pipeline performance and cloud-cost efficiency.
• Compliance with security, data-governance and engineering standards.
• Positive engagement with product, platform, analytics and business stakeholders.

#J-18808-Ljbffr