Data Platform Engineer

About Guidant Global:

Guidant Global delivers dynamic, tailored workforce solutions that empower businesses to thrive in ever-evolving markets. Through our MSP, RPO, Direct Sourcing, Services Procurement and Consulting services, we help organisations find, engage, and manage the best permanent and contingent talent across the globe.

Part of Impellam Group, we champion A Better Way, a people-first approach that integrates deep industry expertise with advanced technology. Our technology turns workforce complexity into clarity, with real-time insights that support smarter talent decisions at scale.

With over 1.5 million STEM and professional experts in our network across 80+ countries, we don't just solve today's challenges. We're helping to shape the workforce of tomorrow.

About Guidant Global India:

At Guidant Global India, we play our part in shaping the future of work by powering recruitment and support services for Guidant teams across APAC and wider global markets. As a strategic delivery hub and global capability centre, we partner with stakeholders worldwide to raise quality, strengthen compliance, and deliver consistently at scale.

We're not standing still. We're building deeper capability in-house, taking ownership of the work that matters most, and designing an agile, future-ready delivery model built for where the business is heading next.

Join us, and you'll be part of a people-first team that's expert, ambitious, and collaborative, delivering A Better Way, every day.

About the Role:

As a Data Platform Engineer, you will own the reliability, scalability, and observability of the Headless Data Architecture (HDA), built on Databricks and Azure. You will ensure data and AI pipelines run reliably in production across multiple regions and continuously improve the platform through automation, Infrastructure-as-Code, and cost optimisation.

Working at the intersection of DataOps, MLOps, and Platform Engineering, you will also help enable AI capabilities through evaluation, prompt management, guardrails, and AI-assisted tooling.

This role sits at the foundation of HFGs ambition to become an AI-powered Workforce-as-a-Service (WaaS) organization. Your work ensures that everything built on top of the platform from analytics and data products to AI applications can run reliably, securely, and at scale.

You will collaborate closely with Data Engineers, AI/ML Engineers, Platform Engineers, Cloud Engineers, and governance stakeholders to continuously improve the platform and enable the organization to make better use of data and AI.
Key Responsibilities:

• Operate and maintain the end-to-end HDA data and AI platform on Databricks and Azure, covering data ingestion, processing, analytics, AI, and serving layers.
• Manage and evolve Unity Catalog governance, including workspace and metastore configuration, access control, credential management, and regional data-sharing structures.
• Monitor, troubleshoot, and resolve production issues across data and ML pipelines, driving structured incident response, root-cause analysis, and continuous reliability improvements.
• Build and maintain data ingestion and integration pipelines, including SnapLogic-based integrations between VMS, HR systems, and the HDA.
• Implement and operate platform-wide observability using Grafana, OpenTelemetry, and Databricks Lakehouse Monitoring, providing both operational monitoring and strategic platform insights.
• Automate and standardize platform operations using Infrastructure-as-Code and Databricks deployment tooling, including compute management, job scheduling, scaling, and cost optimisation.
• Contribute to the teams AI-accelerated engineering tooling, including the UC4 AI-powered development workbench and supporting MCP-based capabilities.
• Manage the AI-platform lifecycle, including prompt versioning, model evaluation, deployment workflows, and environment promotion.
• Implement and operate AI guardrails and safety mechanisms, including input/output validation, PII protection, toxicity detection, and monitoring of AI quality and reliability.
• Work closely with Data Engineers, AI/ML Engineers, Platform Engineers, and Cloud Engineers to ensure the data and AI platform is secure, reliable, scalable, and production-ready.
• Continuously improve platform performance, resilience, automation, and operational efficiency across regions and environments.

TECHNICAL REQUIREMENTS:

• Strong hands-on experience with Databricks in production, including cluster and compute management, job orchestration, Unity Catalog administration, Delta Live Tables / Lakeflow, and production troubleshooting, workspace administration, compute policy enforcement, Databricks Apps management, and multi-environment workspace configuration.
• Experience administering a Databricks workspace at an organisational level, including Unity Catalog metastore setup, workspace-to-metastore binding, catalog and schema hierarchy design, storage credential and external location management, attribute-based access control, data lineage propagation, and Delta Sharing configuration for cross-workspace and cross-region data sharing.
• Experience managing Databricks compute governance, including cluster policies, instance pool configuration, autoscaling rules, spot and on-demand compute mix, cost attribution tagging, and budget alerting through Azure Cost Management integration.
• Hands-on experience with Databricks Asset Bundles (DABs) for deploying notebooks, jobs, pipelines, and model-serving endpoints across development, staging, and production environments, including bundle configuration, deployment targets, and integration with CI/CD tooling.
• Experience operating the Mosaic AI platform, including the AI Gateway for LLM call routing and rate limiting, Model Serving endpoint configuration and traffic management, MLflow model registry governance, and Databricks Apps deployment.
• Strong experience with Azure data platform services, including ADLS Gen2, Entra ID (Azure Active Directory), networking, Key Vault, resource management, and cost optimisation.
• Experience building and operating data ingestion and integration pipelines using SnapLogic or a similar iPaaS, with a good understanding of REST, JDBC, and file-based integrations.
• Strong understanding of Delta Lake and data engineering patterns, including ACID transactions, table optimisation, streaming ingestion, and performance tuning.
• Experience with Infrastructure-as-Code and GitOps, using technologies such as Terraform and Databricks Asset Bundles.
• Experience with CI/CD for data platforms, including GitHub Actions, Azure DevOps, automated testing, deployment gates, and environment promotion.
• Experience with observability and platform reliability, including OpenTelemetry, Grafana Cloud, dashboards, alerting, and defining meaningful SLOs/SLAs.
• Strong automation skills using Python and/or Scala.
• Experience implementing data quality frameworks, such as Great Expectations or Databricks-native data quality capabilities, including automated validation and expectation management.
• Experience with AI/ML evaluation and validation, using frameworks such as Mosaic AI Evaluation, RAGAS, or equivalent, including LLM-as-Judge approaches and metrics such as faithfulness, relevance, and groundedness.
• Experience managing the AI lifecycle, including prompt versioning, prompt-to-model binding, environment promotion, and model evaluation across development, staging, and production.
• Understanding of AI safety and guardrails, including input/output validation, PII protection, toxicity detection, hallucination monitoring, and integration with model-serving endpoints.
• Experience designing or maintaining governed registries for AI capabilities, including agent skills, tool-use bindings, and MCP server endpoints, with versioning and lifecycle management.
• Strong understanding of data platform operations, including incident management, root-cause analysis, performance optimisation, resilience, and cost management.
• Ability to translate data, AI, and platform requirements into reliable, scalable, and operationally maintainable solutions.
• Experience working effectively across Data Engineering, AI/ML, Platform Engineering, and Cloud Engineering teams.