Manager, Data Engineering

Manager, Data Engineering RealTime CDP.

Position Summary:

We are seeking a Manager, Data Engineering to lead hands-on technical delivery for key components of our RealTime Customer Data Platform (CDP). This role is deeply focused on core data engineering: building scalable streaming pipelines, managing stateful processing, ensuring data quality and reliability, and operating production-grade data systems.

Primary Responsibilities:

RealTime CDP & Streaming:

• Architect, build, and operate high-throughput, low-latency streaming pipelines for clickstream, transactional, and behavioral data using Kafka, Flink, Spark Structured Streaming, or Dataflow/Beam.
• Design stateful stream processing for sectorization, windowing, deduplication, enrichment, and late-arriving data handling.

Core Data Engineering:

• Design and implement robust data models for events, customer attributes, and derived features.
• Manage schema evolution, backward compatibility, and data contracts. Build scalable batch pipelines to support backfills, reprocessing, and historical precomputation aligned with real-time logic.
• Data Quality, Reliability & Operations
• RealTime CDP & Streaming
• Audience Building & Segmentation
• Customer Data Products & Quality
• Leadership & Delivery
• Cross Functional Collaboration

Work location: Hyderabad, India

Work pattern: Full time role.

Work mode: Hybrid.

Who Were Looking For:

A hands-on, full stack Data Engineering leader who can elevate real time customer data and audience/segmentation capabilities, scale them globally, and develop worldclass engineering teams that deliver privacy safe, high-performance activation.

• Implement data quality checks (completeness, validity, timeliness), pipeline-level SLAs/SLOs, and end-to-end observability.
• Own production readiness including monitoring, alerting, capacity planning, and on-call support with SRE partners.
• Architect and scale real-time pipelines for clickstream, transactional, and behavioral data using Kafka, Flink, Spark Structured Streaming, or Dataflow/Beam.
• Design and evolve customer event models, session-inaction, and cross channel stitching to maintain a unified, channel stitching to maintain a unified, privacy aware customer view.
• Implement low latency activation APIs used by apps, web, CRM, loyalty, kiosks, and marketing orchestration platforms.
• Implement Eventing frameworks like managed Kafka or Confluent
• Establish standards for observability, SLAs/SLOs, schema evolution, lineage, and cost efficiency across streaming and batch paths.
• Engineer data pipelines and services that power audience segmentation, attribute computation, and activation feeds.
• Ensure efficient data delivery to downstream systems (e.g., CDPs, marketing platforms, APIs) with strict latency and freshness guarantees.
• Build dynamic audience services for behavioral and lifecycle cohorts, rules driven propensity groupings, and event triggered real-time segments.
• Define data contracts and versioning for attributes, traits, and segment definitions to ensure reuse, durability, and safe change management.
• Set and abide by audience governance standards (freshness SLAs, recency/frequency windows, cardinality limits, consent gates) and ensure theyre consistently enforced.
• Partner with Mar Tech and Product to templatize audience playbooks (e.g., reactivation, onboarding, churn risk, high value, cart abandon).
• Implement comprehensive data quality, validation, and lineage across all audience and profile pipelines.
• Create reference patterns and templates so global markets and channels can integrate quickly and safely.
• Drive CDP capabilities and roadmaps, manage dependencies, and deliver against business outcomes with clear KPIs and executive reporting.
• Build and maintain engineering excellence: testing, automation, code quality, observability, and operational readiness.
• Lead technical execution, review designs and code, mentor engineers on data engineering best practices, and drive disciplined delivery through sprint planning, estimation, and operational follow-through.
• Collaborate with Product, Mar Tech, Loyalty, Architecture, Data Governance, Security, Legal, and Compliance to align roadmaps and ensure privacy-by-design and security-by-default.
• Translate marketing and product activation needs into reusable audience capabilities and APIs.
• SQL Very Strong proficiency in native SQL, has used Big Query or Athena Advanced performance tuning on large datasets.
• Languages: Python (primary), JavaScript, Node.js plus Java.
• Streaming & Processing: Kafka, Flink, Spark/PySpark, Dataflow/Beam.
• Audience/Segmentation: Handson experience building audience engines, cohort generation logic, and audience APIs for activation.
• Data Platforms: GCP, Databricks; Big Data ecosystems (Hadoop, lake house patterns); NoSQL; columnar formats (Parquet).
• Cloud: GCP preferred (Pub/Sub, Big Query, Dataflow, Cloud Run); AWS/Azure acceptable.
• Pipelines & Orchestration: ETL/ELT, Airflow/Luigi, CI/CD for data.
• Data Management: Metadata management, schema evolution, data contracts, lineage.
• Governance & Reliability: Observability, SLAs/SLOs, validation, consent/privacy controls.
• 7-10 years in largescale Data Engineering / Distributed Systems.
• 5+ years with GCP or AWS (GCP preferred).
• 3+ years working on real time customer data and/or segmentation platforms.
• Experience with CDPs (mParticle, Adobe RTCDP, Braze, Tealium).
• Designing real-time audience builders, rule engines, and activation frameworks.
• Multiregional deployments, data residency, and consent management at global scale.
• Strong stakeholder communication; ability to simplify technical concepts for marketers and product leaders.
• Systems thinker with strong architectural judgment and influence.