4.0 Data Engineer

The scope of this project encompasses the following deliverable base tasks: Deliverable 1: Infrastructure Assessment and Recommendations This assessment will include a detailed analysis of ETL tools, processes, and patterns, encompassing Python scripts, Prefect orchestration, data pipelines, Snowflake database, and dbt workflows. The vendor will also evaluate the current data schema organization, ETL processing standards, and methodology to identify inefficiencies and opportunities for optimization. Key areas of focus will include performance, scalability, security, resilience, and reliability of the existing architecture.

Deliverables for this phase will include a comprehensive assessment report, a target state architecture document, and a future-state process design. These outputs will serve as the foundation for the implementation phase, ensuring that the Snowflake platform is built on a solid, well-defined framework. The vendor will work closely with client teams to review and approve these deliverables, ensuring alignment with organizational goals and priorities.

This phase will also establish the groundwork for strong collaboration between the vendor and client, setting expectations for the subsequent implementation phase. Solution design will adhere to client policies and software assessments. Deliverable 2: Data Platform Implementation Building on the recommendations from the assessment phase, the vendor will design, build, and implement a modern data platform using Snowflake Data Cloud, Microsoft Azure, dbt, Prefect, and related technologies.

This platform will serve as a centralized, secure, and scalable data hub, enabling client to streamline data access, enhance analytics capabilities, and support data-driven decision-making across the enterprise. The implementation will follow a Hub & Spoke architecture, with the Hub serving as the centralized repository for standardized, governed, and curated data, and the Spokes supporting specific data applications and business functions. Separate environments for Development, User Acceptance Testing (UAT), and Production will be established to support testing, validation, and deployment processes.

The vendor will also implement robust data governance practices, including lineage tracking, monitoring tools, and access controls, to ensure consistent data quality and security. Logical and physical data models, such as star and snowflake schemas, will be defined and implemented to support advanced analytics and reporting. Semantic layers will be created to enable business users to access and analyze data efficiently.

Deliverables for this phase will include a fully configured data platform, data pipelines, security, and documentation. The data platform is a Snowflake data Hub environment with integrations, security, documentation, process flows, component diagrams, dbt model documentation, and CI/CD scripts that implement the design and architecture from Deliverable 1. The vendor will also deliver starter data models and fully functional Spokes, including ELT workflows, dbt models, and Prefect orchestration that address operational business needs throughout the Investment Office.

The vendor will provide detailed documentation and training materials to support ongoing operations and future enhancements. This phase will culminate in the delivery of a secure, scalable, and cost-effective data platform aligned with client's organizational goals and priorities. Deliverable 3: Knowledge Transfer and Operational Readiness The vendor will conduct comprehensive knowledge transfer sessions to ensure client staff are fully equipped to manage, optimize, and extend the Snowflake Data Cloud platform independently.

These sessions will include hands-on training, workshops, and detailed walk-throughs of the platform's architecture, ELT pipelines, data governance practices, and security configurations. The vendor will provide documentation covering all aspects of the implementation, including technical architecture, process flows, data pipelines, CI/CD scripts, operating model, and best practices for ongoing maintenance and scalability. Additionally, the vendor will deliver a support plan and troubleshooting guidelines to empower client teams to address future challenges effectively.

By the end of this phase, client staff will have the skills, tools, and confidence to operate, optimize, and expand the Snowflake Data Platform.