Title:  Director, Data Engineering

Requisition ID: 268951

Thanks for your interest in ScotiaTech, Scotiabank's new and innovative Technology hub in Bogota.

Join a purpose driven winning team that promotes creativity and innovation in a fast-paced environment, where we’re always committed to results, in an inclusive, diverse, and high-performing culture.

 

Purpose
We are seeking a skilled and visionary Data Engineering Director to lead a dynamic team responsible for driving our data development initiatives as we transition from on-premises infrastructure and legacy cloud environments to Azure Databricks as the target enterprise data platform. This pivotal role will oversee the strategy, design, and execution of scalable lakehouse architectures, secure data ingestion patterns, and high-performing data pipelines using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage, and related Azure services. The purpose of this position is to champion the modernization of our data ecosystem, enabling advanced analytics, AI/ML readiness, real-time insights, stronger governance, and operational efficiency on the Azure platform. Reporting directly to senior leadership, the Data Engineering Director will collaborate with cross-functional teams to deliver a future-ready data platform that powers innovation, regulatory compliance, and business growth. This position will oversee our Data Engineering operations for GBS Bogota serving IB. 

Accountabilities 

•     Strategic Leadership: Define and execute the data engineering roadmap for migrating on-premises data systems to the cloud, aligning with organizational objectives and timelines. 

•    Team Management: Lead, mentor, and grow a high-performing team of data engineers, fostering a culture of collaboration, innovation, and technical excellence. 

•    Azure Databricks Migration Oversight: Oversee the end-to-end migration of data pipelines, data stores, and analytical workloads from on-premises and legacy platforms to Azure Databricks, ensuring secure, scalable, and cost-effective delivery. 

•    Lakehouse Architecture Design: Design and implement robust, scalable, and secure lakehouse architectures on Azure Databricks, including Delta Lake, Unity Catalog, medallion architecture patterns, and integration with Azure Data Lake Storage and downstream analytics platforms. 

•    Data Pipeline Development: Direct the creation and optimization of ETL/ELT, batch, streaming, and incremental data pipelines using Azure Databricks, Apache Spark, SQL, Python/PySpark, and orchestration capabilities across Azure and Databricks workflows. 

•    Collaboration: Partner with data scientists, analysts, IT teams, and business stakeholders to understand requirements and deliver tailored data solutions. 

•    Quality, Governance, and Security: Establish and enforce best practices for data governance, quality, security, lineage, access controls, encryption, compliance, and auditability using Unity Catalog, Azure security controls, and enterprise data management standards. 

•    Technology Evaluation: Assess and recommend Azure-native and Databricks-native capabilities, including Lakeflow, Delta Lake, Unity Catalog, Azure Data Factory, Azure Monitor, CI/CD tooling, and DevOps automation to enhance data processing, storage, governance, and accessibility. 

•    Performance Monitoring: Implement monitoring, optimization, and cost-management strategies for Azure Databricks workloads, including cluster/serverless compute selection, Spark job tuning, pipeline observability, SLA tracking, and incident response. 

•    Budget & Resource Management: Manage team resources, project timelines, and budgets effectively to deliver on commitments. 

Education / Experience / Other Information 

Education:
•    Bachelor’s degree in Computer Science, Data Science, Information Technology, Business Administration, or a related field. 

Experience: 
•    Minimum of 10 years of experience in data engineering, with at least 5 years in a leadership or management role. 
•    Proven track record of successfully leading large-scale data migration projects from on-premises systems to cloud platforms. 
•    Hands-on experience with Microsoft Azure and Azure Databricks as the target data platform, including Azure Data Lake Storage, Azure Data Factory, Azure Synapse or Fabric integration, Azure Monitor, Microsoft Entra ID, and enterprise networking/security patterns. 
•    Strong knowledge of Azure Databricks, Apache Spark, Delta Lake, Unity Catalog, SQL, Python/PySpark, ETL/ELT design, batch and streaming ingestion, data warehousing, and lakehouse architecture patterns. 

Skills:
•    Exceptional leadership and communication skills, with the ability to translate technical concepts to non-technical stakeholders. Deep understanding of data modeling, ETL/ELT processes, cloud architecture principles, Azure Databricks operating models, secure coding practices, data governance, CI/CD, DevOps, performance tuning, and cost optimization for enterprise-scale data platforms.

Ideal:
•    Education: Master’s degree in a relevant technical discipline or an MBA with a focus on technology management.

•    Experience:
o    Prior experience as a Data Engineering Director or similar senior-level role in a cloud-first organization.
o    Expertise in Azure-first or hybrid cloud environments with Azure Databricks as the primary platform for lakehouse, analytics, AI/ML enablement, and enterprise data product delivery.
o    Background in industries with complex data needs (e.g., finance, healthcare, or tech).
o    Advanced familiarity with Azure Databricks capabilities such as Unity Catalog, Delta Lake, medallion architecture, Lakeflow Jobs, Spark Declarative Pipelines, Auto Loader, Structured Streaming, workflow orchestration, and integration with Power BI or Microsoft Fabric.

•    Certifications: Microsoft Azure Data Engineer Associate, Azure Solutions Architect Expert, Databricks Certified Data Engineer, Databricks Certified Professional, or equivalent cloud/data platform certifications are a strong plus.

Working Conditions
•    Work in a standard office-based environment, with standard hours.   

#LI-HYBRID

Location(s):  Colombia : Bogota : Bogota 

ScotiaTech is a business unit within ScotiaGBS, a Scotiabank Group company located in Bogota, Colombia. The ScotiaTech hub was created to support different technology systems and processes of the Bank. We offer an inclusive, positive work environment, and competitive benefits.

At ScotiaTech, we value the unique skills and experiences each individual brings and are committed to creating and maintaining an inclusive and accessible environment for everyone. Candidates must apply directly online to be considered for this role. We thank all applicants for their interest in a career at ScotiaTech; however, only those candidates who are selected for an interview will be contacted.

Note: All postings in me@Scotiabank will remain live for a minimum of 5 days.