Data Ops

Purpose

A DataOps is responsible for building and maintaining efficient, reliable, and automated data pipelines and systems to ensure seamless data flow across the IBDF. This role focuses on ensuring data infrastructure availability, performance, and scalability through the use of automation and monitoring tools. A DataOps Engineer collaborates with cross-functional teams to optimize data operations and troubleshoot incidents. 

Accountabilities 

o    Experience in building and maintaining data pipelines using tools like Apache Airflow, Spark or BigQuery.
o    Strong knowledge of database management systems (SQL, and NoSQL, e.g. PostgreSQL, MySQL, MongoDB).
o    Familiarity with automation and monitoring tools to ensure data infrastructure performance (e.g. Cloud Logging, Apache Airflow).
o    Knowledge of agile methodologies for project management and delivery.
o    Excellent collaboration skills with operational and infrastructure teams.
o    Attention to detail to troubleshoot and fix system errors.
o    Time management skills to respond to incidents efficiently.
o    Proactive mindset to optimize and improve existing processes.
o    3-5 years of experience in DataOps or data engineering roles focused on pipelines.
o    Proven expertise in managing cloud-based data infrastructure (AWS, Azure, GCP).
o    Previous experience working with DataOps or DataOps methodologies.

Reporting Relationships

Primary Manager: Director, International Analytics Engineering
Direct Reports:     NA
Shared Reports (solid/dotted if applicable):    NA

Dimensions NA

Education / Experience / Other Information

Education:

Bachelor's degree in computer science, Data Engineering, or a related field.
Certifications in: AWS Certified DevOps Engineer or Google cloud professional Data Engineer.

Key task

•    Design and implement highly automated data pipelines.
•    Monitor and troubleshoot data flows in real time.
•    Implement data-driven DevOps practices for pipelines.
•    Manage permissions and access to data in secure environments. 
•    Optimize infrastructure to handle large volumes of data. 
•    Document processes for ease of maintenance and scalability.

Competencies

Working Conditions

Work in a standard office-based environment; non-standard hours are a common occurrence