Divya Tiwari

Data Engineer

Munich, Bavaria, Germany6 yrs 3 mos experience
Highly Stable

Key Highlights

  • Achieved 50%+ faster processing in ETL pipelines.
  • Reduced infrastructure costs by 60% through optimization.
  • Engineered pipelines with 99.9% uptime and 80% manual effort reduction.
Stackforce AI infers this person is a Data Engineering expert specializing in cloud migrations and data optimization.

Contact

Skills

Core Skills

GcpEtl/eltBigquery

Other Skills

Uc4SQLETLDockerAirflowTerraformCI/CDCloud ComposerData FusionDataflowTeradataSnowflakeOracleSQL ServerPython

About

I am a Senior Data Engineer with 6+ years of experience designing, building, and optimizing scalable data platforms and pipelines on Google Cloud Platform (GCP). I specialize in migrating complex legacy systems (Teradata, Snowflake, SQL Server, Oracle, AS400) to modern cloud data warehouses like BigQuery, and in building reliable batch and streaming data workflows for analytics and ML. My core skills span GCP (BigQuery, Dataflow, Data Fusion, Cloud Composer, Cloud SQL, Pub/Sub, Cloud Functions), Python, advanced SQL, and data modeling for analytics and BI. I have led end-to-end implementations of ETL/ELT pipelines, containerized data workflows using Docker and Airflow, and infrastructure-as-code using Terraform and CI/CD (GitHub Actions). Some examples of impact: Led Teradata-to-GCP migrations, redesigning ETL pipelines and orchestration to achieve 50%+ faster processing, 60% infrastructure cost savings, and zero data loss. Optimized BigQuery and Teradata queries, improving query performance by up to 70% for 500+ daily users and reducing storage costs by 35%. Orchestrated and automated 500+ ETL pipelines across SQL Server, AS400, and Oracle into BigQuery with 99.9% uptime and 80% reduction in manual effort. Co-developed a Manufacturing Data Engine with Google, ingesting 1 TB+ PLC data daily with sub-second latency for real-time insights. Built Kafka-to-BigQuery streaming pipelines with Dataflow for high-throughput, real-time analytics. I enjoy solving complex data problems, improving performance and reliability, and collaborating with cross-functional teams to turn raw data into usable, trustworthy, analytics-ready datasets. Currently based in Munich and open to challenging data engineering opportunities across Europe. My core skills span GCP (BigQuery, Dataflow, Data Fusion, Cloud Composer, Cloud SQL, Pub/Sub, Cloud Functions), Python, and advanced SQL, with strong experience in data modeling, ETL/ELT, and dimensional data warehousing. I have worked extensively with Teradata, Snowflake, SQL Server, Oracle, and AS400, and build production-grade pipelines using Airflow, Docker, Terraform, Git, and CI/CD (GitHub Actions

Experience

Exl

Data Engineer

Jul 2023Aug 2025 · 2 yrs 1 mo · Pune, Maharashtra, India · Remote

  • Led Teradata-to-GCP migration: Redesigned ETL pipelines and implemented UC4 orchestration with DALM/Neo for metadata management, ensuring seamless data integration and governance.
  • SQL & Performance Optimization: Tuned complex SQL queries, achieving 50%+ faster processing, 60% infrastructure cost savings, 75% improved data accessibility, and zero data loss.
  • Containerized Data Pipelines: Built Docker- and Airflow-based pipelines, reducing manual operations by 70% and increasing deployment frequency 3×.
  • Scalable Infrastructure Management: Oversaw complex onboarding processes, delivering robust infrastructure that empowers customers to build reliable data pipelines.
  • Accelerated Application Deployment: Reduced deployment time by 50%, boosting user satisfaction scores by 25%.
  • BigQuery Optimization: Engineered a pipeline to decommission 2,000+ obsolete objects, improving query performance by 25%, optimizing slot utilization, and cutting storage costs by 35%.
  • Git & Collaborative Development: Hands-on experience in branching, merging, and pull requests within team-based workflows.
  • DevOps & CI/CD Implementation: Designed and maintained CI/CD pipelines with GitHub Actions, enhancing deployment efficiency and automation reliability.
  • Terraform & Infrastructure as Code (IaC): Designed scalable Terraform solutions to automate cloud provisioning, enforce standardized infrastructure, improve deployment reliability, and reduce configuration drift.
Uc4SQLETLGCPBigQueryDocker+4

Quantiphi

2 roles

Senior Data Engineer

Promoted

Dec 2020May 2023 · 2 yrs 5 mos

  • Performed data merging and feature extraction, creating backtest and metrics evaluation tables in BigQuery. Integrated these seamlessly into pre- and post-processing pipelines for model training and inference, automated through Cloud Composer.
  • Optimized high-latency queries from Teradata, Snowflake, and other databases, transforming them into low-latency queries in BigQuery by applying various query optimization techniques to enhance user experience.
  • Collaborated with Cloud SQL to deliver data to front-end applications, managed schema tasks, and transformed BigQuery data to align with Recommendation AI’s standard schema for model inputs and recommendation generation.
  • Developed ETL pipelines using Data Fusion and Dataflow to transfer data from Oracle, SQL Server, Teradata, and other sources to BigQuery, enabling faster processing and consistent performance. Efficiently managed data to reduce reliance on external ETL tools and deliver faster, more accurate insights.
  • Partnered with Google's product team to develop the Manufacturing Data Engine (MDE), enabling seamless batch and streaming data transfer from PLCs directly into GCP BigQuery.
BigQueryCloud ComposerData FusionDataflowTeradataSnowflake+5

Data Engineer

Mar 2019Dec 2020 · 1 yr 9 mos

Education

Centre for Development of Advanced Computing (C-DAC)

PG Diploma — Advance Computing

Jan 2018Jan 2019

Shivajirao S Jondhale College of Engineering Vishnu Nagar Dombivli (West) Dist Thane

Bachelor of Engineering - BE — Information Technology

Jan 2014Jan 2018

Hindi Vidya Prachar Samitis Ramniranjan Jhunjhunwala College Opp Railway Station Ghatkopar West Mumb

HSC — Science

Jan 2013Jan 2014

Stackforce found 100+ more professionals with Gcp & Etl/elt

Explore similar profiles based on matching skills and experience