Shubha Gupta

Data Engineer

Dubai, United Arab Emirates9 yrs 6 mos experience

Key Highlights

  • Over 8 years of experience in data engineering.
  • Significantly reduced data engineering costs.
  • Expert in building end-to-end data pipelines.
Stackforce AI infers this person is a Data Engineering expert in SaaS with a focus on cloud technologies.

Contact

Skills

Core Skills

Data EngineeringData Architecture

Other Skills

Amazon Web Services (AWS)Apache KafkaApache SparkApache Spark StreamingAzure DatabricksBig DataCC++Ci cdCloud ComputingCluster AnalysisCost OptimizationData IntegrationData LakeData Lineage

About

As a Senior Data Engineer, I integrate, optimize, and enhance data solutions for the growth team, using Azure Databricks and other cloud technologies. I have reduced data engineering costs significantly in a short period of time by optimizing existing pipelines and identifying redundant processes. I have a Master of Science in Mathematics and Computing from Indian Institute of Technology, Guwahati, where I learnt my computing skills. I have more than 8 years of experience in designing and building end-to-end data pipelines, data lakes, data migrations, data warehousing, and data marts, working with various clients and industries. I enjoy solving complex and challenging data problems and delivering value to the business.

Experience

9 yrs 6 mos
Total Experience
1 yr 7 mos
Average Tenure
1 yr 11 mos
Current Experience

Binance

2 roles

Senior Data Engineer ll

Jul 2025Present · 11 mos · Dubai, United Arab Emirates

Senior Data Engineer

Jul 2024Jul 2025 · 1 yr · Dubai, United Arab Emirates

Kingmakers

2 roles

Senior Data Engineer ll

Apr 2024Jul 2024 · 3 mos · Dubai, United Arab Emirates

Senior Data Engineer l

Apr 2022Apr 2024 · 2 yrs · Dubai, United Arab Emirates

  • Integrated Google Analytics complex and nested data into data lake, simplifying access for the growth team and improving decision-making.
  • Reduced data engineering costs by 25% in 3 months by refining pipelines and cutting unnecessary spending.
  • Conducted optimization and learning sessions to help data engineers and analysts improve pipelines
  • performance and data querying.
  • Decreased Analytics costs by 80% by migrating bigquery data to the data lake, resulting in significant savings.
  • Enhanced CRM capabilities by bringing Salesforce data into the data lake, improving accessibility and
  • decision-making.
  • Developed an offset manager tool to optimize data ingestion, reducing processing time and improving
  • eff iciency.
  • Identified and mitigated duplicate user accounts using website events data, leading to better campaign
  • decisions and fraud prevention.
Azure DatabricksScalaCi cdExtract, Transform, Load (ETL)Apache SparkGoogle Analytics+9

Zand

Data Engineer

Mar 2020Apr 2022 · 2 yrs 1 mo · United Arab Emirates

  • Developed more than 500+ data lake ETL pipelines using an in-house ETL
  • framework in the minimum possible time, resulting in streamlined data
  • processing and analysis.
  • Designed and developed a comprehensive solution for all incremental loads,
  • resulting in faster and more robust data pipelines for efficient data
  • processing and analysis.
  • Contributed to the development of an in-house Fund Transfer Pricing (FTP)
  • model for our bank, working closely with the BI team to ensure the accurate
  • implementation of complex logic, resulting in an effective and efficient FTP
  • model.
  • Implemented a challenging streaming logic that involved transforming real-
  • time transactions and enriching them with customer data, resulting in the
  • successful delivery of enriched real-time transactional data to the customer.

Artefact

Data Engineer

Feb 2019Mar 2020 · 1 yr 1 mo · Dubai

  • Designed and implemented data architecture primarily using Microsoft Azure stack
  • Designed and developed operational ETL pipelines using Streamsets, Spark, Databricks and Apache
  • airflow.
  • Implemented a challenging streaming logic independently in order to send transformed real-time
  • transactions to the consumer.
  • Generated BRF reports for Central Bank of UAE to comply with UAE regulations.
  • Created end-to-end pipelines for management dashboards using azure cloud and power BI.

Exl

Data Engineer

Jul 2018Dec 2018 · 5 mos · Gurgaon, Haryana, India

  • Facilitated client to identify fraud saving opportunity for top US issuing banks.
  • Designed and developed end-to-end solution to automate the standard fraud reporting processes with oozie.
  • Significant reduction in error achieved in monthly report generation by end-to-end automation of processes.

Cognizant

Data Engineer

Jul 2016Jul 2018 · 2 yrs · Chennai Area, India

  • Worked with a major retail client on migration from Hadoop to AWS to offload on-premises cluster
  • Worked with a major health insurance provider of US to recommend nearest hospitals based on the customer address
  • Built 360' views from surveys, demographics, cdc data.
  • Worked on a POC to design and develop a recommendation tool for best practices in hive.

Education

Indian Institute of Technology, Guwahati

Master of Science (MSc) — Mathematics and Computing

Jan 2014Jan 2016

Stackforce found 100+ more professionals with Data Engineering & Data Architecture

Explore similar profiles based on matching skills and experience