Shubham Upadhyay

Data Engineer

Seattle, Washington, United States9 yrs 7 mos experience

Key Highlights

  • Over 9 years of experience in Data Engineering.
  • Expertise in cloud platforms and big data technologies.
  • Proven track record in optimizing data systems.
Stackforce AI infers this person is a Data Engineering expert with a focus on media and e-commerce industries.

Contact

Skills

Core Skills

Data EngineeringCloud ComputingBusiness IntelligenceAnalyticsData AnalysisSoftware Development

Other Skills

Apache SparkAirflowAmazon Web Services (AWS)Extract, Transform, Load (ETL)ScalaPythonSQLSnowflakeDatabricksETLAWSDBTSystem PerformanceData PipelinesGitHub

About

I am currently working as a Senior Data Engineer in The Walt Disney Company. I have also contributed to Data Engineering and Analytics teams at Warner Bros. Discovery (Max, HBO Max), Amazon (Prime Video, Customer Experience), Uber, Fiserv, and Atos. About Me: As a Senior Data Engineer with over 9 years of experience, specialize in designing and implementing scalable data architectures that drive business intelligence and data-driven decision-making. My expertise spans across cloud platforms like AWS, advanced data warehousing solutions, and big data technologies, including Snowflake, Redshift, Spark, and Hive. I excel in building robust ETL pipelines, optimizing complex SQL queries, and ensuring high-performance data processing in cloud environments. I am also proficient in modern data architectures, such as data lakes and data lakehouses, and have successfully led the migration of legacy data systems to the cloud, ensuring seamless integration and enhanced system resilience. Throughout my career, I’ve been committed to continuous learning and adaptability, thriving in fast-paced environments where technology evolves rapidly. My ability to collaborate effectively with cross-functional teams and communicate complex technical concepts clearly has been key to my success in delivering impactful data solutions. I’m passionate about leveraging data to solve complex problems and am always on the lookout for innovative ways to optimize and scale data systems. Skills: SQL | Python | Spark | Scala | Airflow | ETL | DBT | Looker | Tableau | AWS | GitHub | Databricks | Snowflake | Redshift | BigQuery | Data Warehousing | Data Modeling | Statistical Analysis | Java

Experience

9 yrs 7 mos
Total Experience
1 yr 6 mos
Average Tenure
5 mos
Current Experience

Meta

Senior Data Engineer

Dec 2025Present · 5 mos · Seattle, Washington, United States · Hybrid

The walt disney company

Senior Data Engineer

Dec 2023Dec 2025 · 2 yrs · Seattle, Washington, United States · On-site

  • Disney and Hulu Ads Platform: Processing petabytes of data data
  • ▪Design and implement scalable data pipelines using Apache Spark in Scala, to efficiently process large data volumes & ensure rapid data delivery.
  • ▪Write and optimize SQL queries for data transformation and aggregation, ensuring high performance and scalability in complex data environments.
  • ▪Develop ETL pipelines orchestrated with Airflow, reducing data processing time by 50% and enabling faster access to data-driven analytical insights.
  • ▪Built data infrastructure and data lakes on AWS, utilizing services such as S3, Glue, and Athena to support cloud-based data solutions.
  • ▪Led the migration of legacy on-prem data systems to AWS, implementing observability mechanisms to enhance data monitoring and system resilience.
  • Exposure: SQL, Spark, Scala, Python, Snowflake, ETL, Airflow, Git, Jenkins
Apache SparkAirflowAmazon Web Services (AWS)Extract, Transform, Load (ETL)ScalaPython+5

Warner bros. discovery

Senior Analytics Engineer

May 2023Dec 2023 · 7 mos · Seattle, Washington, United States · Hybrid

  • Max Data & Analytics: Developing Data Engineering, Analytics and Business Intelligence solutions of Max Streaming App
  • Design and Implement Semantic Layer Data Pipelines for Subscription (SVOD) for Max, Legacy HBO-MAX and Discovery+.
  • Write optimized SQL queries in Databricks to build KPIs and metrics for assessing business performance and generating insights.
  • Develop LookML models, Explores in Looker, for intuitive data visualization.
  • Implement DBT models with incremental logic, streamlined data transformations and workflow automation and DBX workflows to enhance efficiency and scalability.
  • Build ELT pipelines, using Airflow orchestration to create analytics layer.
  • Technologies: SQL, Python, Looker, Databricks, DBT, ETL/ELT, Airflow, Git
DatabricksDBTAnalyticsSystem PerformanceSQLData Pipelines+10

Amazon

Senior Business Intelligence Engineer

Jul 2021Mar 2023 · 1 yr 8 mos · Hybrid

  • Perfect Order Experience: Leading the Customer Experience (CX) Business Intelligence and Analytics Team.
  • Build datasets from large multi-dimensional data sources into leaner and structured data tables using Python.
  • Design databases and implement Data Warehousing for data storage in Redshift and analysis.
  • Write SQL queries to retrieve data and formulate business critical metrics from various data clusters.
  • Build end-to-end ETL pipelines and automating the process using AWS resources such as S3, Lake Formation, Step Function and Lambda.
  • Create Tableau and Quicksight dashboards to evaluate, derive insights and drive strategic recommendations.
  • Code scalable and efficient automations for large scale data processing and analysis.
  • Amazon Transportation Services: Optimize Supply Chain
  • Complete end to end automation for various processes inside ROC to foster quick resolution and reduce man hours on manual tasks.
  • Writing complex SQL queries, on-boarding it to the ETL pipeline and streamlining the process
  • Creating reports and dashboards for IXDs, Totes, RELO etc. in Tableau, Excel to have complete view of the trend over historical events and taking decisions on it.
  • Working on process optimization and improving the forecast accuracy in Totes network for better scheduling.
  • Building time series forecasting model to predict week ahead volume.Amazon Transportation Services: Optimize Supply Chain • Complete end to end automation for various processes inside ROC to foster quick resolution and reduce man hours on manual tasks. • Writing complex SQL queries, on-boarding it to the ETL pipeline and streamlining the process • Creating reports and dashboards for IXDs, Totes, RELO etc. in Tableau, Excel to have complete view of the trend over historical events and taking decisions on it. • Working on process optimization and improving the forecast accuracy in Totes network for better scheduling. • Building time series forecasting model to predict week ahead volume.
Data ScienceStatisticsA/B TestingAnalyticsSystem PerformanceStakeholder Management+29

Uber

Data Analytics Specialist

Feb 2021Jul 2021 · 5 mos

  • Uber’s Analytics and Reporting:
  • Drive WBRs and MBRs and gather insights on contributing factors of anomalies using Statistical Analysis.
  • Create metrics and KPIs using SQL and automating the mechanism using Python
  • Create dashboards and reporting using BI tools like Tableau & Data Studio.
  • Implementing Python - Dash code to build a web dashboard to save costs of Tableau licenses for agents and non-executives of Uber.
Data ScienceStatisticsA/B TestingAnalyticsSystem PerformanceStakeholder Management+28

Fiserv

Application Developer

Jul 2019Feb 2021 · 1 yr 7 mos

  • Analyze data, gather insights using Power BI dashboards from different data sources through SQL and Python.
Microsoft Power BIStatisticsAnalyticsStakeholder ManagementObject-Oriented Programming (OOP)SQL+27

Atos

Software Engineer

Jul 2016Jun 2019 · 2 yrs 11 mos

  • Developing SQL queries for extracting data from Enterprise Discount Database
  • (EDD). Code development in Python and Java to derive package pricing
Microsoft Power BIStatisticsAnalyticsStakeholder ManagementObject-Oriented Programming (OOP)SQL+28

Education

Indian School of Business

Advanced Management Program in Business Analytics — Business Analytics

Jan 2020Jul 2022

D. Y. Patil College of Engineering ( DYPCOE ) , Akurdi, Pune

Bachelor of Engineering - BE — Information Technology

Jan 2012Jan 2016

Stackforce found 100+ more professionals with Data Engineering & Cloud Computing

Explore similar profiles based on matching skills and experience