Shubham Gupta

Software Engineer

Hyderabad, Telangana, India8 yrs 8 mos experience

Highly Stable

Key Highlights

Expert in optimizing data workflows and ETL processes.
Proven track record in reducing operational costs significantly.
Strong foundation in multiple programming languages and frameworks.

Stackforce AI infers this person is a Data Engineering and AdTech specialist with strong software development skills.

Contact

Skills

Core Skills

JavaSoftware DevelopmentData Engineering

Other Skills

RegexPHPJavaScriptHTMLCSSSQLRedisPythonAirflowPrestoHiveSqoopData StructuresAlgorithmsAndroid Development

About

"If I can't build it, I don't understand it" - Richard Feynman Amidst breakneck pace of evolution of technology, I prefer learning things gradually but surely. I'm one of those who, given 6 hours to cut a tree, spend 5 hours sharpening their axe; because great preparation brings great results. I am, and will forever remain a student - curious to learn but never satisfied. --- Know me - Web: [y2k-shubham](http://google.com/search?q=y2k-shubham) - About.Me: [y2k-shubham](https://about.me/y2k-shubham) --- Connect with me - StackOverflow: [y2k-shubham] https://stackoverflow.com/users/3679900/y2k-shubham?tab=profile - GitHub: [y2k-shubham] https://github.com/y2k-shubham - Disqus: [y2k_shubham] https://disqus.com/by/y2k_shubham . - Twitter: [y2k_shubham] https://twitter.com/y2k_shubham - Instagram: [y2k.shubham] https://www.instagram.com/y2k.shubham - Facebook: [y2k.shubham] https://www.facebook.com/y2k.shubham --- SKILLS - Languages: Python, Java, PHP, Scala, C / C++ - Frameworks: Airflow, Spark - Database: MySQL, Hive, SQL - AWS: S3, EMR, Lambda, boto3 - Misc: Docker, Regex --- INTERESTS - Art: Clean code, design-patterns, unit-test - Paradigm: Functional, Parallel programming - Contribute: Discussion forums - Learn: Tech talks, blogs --- MISC - StackOverflow: Airflow https://stackoverflow.com/tags/airflow/topusers - Regex: Love to play - OSS: noculture/notes https://github.com/noculture/notes/graphs/contributors , nektro/go.sdrie https://github.com/y2k-shubham/go.sdrie - SQL: CTEs for complex queries

Experience

Amazon

SDE2

Mar 2022 – Present · 4 yrs · Bengaluru, Karnataka, India

Identity Services

JavaRegex

Zomato

Software Development Engineer

Jul 2017 – Feb 2022 · 4 yrs 7 mos · Gurgaon, India

Ads-Tech & Data-Platform
PROJECTS
[Ads, ZERP internal dashboards, PHP, JS, HTML, CSS]
Notif-Creatives dashboard: Automated picking & cropping of image; saved 2.5 man hours daily
Other dashboards: Creative approval dashboard (dynamic SQL filtering & ordering, achieved 92% speedup using two part paginated query), Video uploader
[POI Curation & Serviceability workers & cron, PHP, Redis]
Curation-worker: Chained & ordered computations. Optimizing Redis storage to overcome worker memory overflow
Serviceability-worker: Consul-based kill switches. Bucketized batching of requests achieving 1.5K RPS with 512 MB memory & max delay of 35 s
Serviceability-cron: State persistence in Redis hash bringing down no of pushed events by 85x
[ETL Platform, Presto Derived Tables, Airflow, Presto, Hive, Python]
ETL Platform: Reactive triggering of dependent DAGs, composing heterogeneous workflows. Overridable workflow configs. ETL configs validation using Travis. Reduced TAT of pushing new ETLs from ~ 3 days to ~ 4 hours
Derived Tables: Creating & populating Hive table via Presto query, schema validation. Backfill support, arbitrary sanity-check alerts via Prometheus, Grafana. 69% speedup on Presto queries
[MySQL sync, Airflow, Sqoop, Hive, Python]
Initializing DAGs from UI. Load limiting via Pools; column transforms & partitions. Schema auto-update, delta-sync reducing sync time upto 95%. Scaled up pipeline to handle 4x no of tables
On-the-fly infra: Aurora snapshot restoration via SNS saving 85% cost. EMR cluster creation to reduce costs by 70%
[MySQL sync, Spark, Scala]
Partition size persistence in meta-db with weekly recomputation; Splitting skewed indexes to reduce sync duration by 66%
Functional programming: Connection-pool manager, Query-manager. Extensive unit-testing
[Guzzle, PHP]
Guzzle-HTTP query retry handler. Decreased cron failures by 90%
Presto client: exponential backoff, lazy load memory-optimization