Vishal Agarwal

Data Engineer

Bengaluru, Karnataka, India11 yrs 4 mos experience

Highly Stable

Key Highlights

Led ETL data pipelines saving over $800k annually.
Developed archival solutions saving $250k for JPMorgan Chase.
Expert in Big Data technologies and data engineering.

Stackforce AI infers this person is a Big Data Engineer with expertise in SaaS and Fintech industries.

Contact

Skills

Core Skills

Data EngineeringBig Data Analytics

Other Skills

API DevelopmentAlgorithmsAndroid DevelopmentApache KafkaApache SparkApache Spark StreamingApache SplineApplication DevelopmentAuthenticationAuthorizationBig DataBig Data MigrationBusiness AcumenCComputer Science

About

I am a Big Data Engineer at Uber and former Walmart employee and a former Associate at J P Morgan Chase and a Computer Science and Engineering graduate from Indian Institute of Technology (BHU) Varanasi. Interests: Big Data, Application Development, Algorithms, Data Analysis.

Experience

11 yrs 4 mos

Total Experience

1 yr 11 mos

Average Tenure

2 yrs

Current Experience

Uber

Data Engineer II

Jun 2024 – Present · 2 yrs · Bengaluru, Karnataka, India · Hybrid

Business AcumenMapReduceHiveData ModelingHBaseETL Tools+7

Walmart global tech india

2 roles

Senior Data Engineer

May 2023 – Jun 2024 · 1 yr 1 mo · On-site

Lead Supply Chain ETL Data Pipelines for various last mile events which caters catalogued data to business for reporting.
Lead Data Lineage team using Apache Spline for all data lineage-based use cases. Onboarded various internal teams to Spline for
cross team lineage usage. Open-source contributions to Apache Spline project.
Worked on HUDI on Spark Rapids together with NVIDIA and Onehouse teams on a cross company collaboration basis.
Lead various pipleine migration from Spark2 to Spark3, Managed BQ tables to HUDI tables. Save of over $800k yearly compute cost

Business AcumenSubject Matter ExpertsSQLData ArchitectureMapReduceHive+24

Software Engineer III

Aug 2021 – May 2023 · 1 yr 9 mos · On-site

Worked on Data Pipelines using Kafka Connect which ingested more than 1k legacy Informix tables spanning across 1k Distribution
centers for a centralised Data Lake. Reduced throughput from 1W to 1 hr.
Lead PoCs on various opensource Kafka Connect connectors for different ETL usecases and successfully implemented in production.

Business AcumenSubject Matter ExpertsMapReduceHiveSpring FrameworkData Modeling+12

Jpmorgan chase & co.

3 roles

Associate

Jan 2021 – Aug 2021 · 7 mos

1. Worked on archival solution for various raw trade messages. Adhoc retrieval of messages over an API using sync and async. Save of over
2. 250K dollars for the firm using this solution

Business AcumenSubject Matter ExpertsMapReduceHiveSpring FrameworkData Modeling+12

Software Engineer II

Promoted

Jan 2020 – Jan 2021 · 1 yr

Worked on migration of OLTP/OLAP based trade reporting system to Big Data stack.
Implemented authentication and authorization module for the application.
PoC on Neo4j for serving reference data to users. Created a shared MongoDB cluster on Kubernetes.

Business AcumenSubject Matter ExpertsMapReduceHiveSpring FrameworkData Modeling+12

Software Engineer

Jul 2018 – Jan 2020 · 1 yr 6 mos

Business AcumenSubject Matter ExpertsMapReduceHiveSpring FrameworkData Modeling+12

Maq software

Software Intern

May 2016 – Jul 2016 · 2 mos · Greater Hyderabad Area

Worked for Data Warehousing for Microsoft Partner Network Data. Analysed data over different geographies and prepared dashboards
for the same

Technical Discussions

Indian institute of technology(bhu),varanasi

Training and Placement Representative

May 2015 – Apr 2018 · 2 yrs 11 mos · Varanasi Area, India

Indian institute of technology, kharagpur

Summer Research Intern

May 2015 – Jul 2015 · 2 mos · Kharagpur I, India

Analysis of Associated Hashtags in Social Network(Twitter) using ego structures and implementation of different algorithms for finding the core hashtags.

Python (Programming Language)Technical DiscussionsInformation Technology

Tedx iitbhu

Design and Creatives

Dec 2014 – Apr 2015 · 4 mos · Varanasi Area, India

During this tenure, worked with a team of about 50 students to host a stellar line up of 9 speakers at TEDxIITBHU. The theme of the conference was "Order in Chaos". The speakers at the event were:
1. Dr. Ananda Shankar Jayant
2. Mr. Kanishka Sinha
3. Mr. Kartik Iyer
4. Mr. Harun Robert
5. Mr. Raghu Raman
6. Ms. Abhina Aher
7. Mr. Sanjeev Kapoor
8. Mr. Zubin Sharma
9. Mr. Jahan Peston Jamas

Technex, iit (bhu) varanasi

Event Coordinator

Oct 2014 – Mar 2015 · 5 mos · Greater Allahabad Area

Sampann is an event in Technex'15(27th Feb-1st March 2015) which invites participants from all across the country to come up with ideas and technologies related to energy production, energy conservation, water treatment, water conservation etc.