Oct 2021 – Dec 2024 · 3 yrs 2 mos · Bengaluru, Karnataka, India

• Working with Search Realtime Data Platform Team.This project mainly focus on parsing and stitching user journey on Walmart Search in Realtime to generate signals, populate user metrics and hive tables.
• Impact/Improvement of Realtime Pipeline for Walmart Search
• · Data availability improved by 8 hrs compared to Legacy Batch Pipeline.
• · With Improved Data Availability, Query Signals are generated and published faster.
• · Signals, Trending Items can be identified in Realtime
• · Clicks, Add to carts, Out of Stocks metrics are generated in Realtime
• Technologies Used:
• · Apache Flink is leveraged for Stream processing
• · Jobs are deployed in GCP Dataproc Clusters using Yarn
• · For storage of Data, Google Cloud Storage is leveraged
• · Orchestration Job are deployed and run using Airflow
• · Data is temporarily stored in Cassandra for efficient processing
• · Prometheus and Grafana are used for Monitoring and Alerting
• · Apache Kafka is leveraged to transfer data between job
• · Hive tables are created on transformed data

Aug 2021 – Oct 2021 · 2 mos

Jul 2019 – Aug 2021 · 2 yrs 1 mo

• Worked mainly on four projects:PNR, BFM,Travel Data Batch and GCP Migration.
• ● PNR project was developed from scratch which includes data ingestion using kafka and then using spark/hadoop for processing data of large size. Project was mainly developed in java which used VTD-XML parser to process xmls and populating tables.
• ● BFM project was developed in Talend which populates multiple hive tables after checking success file of previous job.
• ● Worked on POC for Travel Data Batch and migration of all projects into GCP.

Jan 2019 – Jul 2019 · 6 mos