Mar 2025 – Present · 1 yr 3 mos

Nov 2022 – Mar 2025 · 2 yrs 4 mos · On-site

• Led the architecture, design, and development of ETL pipelines for the Commercial domain, handling large volumes of data efficiently.
• Streamlined CI/CD workflows by leading the Jenkins pipeline migration and transitioning deployments to Tesco’s Private Cloud environment.
• Eliminated legacy/dead code and modernized core ingestion systems, significantly improving maintainability, scalability, and developer onboarding.

Aug 2021 – Aug 2022 · 1 yr

• The ETL platform backend was modernised by transitioning from PL/SQL
• to Apache Spark with Scala, led by me. This reduced lot of load on older
• infrastructure.
• I designed ETL pipelines using Spark to ingest data from various sources
• such as MongoDB, Oracle, Postgres, Kudu, and Hive, providing data solutions for the entire company.

Sep 2019 – Aug 2021 · 1 yr 11 mos

• The development of "Vault," a data quality validation framework implemented using Apache Spark and Scala, was led by me.
• This framework assesses and certifies data quality through the application of diverse rules and generates detailed reports. It even defines the
• data strategy for the clients and troubleshooting issues.
• Data modeling for this project was crucial due to its integration with
• various datasets. After collaborating with multiple stakeholders, the application was meticulously designed.

Mar 2018 – Sep 2019 · 1 yr 6 mos

• Built Micro-service with the help of Scala Play Framework to validate the consistency incoming data in Avro Format. This framework uses non-blocking asynchronous I/O operations. It is designed to avoid any kind of blocking request. Every operation returns immediately, freeing the running thread and resuming execution when it is over. Accessing the database is not a bottleneck.

Mar 2017 – Feb 2018 · 11 mos

• Developed a Real Time Data Streaming platform from OLTP to the OLAP Platform.
• The streaming happens from OLTP layer to OLAP layer with the help of Apache Spark and Striim.
• The USP of this project is that data is insync in the OLAP Platform within a few milliseconds.
• We have been using Change Data Capture Mechanism to read from OLTP. A number of challenges were involved in this project, which includes:
• (a) Sanity of Data and consistency of Data
• (b) Data Type sync related issues
• (c) Zero Down Time and Backup
• (d) Loading Balancing

Jul 2015 – Feb 2017 · 1 yr 7 mos

• Got Introduced in the field of Data Analytics. With the help of Hadoop Infrastructure solved many business problems for the firm. Being in an organization where data is the backbone, it was important to find insights from the data and make it more meaningful.
• Wrote a number of algorithms with Map-Reduce and HQL. Python/Java and SQL has been extensively used in these projects and scheduler used is oozie.

Jul 2014 – Jun 2015 · 11 mos

• Worked on enrichment of data on the aggregation platform, the project intended to minimize data quality issues on the application. As Financial Aggregators and handling multiple financial institutions there exists a diversified segment for quality related issues.
• The role involved to write complex PL/SQL queries and generate reports from Oracle Server and based on the issues reported fix the issues on the aggregation platform.
• Technologies Used - Core Java, Selenium, PL/SQL