Kaushal Kumar

Senior Software Engineer

Bangalore, Karnataka, India5 yrs 8 mos experience

Key Highlights

Over 5 years of experience in data engineering.
Expert in building robust data platforms across industries.
Skilled in architecting data pipelines and data models.

Stackforce AI infers this person is a Data Engineer specializing in Big Data and Cloud Computing for Fintech and SaaS industries.

Contact

Skills

Core Skills

Data EngineeringBig Data AnalyticsCloud Computing

Other Skills

Core JavaJavaScalaSpark StreamingKafkaGCP BigQueryPostgreSQLHiveSparkSpring BootPySparkPythonAWS S3AWS GlueAWS Lambda

About

IIT Delhi graduate with over 5 years of experience in building robust data platforms across diverse domains such as e-commerce and banking. Skilled in architecting data pipelines and designing efficient data models to solve complex business problems. Passionate about taking on challenging projects and collaborating with inspiring leaders to drive innovation and continuous growth.

Experience

5 yrs 8 mos

Total Experience

1 yr 11 mos

Average Tenure

1 yr 9 mos

Current Experience

Target

Senior Software Engineer

Sep 2024 – Present · 1 yr 9 mos · India · Hybrid

Re-architected the batch processing data pipeline, which previously relied on full snapshot loads from upstream systems, into a near real-time data pipeline.
As part of the re-architecture, migrated the on-premise legacy system to GCP. Implemented incremental data consumption from Kafka streams and processed the data using Spark Streaming before loading it incrementally into BigQuery. The solution primarily utilized Scala, Spark Streaming, Kafka, and GCP BigQuery.
Worked on developing a custom logging framework that captures all types of API failures in a PostgreSQL database, enhancing system visibility, reliability, and auditability. This framework also enabled the implementation of an automatic retry mechanism for failed APIs
Worked on developing multiple ingestion pipelines for Criteo APIs, where data is fetched through the Criteo API, processed, and then stored in Hive tables for end users.
Worked on developing an ingestion pipeline to fetch campaign insights and various levels of data from the Meta SDK. Also contributed to processing the ingested data and creating the final table for end users. Primarily used Java, Spring Boot, Spark, Scala, and Hive to achieve this

Core JavaJavaScalaSpark StreamingKafkaGCP BigQuery+2

Idfc first bank

Senior Data Engineer

Mar 2023 – Aug 2024 · 1 yr 5 mos · Bengaluru, Karnataka, India · Hybrid

Worked on building the datalake platform at idfc first bank. Majorly involved in migrating the reporting layer from legacy platform to new IDP platform using pyspark, python, hive etc. I have also worked on migrating the few pipelines on AWS cloud using S3, glue, lambda, Athena , EME services

PySparkPythonHiveAWS S3AWS GlueAWS Lambda+3

Standard chartered bank

Big Data Developer

Aug 2020 – Feb 2023 · 2 yrs 6 mos · Bangalore Urban, Karnataka, India

Responsible for both real time and batch trade processing. Working in various structured and unstructured data transformations to cater to daily needs of various end users. Working in Scala+spark+ java and hive currently. Designed the bigData related pipelines of PacMan from Scratch. Deal with GBs of data every hour and process them successfully. Written the monitoring script to send alert to client for report generation. Written unit test cases to test clases in different functionality

ScalaSparkJavaHiveSqoopBig Data Analytics+1