Subhrant Pandey

DevOps Engineer

Noida, Uttar Pradesh, India7 yrs 7 mos experience

AI ML PractitionerHighly Stable

Key Highlights

7 years of experience in Data Engineering and Backend Development.
Led cost-saving initiatives reducing operational costs by 40%.
Developed automation frameworks enhancing data processing efficiency.

Stackforce AI infers this person is a Backend-focused Data Engineer with extensive experience in Fintech and Big Data solutions.

Contact

Skills

Core Skills

Backend EngineeringBig DataData Engineering

Other Skills

Data PipelinesBig Data FrameworksREST APIsMicroserviceETL FrameworkSpark JobsKafkaMongoDBS3 GlacierAnomaly DetectionAutomationSparkEvent-based NotificationsChatbot DevelopmentVector DB

About

An Inquisitive 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 Development 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 with 7 years of experience in IT industry, working on Data Pipelines, Processing & Warehousing, Big Data Frameworks, Rest API's and have good knowledge of Software Development Lifecycle. 𝐀𝐥𝐬𝐨 𝐩𝐫𝐨𝐟𝐢𝐜𝐢𝐞𝐧𝐭 𝐢𝐧 𝐁𝐚𝐜𝐤𝐞𝐧𝐝 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 𝐚𝐧𝐝 𝐆𝐞𝐧𝐀𝐈.

Experience

7 yrs 7 mos

Total Experience

2 yrs 10 mos

Average Tenure

1 yr 10 mos

Current Experience

Oracle

Senior Member of Technical Staff

Aug 2024 – Present · 1 yr 10 mos · Bengaluru, Karnataka, India · Remote

➢ Working as Senior Data + Backend Engineer in Oracle Health Data Intelligence Team.

Data PipelinesBackend EngineeringBig Data FrameworksREST APIsBig Data

Paytm

Senior Software Engineer

Dec 2021 – Jul 2024 · 2 yrs 7 mos · Noida, Uttar Pradesh, India · Remote

➢ Part of Paytm's 𝐂𝐞𝐧𝐭𝐫𝐚𝐥 𝐃𝐚𝐭𝐚 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐓𝐞𝐚𝐦, tasked with the upkeep and improvement of an in-house
microservice based Ingestion/ETL framework that dealt with 𝐏𝐁𝐬 𝐨𝐟 𝐝𝐚𝐭𝐚 with support for deduplication,
schema evolution, versioning, multi-queue scheduler.
➢ Led the Ingest 2.0 initiative:
Reduced 40% cost ($136𝐤/𝐦𝐨𝐧𝐭𝐡) by upgrading EMR, Stale data deletion & Optimising Spark Jobs.
Implemented 𝐢𝐧𝐜𝐫𝐞𝐦𝐞𝐧𝐭𝐚𝐥 𝐝𝐚𝐭𝐚 𝐜𝐨𝐧𝐧𝐞𝐜𝐭𝐨𝐫𝐬 for Kafka and Mongo in Scala Spark.
Automated LCP-based 𝐚𝐫𝐜𝐡𝐢𝐯𝐚𝐥 𝐟𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤 to S3 Glacier.
𝐊𝐚𝐟𝐤𝐚 𝐫𝐞𝐜𝐨𝐧𝐜𝐢𝐥𝐢𝐚𝐭𝐢𝐨𝐧 framework for anomaly detection & correction.
Automated freshness monitoring for ingest tables, triggering alerts on staleness.
➢ Established 𝐚 𝐜𝐞𝐧𝐭𝐫𝐚𝐥𝐢𝐳𝐞𝐝 𝐛𝐚𝐜𝐤𝐟𝐢𝐥𝐥 𝐬𝐞𝐫𝐯𝐢𝐜𝐞 automating the entire backfill process, including tracking
backfill requests, downstream notifications, realtime status update, event based slack alerts and
Approve/Reject/Terminate built-in functionality.
➢ Developed 𝐚 𝐜𝐞𝐧𝐭𝐫𝐚𝐥𝐢𝐳𝐞𝐝 𝐜𝐡𝐚𝐭𝐛𝐨𝐭 serving as a 24/7 on-call, with automated feed pipeline for Confluence
and RDS data. Utilising vector based similarity search it reduced production issue/data quality
communication by 60% (emails & meetings).

MicroserviceETL FrameworkSpark JobsKafkaMongoDBS3 Glacier+4

Tcs digital

Software Engineer

Sep 2018 – Nov 2021 · 3 yrs 2 mos · Gurugram, Haryana, India · On-site

Worked as Data Engineer in DCS lab (Data Content Services lab).
Worked on File based and 𝐂𝐡𝐚𝐧𝐠𝐞 𝐃𝐚𝐭𝐚 𝐂𝐚𝐩𝐭𝐮𝐫𝐞 tools based Ingestion projects using 𝐒𝐩𝐚𝐫𝐤 & 𝐇𝐚𝐝𝐨𝐨𝐩.
Crafted 𝐒𝐪𝐨𝐨𝐩 based Ingestion Framework.
Developed various shell and python scripts for file handling, parallel processing and error logging, these
can be added to the Ingestion framework as per requirements.
Crafted HQL based 𝐇𝐢𝐯𝐞-Table creation Automation framework termed as Orchestration.

Data EngineeringSparkHadoopShell ScriptingPython