Apr 2025 – Present · 1 yr 2 mos · Hybrid

• Driving the modernization of PayU's data platform — from real-time CDC ingestion to cost-optimized analytics.
• Key initiatives:
• Architected a real-time CDC pipeline (MySQL → Debezium → Kafka → PyFlink → Apache Iceberg on S3) processing 5+ TB/day at 10K events/sec per table, with 2-3 minute end-to-end latency. Live in production, scaling to 20 tables by end of year with planned 4x infrastructure expansion.
• Designed a two-job streaming architecture: stateless staging job (append-only audit trail) + stateful dedup job using ROW_NUMBER over RocksDB with 6-hour TTL and Iceberg merge-on-read upsert. Achieves fault isolation, independent scaling, and schema evolution without reprocessing Kafka.
• Built a production Iceberg health framework on Spark/EMR with automated 5-step optimization (compaction, delete-file resolution, snapshot expiry, orphan cleanup), conflict-aware retries, and SES-based HTML alerting.
• Developed a sync-lag monitoring system using progressive time-window scanning over Iceberg per-file stats, tracking Debezium lag, Flink lag, and end-to-end staleness with severity-based alerting.
• Evaluating Hive Metastore migration to replace AWS Glue Catalog after identifying Athena query costs on Iceberg as unsustainable at scale.
• Continuing to extend the business-centric data mart to cover the finance revenue model. Built an AI-powered agent that auto-converts legacy queries to the new mart schema and onboards business teams.
• Stack: Apache Flink, PyFlink, Apache Kafka, Debezium, Apache Iceberg, Apache Spark, Amazon EMR, Amazon S3, AWS Glue, AWS SES, Python, SQL.

Jan 2023 – Jun 2025 · 2 yrs 5 mos · Hybrid

• Highlights:
• Business-Centric Data Mart on Redshift — Identified that multiple business teams were running redundant full-table scans, driving runaway compute cost. Led a cross-functional initiative spanning all PayU offices to design two star-schema data marts with pre-joined flat tables, standardizing previously undocumented business logic. Result: monthly Redshift spend dropped from $100K to $40-45K while enabling 10x traffic on the same infrastructure. Mart serves 100+ users with 10-15 concurrent analysts.
• Redshift Optimization — Optimized cluster with sort keys, distribution keys, and workload scheduling. Eliminated 8-node concurrency scaling entirely on the primary cluster and consolidated a secondary cluster from 4 to 2 nodes. Drove team-wide adoption of best practices.
• Query Migration at Scale — Built a SQLGlot-based migration script that auto-rewrote 20,000+ legacy queries onto the new mart with 90% success rate, with the remaining 10% migrated manually. Later developed an AI-powered agent for ongoing query conversion.
• Real-Time ML Feature Pipelines — Built feature systems for domestic and international fraud detection models. Streamed raw data into Delta Lake on S3, computed 1-minute aggregated features using Spark Structured Streaming, and stored them in Redis with memory-efficient keys for sub-millisecond model serving. Designed batch fallback workflows from Redshift through S3 with Spark batch jobs.
• Data Governance — Deployed OpenMetadata (2023) as the organization's data catalog, enabling discovery, lineage tracking, and ownership management. Built a self-hosted Great Expectations framework (2024, during audit) for automated data quality validation across NRT and data mart tables.
• Earlier Work — Built dashboards in Apache Superset, ran a Mage AI POC, and collaborated with DS team on user-activity detection models.
• Recognized with the PayU ThankU Award and PayU Ace Award for collaboration, ownership, and delivering high-impact results.