Jul 2024 – Present · 1 yr 11 mos · Bengaluru · Hybrid

• Engineered a scalable Dataproc PySpark–driven Property and Destination Master pipeline processing 2 TB of multi-source data (TSV, SQLite, partner feeds) daily, implemented branching logic, validation, joins, and modular refresh steps (DSA/DM/PM), improving data consistency and reducing manual interventions.
• Engineered and optimized Spark-based ETL pipelines to ingest and process 700-900 GB of daily Meta partner conversation data, including ad performance metrics and user engagement logs; reduced runtime by 35% via partitioning and caching, enabling near-real-time dashboards for accelerated business decisions.
• Developed Spark ML feature pipelines for 1.4-2 TB of social/app reviews, integrated text cleaning, language detection and topic modeling, delivering optimized datasets and aggregate views via clustering and Streams Tasks to empower model iteration and stakeholder insights.
• Revamped traveler clickstream pipeline in Spark by migrating to a new data source, deriving 100% of column mappings for full integrity, collaborated with upstream teams to standardize schemas, boosting efficiency by 30%.
• Migrated and optimized 23 Hive SQL queries to Trino SQL, resulting in improved query performance and faster root cause analysis of dataset issues in production.