Feb 2023 – Jul 2023 · 5 mos · Bangalore Urban, Karnataka, India

• Research and Tech

Feb 2021 – Feb 2023 · 2 yrs · Bangalore Urban, Karnataka, India

Apr 2020 – Feb 2021 · 10 mos

• Data Quality Tool as a service for metric computation and MYSQL based metrics repository feeding grafana dashboards providing graphical representation of data and alerting.
• Apache Spark framework for configurable, generic ETL and cross connectivity across data sources.
• Working on custom rule based realtime ETL engine based on Apache Flink.

Nov 2019 – Apr 2020 · 5 mos

• Serverless data platform backed by AWS S3, Glue and athena, powered by Apache Spark, Kafka, Sqoop, Airflow.
• Apache Atlas based data discovery layer providing insight into platform data.
• Realtime data pipeline (Kafka) with Avro, Schema registry and Kafka Connect consumer and Apache flink for stre processing.
• Airflow Scheduler

Jan 2019 – Nov 2019 · 10 mos · Gurgaon, India

• Serverless Reporting Platform
• Created pyspark/scala ETL framework leveraging Apache Spark facilitating analytics on app,sensor data.
• Created DAG framework for Apache Airflow facilitating scheduling of data pipeline.
• Developed enhanced Kafka Consumer with added support for custon JSON message parsing and S3 integration
• Pipeline integration with AWS services, S3, Athena, Glue, Apache Superset.
• Expected Time of Arrival (ETA)
• Benchmarked performance of existing model
• Revamped the input data pipeline making it 20x more efficient down to 2 hours.
• Improved features to reflect heavier weightage of recent data.
• Key Result Areas for Transport Network
• Designed KRAs to measure network health, automated for regular reporting to management.
• Maintenance of Spring Boot based microservice for auto approval of fuel bills using OCR.
• Maintenance of flask based python API service for computation of best route using Dijkstra's and other constrains.