Dinesh Jagannathan

Software Engineer

Bengaluru, Karnataka, India17 yrs 6 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

16+ years of experience in distributed systems and big data.
Led multiple high-impact projects at Uber and Flipkart.
Expert in real-time streaming and data modeling.

Stackforce AI infers this person is a Big Data Architect with expertise in SaaS and E-commerce analytics.

Contact

Skills

Core Skills

Distributed SystemsData ModelingEtlBig DataData EngineeringAnalyticsMicroservices

Other Skills

JavaHadoop (HDFS, Map Reduce, Spark, Hive)Apache FlinkRESTful WebServicesApache KafkaApache SparkFlinkSparkHBaseAzkabanMySQLDropwizardMapReduceRedisMongoDB

About

Architect with 16+ years of commendable experience on distributed systems, big data, real time streaming, micro services, data modelling, application development and web technologies Please note: 1️⃣ Job titles mentioned on this profile are last ones held at each company. 2️⃣ Projects mentioned are important ones only.

Experience

17 yrs 6 mos

Total Experience

2 yrs 11 mos

Average Tenure

4 yrs 7 mos

Current Experience

Uber

Staff Software Engineer - TLM

Oct 2021 – Present · 4 yrs 7 mos · Bengaluru

1. Redesigning Uber's batch authoring platform for improved developer productivity, reliability and deployment safety
2. As a tech lead in UberForBusiness(U4B) risk backend team , responsible for leading, designing and building the B2B trips auth hold capabilities .
3.Led the Data Drift Detector Framework (D3) project to automatically identify data drifts in critical columns in all of Uber wide datasets.
○ Co-Authored Uber Eng blog on the same.
https://www.uber.com/en-IN/blog/d3-an-automated-system-to-detect-data-drifts/
4. Architect for Sparkle - A framework on top of spark for writing modular ETLs at Uber. This framework abstracts out the boilerplate code, provides out of the box ETL testing capabilities and helps developers focus on writing business logic either as SQL or as procedural code in Java/Scala or Python.
○ Authored Uber Eng blog on the same.
https://www.uber.com/en-IN/blog/sparkle-modular-etl/
5. Built Flow Analysis Framework which helps in improving the complex funnel analysis
from weeks to within hours and provide rule based attribution support.

JavaHadoop (HDFS, Map Reduce, Spark, Hive)Distributed SystemsData ModelingApache FlinkRESTful WebServices

Flipkart

Data Engineer 4

May 2019 – Oct 2021 · 2 yrs 5 mos

I was part of Data Products team in Flipkart and below are some of the projects I worked on.
1. Responsible for leading and building an in-house clickstream solution that enables consumers with realtime and batch analytics on Flipkart clickstream data with a scale upto 100kqps traffic. Tech stack includes flink, spark, druid, hbase and azkaban.
2. Designed and developed Unified Metric Store(UMS), serving both as OLAP processing engine plus datastore over Hadoop.
With UMS, users can configure MOLAP cubes over Kafka(realtime) and hive(batch) tables, supporting metrics compute and consumption at different dimensions within a few seconds of latency.
It has support for various aggregators including approx aggregators like datasketches and handles a large cube cardinality (500 million+), and manages high traffic data (10TB/day).
UMS allows for flexible roll-ups, schema evolution and automated backfill.
The tech stack includes Spark, Flink, HBase, Dropwizard,Azkaban, MySQL, and Ansible. UMS successfully scaled for Big Billions Days,processing petabytes of data within SLAs and experiencing zero downtime.

FlinkSparkHBaseAzkabanMySQLBig Data+1

Near

Lead Software Engineer

Dec 2017 – Apr 2019 · 1 yr 4 mos · Bengaluru Area, India

Near is the world's largest source of intelligence on people and places, processing data from over 1.6 billion monthly users across 44 countries. The Near Platform powers data-driven marketing and enrichment offerings through a suite of SaaS products. As Lead engineer below are my role and responsibilities
1. Designed and implemented the Near rule engine framework where around 200gb of user data per day is profiled across 30k card rules over 18+ countries.
2. Designed and implemented the rule replayer framework where the newly created card rules are profiled across 30 to 90 days user data.
3. Redesigned entire campaign analytics module from MapReduce jobs to Spark jobs and consolidation of datastores as part of server optimization project.
4. Redesigned and implemented footfall analytics module that has 3TB source data per day from MapReduce to Spark jobs and optimized the job runtime from 23 hours to 4 hours per day with just half of the resource utilization.
5. Designed and implemented attribution and lift index logic for the ad-campaigns that helps in deciding the effectiveness of the impressions delivered.
6. Designed and implemented a real time campaign analytics streaming service using flink that helps in frequency capping of ad-serving campaigns.
7. Built active users api module for audience insights services and footfall analytics api module for user visits services
8. Data modelled and implemented redis and mongo clusters as data stores for footfall and campaign analytics modules respectively.

SparkMapReduceFlinkRedisMongoDBData Engineering+1

Akamai technologies

Senior Engineer II - Data

Jan 2012 – Dec 2017 · 5 yrs 11 mos · Bangalore

1. Real-time data streaming services using message queues and pub-sub pattern.
2. Developed RESTFUL micro services for Account and Contract Hubs of DaSH platform.
3. Batch pipelines and data modelling
4. Design and implement DaSH platform OAuth2 provider that serves as the security layer for all our platform API’s.
4. Design and implement logging and error alerting layer in the data platform
5. Design and implement one click deployment in the ETL platform

RESTful Web ServicesData ModelingMicroservicesData Engineering