Nidhi Jain

Data Engineer

Bengaluru, Karnataka, India10 yrs 3 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Expert in Big Data Systems and Data Engineering.
Proven track record in data quality and integrity.
Strong experience with real-time data processing technologies.

Stackforce AI infers this person is a Backend-heavy Data Engineer specializing in Big Data and Streaming Technologies.

Contact

nidhi.jain@system73.com LinkedIn

Skills

Core Skills

Data ArchitectsStreaming PlatformBig Data AnalyticsData IntegrityData WarehousingData Processing

Other Skills

A/B TestingAWS LambdaAgile MethodologiesAirflowAmazon KinesisAmazon RedshiftAmazon Web Services (AWS)Apache AirflowApache FlinkApache KafkaApache SparkApache Spark StreamingCadence workflowContinuous Integration (CI)Continuous Integration and Continuous Delivery (CI/CD)

About

I’m a Software Engineer with enthusiasm towards working with data at scale & distributed systems. I’m looking forward to learning in-depth architecture and design of Big Data Systems. Previously I’ve worked on Data Lake/Mart, ETL pipelines, recommendation systems, etc. Currently seeking out new opportunities to learn.

Experience

System73® - powering profitable streaming

Data Engineer

Dec 2023 – Present · 2 yrs 3 mos · Worldwide · Remote

Data ArchitectsApache FlinkApache KafkaStreaming Platform

Uber

Software Engineer II

Jan 2021 – Mar 2023 · 2 yrs 2 mos · Bengaluru, Karnataka, India

Developed and refined data quality platform for 30,000+ Hive datasets, ensuring data accuracy, consistency, and reliability. Collaborated with database teams to integrate real-time config sources, including Elasticsearch, Pinot, and MySQL, slashing test times from hours to minutes across 1M+ runs. Further, augmented platform capabilities with advanced multi-column data quality checks, comparing 1000+ columns across diverse data sources

Cadence workflowData ModelingApache KafkaBig Data AnalyticsPython (Programming Language)Data Integrity+10

Razorpay

2 roles

Senior Data Engineer

Apr 2019 – Jan 2021 · 1 yr 9 mos

Apache Spark StreamingData ModelsData ModelingBig Data AnalyticsSQLAWS Lambda+7

Data Engineer

Aug 2018 – Jan 2021 · 2 yrs 5 mos

Engineered a data lake to handle a billion transactions, leveraging Lambda Architecture with Sqoop for batch synchronization from MySQL to S3 and Maxwell for real-time integration. Achieved real-time streaming of 720M daily events using GoLang, routing to Kafka and processing with Spark Streaming into Hive. Additionally, designed a data warehouse utilizing a star schema for optimized dimension/fact modeling.

Apache SparkApache Spark StreamingData ModelsApache KafkaDockerBig Data Analytics+14

Meaww

Backend Developer / Data Engineer

Aug 2017 – Aug 2018 · 1 yr · Bengaluru Area, India

Led the transition of website analytics from Google Analytics to a custom data pipeline utilizing Amazon Redshift and Kinesis Firehose. Enabled advanced search capabilities for articles/authors via ElasticSearch. Also, developed a rule-based system for metadata extraction tailored for Analytics Systems
Created a rule-based meta-extraction system that helps to extract metadata for Analytics Systems.

Apache SparkApache Spark StreamingAmazon KinesisBig Data AnalyticsSQLAWS Lambda+10

Versé innovation pvt ltd

Software Developer

Jul 2015 – Jul 2017 · 2 yrs · Bengaluru, India

Deployed real-time and batch processing for user action logs using Lambda Architecture, enhancing with Apache Flink aggregations, tailored emailers, and dashboards. Developed a User Base-Gen system, personalizing experiences for 60M+ daily users based on attributes like source and language, leveraging Apache Spark.

Apache SparkApache Spark StreamingApache KafkaBig Data AnalyticsSQLA/B Testing+11

Juniper networks

Intern

Jan 2015 – Jun 2015 · 5 mos · Bengaluru Area, India

My responsibilities included analysing the network logs and aggregating information like Bandwidth Usage per user, Application, Firewall Rules, IP Source-Destination pair, etc.