Deepak Singh

Data Engineer

Delhi, India13 yrs 9 mos experience
Most Likely To SwitchHighly Stable

Key Highlights

  • Over 13 years of experience in big data technologies.
  • Led data engineering teams to improve efficiency by 50%.
  • Saved $1.5 million annually through innovative data techniques.
Stackforce AI infers this person is a Big Data Engineering expert with a focus on data architecture and analytics in the Fintech sector.

Contact

Skills

Core Skills

Data ArchitectureBig DataData Warehouse ArchitecturePython (programming Language)Data ModelingApache Spark

Other Skills

People ManagementLooker (Software)TrinoContinuous Integration and Continuous Delivery (CI/CD)JenkinsPySparkSQLGrafanaStarburstAirflowAzakabanApache KafkaExtract, Transform, Load (ETL)Apache ZeppelinCursor AI

About

13+ Years of experience with big data technologies and 8+ years of experience in managing data engineering, analytics teams with leveraging big data technologies like Java, Scala, Python, Spark, Kafka, stream processing and SQL/NoSQL systems. Expertise in building data strategy for the org. Designing data platform and establishing complete data vertical from scratch to build scalable components of platform. Technical excellence in : * Architecting and leading implementation efforts for end to end data applications and frameworks. * Proficient in building high scale analytics system. * Hands on with data engineering paradigm and distributed systems. * Lead multiple data engineering and data application projects from scratch to production deployment.

Experience

13 yrs 9 mos
Total Experience
2 yrs 9 mos
Average Tenure
6 yrs 3 mos
Current Experience

Paytm

4 roles

Data Engineering Manager

Promoted

Jul 2024Present · 1 yr 10 mos

  • 🔹 Developed data strategies and led implementation of end-to-end data applications at Paytm.
  • 🔹 Managed a team of 10 engineers, achieving a 50% improvement in data processing efficiency.
  • 🔹 Saved $1.5 Million annually through innovative data compression techniques.
  • 🔹 Engineered a SQL optimizer, enhancing query performance by 50%, empowering Business Analysts
  • with improved data retrieval.
People ManagementLooker (Software)TrinoData ArchitectureContinuous Integration and Continuous Delivery (CI/CD)Jenkins+15

Senior Technical Lead

Apr 2023Jun 2024 · 1 yr 2 mos

  • 🔹 Efficiently Migrated data processing workflows from Hive to Starburst, resulting in a 50% enhancement in query performance and a 30% reduction in compute costs.
  • 🔹 Developed Python scripts for Starburst POC, performance comparisons, and load testing with Dremio and Hive
  • 🔹 Maintained and optimized a feature store using transaction data to calculate customer features for
  • lending assessments, reducing job run time from 5 hours to 15 minutes.
  • 🔹 Collaborated with Paytm business verticals to understand use cases for analysis and reporting.
  • 🔹 Successfully completed a POC for Looker, testing all features.
  • 🔹 Migrated Paytm's analysis and reporting system to Looker from Google BigQuery, saving $1 million
  • annually.
  • 🔹 Contribute in Google Case Study for successful migration from Google Big Query to Looker in a duration of 1 month
Apache SparkPeople ManagementLooker (Software)TrinoData Warehouse ArchitecturePython (Programming Language)+10

Technical Lead

Nov 2021Mar 2023 · 1 yr 4 mos

  • 🔹 Enhanced multi-partition support for ETL jobs at Paytm, boosting parallel processing and query performance by 63% improvement.
  • 🔹 Developed a robust data architecture, reducing data latency by 60%.
  • 🔹 Developed OLAP solutions processing 1 billion daily rows for Recharge, Utilities, Movies, and Travel verticals, reducing infrastructure costs by 30% and improving data retrieval speeds by 40%.
  • 🔹 Reduced Query run time and amount of data by 80% by implementing Data modelling techniques.
  • 🔹 Achieved a 40% cost reduction in Google Big Query usage by storing data to cache in Google Big Query Table, saving ~$20K per month.
  • 🔹 Collaborated with 50+ cross-functional teams to define data strategies and drive key business decisions.
Apache SparkPython (Programming Language)ScalaHiveGoogle BigQueryJenkins+9

Senior Data Engineer

Nov 2019Oct 2021 · 1 yr 11 mos

  • 🔹 Migrates 30 PB data from on-premises to AWS and develops the PACE (Paytm Advanced Compute Engine) framework to manage all data processing through the creation of 300 OLAPs and 100 Fact tables for Data analytics and Visualization of entire 55 Paytm Business Verticals for smooth transition from Dahleez(on-premises) to Google BigQuery.
  • 🔹 Enabled Star schema design to reduce complexity and improve query performance.
  • 🔹 Use clustered and non-clustered indexes on fact and dimension tables to speed up data retrieval.
  • 🔹 Partition large fact tables to improve query performance and cube processing.
  • 🔹 Define strategic aggregations to balance storage and query performance and Analyze query patterns and create aggregations for frequently accessed data.
Apache SparkData Warehouse ArchitectureScalaApache KafkaApache ZeppelinPySpark+5

Nucleus software

Senior Data Engineer

Dec 2017Nov 2019 · 1 yr 11 mos · Noida Area, India

  • Data Ingestion to HDFS from RDBMS and Archived files using Sqoop
  • Applied ETL operations for data cleansing and data processing using Pig
  • Exposed the cleansed Data in a tabular format using Apache Hive to other APIs
  • Developed a centralized store for capturing customer’s information using NoSQL Columnar DB,
  • HBase for random read and write operations
  • Developed UDFs using Java in Hive for custom operations resulting in lesser number of queries.
  • Analyzed millions of transactional log files using Apache Spark (PySpark) to predict the customer’s
  • transactional behavior and thus improvising customer experience.
Apache SparkPeople ManagementData Architecture

Tata consultancy services

Senior Java Developer

Nov 2016Nov 2017 · 1 yr · New Delhi Area, India · On-site

  • Developed module for migration of existing RDBMS Data to HL7 (US Standard of Health Industry) using Core Java, Hibernate, Oracle database and Big Data technologies
  • Implemented Data Migration from Archived Files (apprx. 40,000 records/file) to HL7 Standard using the concept of External Table reducing efforts in terms of time and complexity by 60%
  • Enhancing current and developing new modules in accordance with the client’s requirements.
Apache SparkData Architecture

Indian agricultural statistics research institute icar), new delhi

System Analyst

Sep 2012Oct 2016 · 4 yrs 1 mo · New Delhi Area, India

  • To create an online application for registration of applicant , who is applied for various post such as T3-Computer, T3-Statistics, T3-Library and Lower Division Clerk in IASRI. I have design the payment option for registration of applicant. There are 1 lakh plus user in it, So maintaining the database on daily basis is also my responsibility. I have created various reports based on requirement by our administration in this application.
  • To create an online application for organizing International Conference on Controlled Atmosphere and Fumigation in Stored Productswhich will be in India on November 7-11, 2016. I have design the complete application including its front view and database. I have created online registration and payment module for Indian as well as International participants. There is also an abstract submission in it. I have created various reports based on requirement

Nucleus software

Software Trainee

Feb 2012Aug 2012 · 6 mos · Noida Area, India

  • I am working on Customer Acquisition System which is related to lending money to customer. This module has multiple workflows. Process starts when customer approaches the bank for his loan requirement. Workflow initiate once we enter the customer de

Education

Punjab Technical University

Bachelor of Technology - BTech — Computer Engineering

Jan 2007Jan 2011

Lovely Professional University

Master of Science — Computer Science

Jan 2016Jan 2018

Stackforce found 100+ more professionals with Data Architecture & Big Data

Explore similar profiles based on matching skills and experience