Harsh Agrawal

Data Engineer

Bengaluru, Karnataka, India8 yrs 2 mos experience
Highly Stable

Key Highlights

  • Architected a scalable Event Analytics Framework.
  • Developed a cost-effective Data Ingestion Framework.
  • Implemented secure Redshift Workload Management.
Stackforce AI infers this person is a Data Engineering expert in SaaS and Education sectors.

Contact

Skills

Core Skills

Data EngineeringEvent Analytics FrameworksData IngestionRedshift Workload Management (wlm)Personal Development

Other Skills

APTAWS CodePipelineAWS GlueAWS IAMAWS KinesisAWS LambdaAirflowAlgorithmsAmazon KinesisAmazon RDSAmazon RedshiftAmazon S3Amazon Web Services (AWS)AutomationAvro

About

Results-Driven Data Engineer | Architecting Robust Data Solutions | Transforming Data into Actionable Insights ★ Passionate about leveraging data engineering to empower businesses with strategic advancements and informed decision-making. ★ Pursuing Masters in Data Science From Liverpool John Moores University, England to enhance my skill in AI. CAREER HIGHLIGHTS ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ 🎯 Architected and implemented a seamless Event Analytics Framework, enabling businesses to capture, process, and analyse diverse data points for invaluable insights. 🎯 Developed a spark-based Data Ingestion Framework, simplifying data movement and reducing workload for data engineers through a simple YAML config file update. 🎯 Implemented Redshift Workload Management (WLM) and role-based access system, ensuring enhanced performance, secure data access, and tailored permissions. 🎯 Designed scalable data pipelines to improve accessibility and reliability, contributing to the overall success of data initiatives. AREAS OF EXPERTISE ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ✅ Data Architecture ✅ Data Solutions Implementation ✅ Event Analytics Frameworks ✅ Data Ingestion ✅ Redshift Workload Management (WLM) ✅ Scalable Data Pipelines COLLABORATIVE APPROACH ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ★ Actively collaborate with cross-functional teams to align data solutions with business goals, fostering synergy and driving successful outcomes. ★ Play a key role in code reviews and mentor junior engineers, fostering a collaborative and knowledge-sharing environment. TECH/TOOLS EXPERTISE ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ 🔧 Languages: Python, PySpark 🔧 Data Analytics: Periscope, Google Data Studio 🔧 AWS Services: S3, Redshift, IAM, CodePipeline, Kinesis Streams, Glue, RDS 🔧 Database: MySQL 🔧 Query Languages: PartiQL, HiveQL, PrestoQL, SQL 🔧 Frameworks: Airflow, ETLs 🔧 Version Control: Git, CodeCommit 🔧 Data Manipulation: Pandas Let's connect and explore how my expertise as a dedicated and skilled Data Engineer can drive data-driven insights and contribute to the success of your organisation. Reach out to me to discuss potential collaborations and exciting opportunities!

Experience

8 yrs 2 mos
Total Experience
2 yrs 2 mos
Average Tenure
1 yr 7 mos
Current Experience

Jpmorganchase

Data Engineer II (Associate)

Nov 2024Present · 1 yr 7 mos · Bengaluru, Karnataka, India

TerraformAWS GlueAWS LambdaData Engineering

Cuemath

2 roles

Senior Data Engineer (SE-II)

Oct 2023Aug 2024 · 10 mos · Bengaluru, Karnataka, India

  • ➤ Architected and implemented an Event Analytics Framework designed to process over 50 million events daily. Leveraging AWS Kinesis, Redshift, and S3, this system efficiently captures, processes, and analyzes diverse data points from multiple sources. The framework plays a critical role in transforming raw data into actionable business insights, enabling organizations to make informed decisions and drive strategic advancements. This solution emphasizes scalability, reliability, and cost-effectiveness, addressing complex data needs in a dynamic business environment.
  • ➤ Direct Mapping Project: Built a system to match student leads from the website with the most suitable teachers using a scoring model based on preferences and performance metrics. Integrated data from Amazon RDS and Redshift with AWS Glue, processed it using Pandas, and applied a model to evaluate and rank teacher-student pairings. The top 20 matches were stored in S3, with notifications sent via SQS to trigger the selection process on the app.
Data LineageQuery OptimizationValidation RulesBusiness MetricsCI/CDData Lakes+18

Data Engineer

Jul 2022Sep 2023 · 1 yr 2 mos · Bengaluru, Karnataka, India

  • ➤Data Ingestion Framework: Developed an automated system to sync production data from Amazon RDS (Postgres) to Amazon Redshift, replacing the costly and manual Amazon DMS solution. Utilised a YAML configuration file for flexible table management, and employed AWS Glue for data ingestion, handling complex data types with Redshift’s SUPER type. Implemented a masked view approach to protect sensitive (PII) data, ensuring that BI tools only access anonymised data. This solution allows developers to easily add new tables without Data Engineering intervention, significantly reducing costs and improving efficiency.
  • ➤ Redshift Workload Management and Role-Based Access System: Implemented Redshift WLM to optimise query performance and resource allocation, while introducing a role-based access system to secure data and tailor permissions for analysts, finance professionals, and engineering teams. Enhanced performance, ensured secure data access, and provided customised permissions to meet diverse user needs.
Data LineageQuery OptimizationAWS CodePipelineGlue StreamingPython (Programming Language)MySQL+38

Nineleaps

4 roles

Software Developer Engineer II

Promoted

Apr 2022Jul 2022 · 3 mos

  • Client: Uber Adtech
  • Key Responsibilities:
  • Data Extraction and Improvement: Developed plugins for ad data extraction, focusing on enhancing data reliability and efficiency.
  • Pipeline Design: Designed scalable data pipelines for improved accessibility and reliability of ad data.
  • Infrastructure Maintenance: Built and maintained data infrastructure to ensure optimal performance and data integrity.
  • Team Collaboration: Contributed to code reviews, mentored junior engineers, and fostered a collaborative environment.
  • Data Integration and Quality: Integrated data from various sources into Hadoop, implementing rigorous monitoring and quality checks.
Data LineageQuery OptimizationPython (Programming Language)MySQLAvroInsight Generation+26

Software Developer Engineer I

Oct 2020Mar 2022 · 1 yr 5 mos

  • Client: Uber Rider, Uber Transit
  • Key Responsibilities:
  • Stakeholder Collaboration: Engaged with multiple stakeholders to understand their needs and deliver tailored insights and dashboards.
  • Data Pipeline Optimization: Monitored and validated data pipelines to enhance reliability and performance, using DFDs and Python-based ETL pipelines.
  • Application Development: Developed scalable applications within the Hadoop Ecosystem to support distributed data processing.
  • Recognition: Received the Nineleaps Spotlight Award for outstanding performance on the project.
Data LineageQuery OptimizationPython (Programming Language)MySQLAvroInsight Generation+25

Member Of Technical Staff - II

Aug 2019Oct 2020 · 1 yr 2 mos

  • ➤ Developing pipelines for extraction of data from a wide variety of data sources.
  • ➤ Working on the implementation of ETL and data processes for structured and unstructured data.
  • ➤ Responsible for the Design, build, test and maintain scalable and stable applications to support distributed processing using the Hadoop Ecosystem.
  • ➤ Identifying ways to improve data reliability and efficiency.
  • ➤ Maintaining data quality for all data ingestion pipelines.
  • ➤ Responsible for finding out the root cause of failure, delivering updates to stakeholders based on analysis and then fixing it.
  • ➤ Writing testable code for optimal level of code coverage.
Python (Programming Language)MySQLAvroAirflowPandas (Software)CI/CD+10

Graduate Engineering Trainee

Jan 2019Aug 2019 · 7 mos

  • Key Achievements
  • ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  • ★ Developed an internal React WebAPI project for Checklist Management System Portal.
  • Tool & Framework Used
  • ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  • ► Languages: NodeJS, JavaScript, CSS, HTML
  • ► Framework: ExpressJS , Mocha (Test Framework), SonarQube (Continuous Code Quality Inspection)
  • ► Database: MySQL, MongoDB
  • ► Documentation: Swagger API
Python (Programming Language)MySQLMongoDBCommunicationNode.jsGit

Ib hubs

Alumni Mentor at iB Hubs Startup School 2021

Jun 2021Jul 2021 · 1 mo · Remote

  • iB Hubs Startup School is a four-week-long student accelerator program where students with innovative ideas are put through intense learning after which, they're ready to successfully build and execute their business plans.
  • They learn many things from thinking about their idea and make them executable to apply the learnings in the lifestyle itself.
  • My main role here was to guide and mentor them with my knowledge and experience.
CommunicationBusiness Model CanvasStartup DevelopmentBusiness DevelopmentMentoring

The entrepreneurship cell, glbitm

President

Oct 2018Jan 2019 · 3 mos · Greater Noida

EVENT MANAGEMENTCommunicationLeadershipTeam Management

Money-wizards

Campus Entrepreneur

Aug 2018Oct 2018 · 2 mos · Greater Noida

  • Money-Wizards, a pioneer company in financial education to bring personal finance awareness
  • among college students across India.
  • The Youth Money Olympiad(YMO) is administered as a personal finance assessment test in the multiple choice question format. It had been a phenomenal success that reached more than 33,000 students across 400 esteemed colleges from 100+ cities in India including students from the esteemed institutes.
  • My job is to promote their event “Youth Money Olympiad” in my college campus. This internship helped me build my communication, marketing and selling skills in the process.
EVENT MANAGEMENT

Ib hubs

STARTUP SCHOOL 2018

Jun 2018Jul 2018 · 1 mo · Lucknow, India · On-site

  • iB Hubs Startup School is a four-week long student accelerator program where students with innovative ideas are put through an intense learning after which, they're ready to successfully build and execute their business plans. They learn many things from thinking about their idea and make them executable to apply the learnings in the lifestyle itself.
  • I learnt a ton of things which I find helpful in my personal as well as professional life. A few things that I learnt in context of my idea were:
  • ➤ Building a product
  • ➤ Creating a strong value proposition
  • ➤ Creating a pitch
  • ➤ Building a strong team
  • ➤ Improving myself everyday
Programmable Logic Controller (PLC)PLC Ladder LogicAutomation

The entrepreneurship cell, glbitm

2 roles

Member of Media and Advertisement

Oct 2017Jul 2018 · 9 mos · Greater Noida

EVENT MANAGEMENTPoster Design

Head of Media and Advertisment

Aug 2017Oct 2018 · 1 yr 2 mos · Greater Noida

EVENT MANAGEMENTPoster DesignLeadershipTeam Management

Finetech controls

Summer Internship (Industrial Automation)

Jul 2017Aug 2017 · 1 mo · Greater Noida · On-site

  • ➤ Gain hands-on experience and practical skills in the dynamic field of industrial automation through this immersive summer internship program.
  • ➤ Collaborate with industry professionals, work on real-world projects, and learn to design, implement, and troubleshoot automation solutions.
  • ➤ Develop expertise in PLC(Programmable Logic Controllers) programming.
  • ➤ Industrial Instrumentation
  • ➤ Coding: Ladder Logic
Personal DevelopmentEarly-Stage StartupsCustomer Value PropositionTeam ManagementPresentation Skills

Education

Liverpool John Moores University

Master of Science - MS — Data Science

Jan 2024Dec 2025

International Institute of Information Technology Bangalore

PG Diploma in Data Science and AI — Deep Learning

Jan 2024Jan 2025

GL Bajaj Institute of Technology and Management

Bachelor of Technology (B.Tech.) Hons. — Electronics and Communication Engineering

Jan 2015Jan 2019

Central Board of Secondary Education

Senior Secondary School — Physics-Chemistry-Math

Jan 2013Jan 2015

Central Board of Secondary Education

High School — Foundation

Jan 2002Jan 2013

Stackforce found 100+ more professionals with Data Engineering & Event Analytics Frameworks

Explore similar profiles based on matching skills and experience