Gaurav Rawat

Data Engineer

Bengaluru, Karnataka, India3 yrs 11 mos experience

Most Likely To Switch

Key Highlights

Expert in building scalable Big Data pipelines.
Proficient in AWS and Databricks for data solutions.
Strong background in data quality and ETL frameworks.

Stackforce AI infers this person is a Data Engineering expert in SaaS with a focus on Big Data solutions.

Contact

Skills

Core Skills

Data EngineeringEtl Development

Other Skills

AWSAirflowCI/CDCloud Cost OptimizationDBTData LakeHouseData QualityData WarehousingDatabricksDockerETLGitHubGrafanaGreat ExpectationsHadoop

About

As a Data Engineer with working in transforming data into growth, I have developed expertise in Hadoop, Hive, Python, Spark, Kafka, SQL, AWS, Databricks, Snowflake, Terraform , Airflow, Kafka, Spark Streaming, CI/CD, data modeling, and data pipeline technologies. • Experience in designing, developing, deploying, debugging and maintaining Big Data pipelines using Hadoop, Apache Spark, Python, SQL, AWS, Airflow and Hive. • Experience in developing high performance and scalable solutions that extract, transform and load Big Data. • Experience developing PySpark Modules using Dataframes API, Spark SQL and their optimization. • Proficiency in writing and optimizing SQL queries. • Experience building and optimizing 'Big Data' Data Pipelines, architectures and datasets involving petabyte and terabyte of data. • Experience in working with unstructured and semi-structured data. • Experiece in AWS Stack : EC2, EMR, S3, Airflow , Athena, CloudFormation Skills: Big Data, Python, SQL, Spark, AWS EMR, Databricks, Snowflake, DBT (Data Build Tool), Hadoop, Hive, Docker, Airflow, CI/CD, Terraform, GitHub, Data Quality, Kafka, Spark Streaming, ETL Development, and Data Warehousing. Feel free to reach out to me at gauravrawat141999@gmail.com

Experience

Autodesk

Software Engineer 2 - Data Engineering

Mar 2024 – Present · 2 yrs · Bengaluru, Karnataka, India · Hybrid

Nike

Data Engineer

Jul 2022 – Feb 2024 · 1 yr 7 mos · Bengaluru, Karnataka, India

. Driving migration of various data use-cases from existing native AWS data lake to Databricks Delta.
. Designed and implemented a centralized Data LakeHouse architecture to consolidate data
from multiple sources, enabling seamless integration and analytics.
. Build Standard ETL Framework to Egress data from Kafka streaming to Raw, Cleanse and Curated layer of databricks Lakehouse to process Terabyte of data.
. I worked on the development of a data quality framework using - Great Expectations. Creating wrapper on top of Great Expectations & integration with Grafana dashboard for visualisation .
. Worked on Cloud Cost Optimization by adopting auto scaling , code Improvements , Graviton Instances, AQE Spark.
. Performing POCs (Proof of Concept) around new and emerging technologies and optimization of existing technology stack of current data products include creation of productivity metrics ,implementation of Graviton instance in existing cluster for reduction of cost.

DatabricksAWSKafkaETLData LakeHouseData Quality+3

Protium

Data Engineer-Intern

Mar 2022 – Jun 2022 · 3 mos

Developed and maintained dynamic business reports to provide real-time insights and aid in data-driven decision-making.
Managed the creation and optimization of loan disbursement portfolios, improving efficiency and accuracy.
Implementation of data transformations into a Data Warehouse using DBT, streamlining data processing and analysis.
Played a key role in setting up Airflow for job scheduling, enhancing automation and workflow management.
Proficiently utilized a tech stack that includes Python, PySpark, AWS Athena, AWS EMR (Step Function), AWS S3, PostgreSQL, DBT (Data Build Tool), and MetaBase.
Collaborated with cross-functional teams to ensure data pipelines and reporting systems met business requirements.

PythonDBTAirflowAWSPostgreSQLData Engineering