Bharat K.

Senior Software Engineer

Bengaluru, Karnataka, India7 yrs 4 mos experience

Key Highlights

7+ years of experience in data engineering.
Expertise in AWS and GCP cloud platforms.
Proven track record in optimizing data pipelines.

Stackforce AI infers this person is a Data Engineer specializing in cloud-based big data solutions.

Contact

Skills

Core Skills

Data EngineeringGoogle Cloud Platform (gcp)Machine LearningAmazon Web Services (aws)Cloud Engineering

Other Skills

AWSApache AirflowApache KafkaApache SparkBashBig DataCData LineageDatabricksDataformDeep LearningExtract, Transform, Load (ETL)GitHadoopKubernetes

About

Bharat is a Data Engineer with 7+ years of experience in cloud platforms (AWS, GCP) and big data technologies (Spark, Airflow, Presto, Kafka, Databricks, Kubernetes). He has a proven track record of optimizing data pipelines, leading large-scale migrations, and driving cost efficiencies across data platforms. He is experienced in solving complex, data-driven challenges with expertise in cloud infrastructure, big data tools, and applied machine learning. He has experience with programming languages like Python, Scala and SQL to deliver high quality solutions on time. He also have hands-on experience with multiple cloud services like EMR, Athena, Glue, Sagemaker, EC2 on AWS and services like BigQuery, DataProc, Pub/Sub, Cloud compose etc on GCP. He has a Masters degree from Georgia Tech, Atlanta, with specialization in Applied Machine Learning. Check out more about him at https://bha159.github.io

Experience

H1

Senior Software Engineer (Data & ML)

Oct 2024 – Present · 1 yr 5 mos · India · Remote

Easyship

Senior Data Engineer

Apr 2023 – Oct 2024 · 1 yr 6 mos · India · Remote

Collaborated to deliver accurate, timely data for powering dashboards and supporting downstream teams, ensuring data availability within defined SLAs. Some of my key achievements are:
● Led the migration of select data pipelines from BigQuery to Spark, cutting costs by 30% to 67% for targeted workloads. Automated the Spark job creation process within Airflow using config-driven templates.
● Implemented watermarking across daily pipelines to enable incremental updates, leading to significant cost savings in data retrieval processes.
● Initiated and led the migration from BigQuery to Dataform (DBT equivalent in GCP), building the codebase from scratch to eliminate redundancy, enhance data lineage, and improve execution scheduling.
● Created and deployed a machine learning pipeline for dangerous goods detection, contributing to operational efficiency and safety.

PythonApache AirflowExtract, Transform, Load (ETL)Data LineageApache SparkDataform+3

Meesho

SDE 2

Nov 2021 – Mar 2023 · 1 yr 4 mos · India · Remote

Worked with data team to maintain and develop data platform. Some of my achievements include:
● Managed two large-scale Airflow migrations: a major version upgrade and a shift from EC2 to Kubernetes, ensuring 99.9% uptime for over 2,500 DAGs. Collaborated with the performance team to stress-test scalability and optimize configurations.
● Built internal and external observability solutions by introducing failure alerts and enhancing logging for Airflow DAGs and Spark jobs. Integrated these systems with Grafana and Prometheus for comprehensive stats collection and monitoring.
● Migrated all Spark jobs from AWS EMR to Databricks while optimizing some spark jobs, leading to improved performance and cost reductions.
● Led the migration of Presto clusters and Metabase instances to EKS with spot instances, achieving a 30% cost reduction for Presto clusters and over 42% for Metabase instances.

PythonDatabricksAmazon Web Services (AWS)Apache AirflowExtract, Transform, Load (ETL)Hadoop+6

Amazon

3 roles

Big Data Engineer

Aug 2020 – Nov 2021 · 1 yr 3 mos · Bengaluru, Karnataka, India

Worked with data team to maintain and develop data platform. Some of my achievements include:
● Migrated data pipelines from an internal scheduling tool to Airflow, improving scalability and reducing technical debt. Achieved a 95%+ uptime, compared to ~53% uptime with the previous internal tool.
● Automated and optimized ETL jobs using AWS services such as EMR and Glue, enhancing data wrangling efficiency and processing speed.
● Designed and implemented event-driven architecture using AWS Lambda and MWAA (Managed Airflow) to streamline the execution of essential enrichment pipelines.
● Collaborated with architects to design and implement data lineage for critical data pipelines, improving traceability and governance.

PythonAmazon Web Services (AWS)Apache AirflowExtract, Transform, Load (ETL)HadoopBig Data+3

Cloud Engineer I (AWS)

Jun 2020 – Aug 2020 · 2 mos · Bengaluru, Karnataka, India

Worked in ‘Big Data’ team of Premium support at AWS. Helped many AWS customers with architectural and migration decision to AWS chiefly with big data services like EMR, Athena, Glue, DynamoDB etc. Educated many AWS customers on how to utilize AWS services by taking training sessions.

LinuxPythonAmazon Web Services (AWS)HadoopBig DataApache Spark+2

Cloud Associate (AWS)

Mar 2019 – Jun 2020 · 1 yr 3 mos · Bengaluru, Karnataka, India

PythonAmazon Web Services (AWS)HadoopBig DataApache SparkSQL+1

Cognizant

Associate - Projects

Aug 2018 – Mar 2019 · 7 mos · Bengaluru, Karnataka, India

PythonHadoopBig DataData Engineering

Green dot india ltd.

Research Intern

May 2017 – Jul 2017 · 2 mos · Sahibabad, Ghaziabad, Uttar Pradesh, India

Bharat sanchar nigam limited

Trainee

Jun 2016 – Jul 2016 · 1 mo · BSNL ALTTC,Ghaziabad, Uttar Pradesh 201002

A 4 Week training program in BSNL covering broader aspects of telecommunication and latest technologies like 2G, 3G, LTE and the basic idea of VLSI designing.