Kiran Gali

Software Engineer

India7 yrs 1 mo experience
Most Likely To SwitchAI ML Practitioner

Key Highlights

  • Designed innovative data solutions at Microsoft.
  • Expert in real-time data processing and streaming systems.
  • Versatile software engineer with strong cloud expertise.
Stackforce AI infers this person is a Data Engineering expert with a focus on real-time processing and cloud-native solutions.

Contact

LinkedIn

Skills

Core Skills

Data EngineeringApache SparkStreaming SystemsReal-time ProcessingFull Stack Development

Other Skills

AIAPI DevelopmentApache FlinkApache KafkaAzureAzure Cosmos DBAzure Event HubBootloaderC++CassandraData QualityDelta LakeElastic SearchEvent hubFastAPI

About

I'm so deeply obsessed with technology that I don't fancy about it anymore and the excitement isn't really in the technology itself anymore for me. I started to believe it's in the meaningful things that we can build with it. I'm a versatile, self-taught software engineer with strong knowledge of algorithms, data structures, and solid fundamentals in software engineering principles. I’ve helped teams build data platforms across the entire spectrum , from systems running on a single VM that move data using basic Linux copy commands, to modern, cutting-edge streaming platforms deployed on Kubernetes that supports modern streaming engines like Flink and Spark. For customers who prefer lower operational overhead and are willing to invest more, I’ve also designed and implemented fully managed cloud-native solutions: on Azure, setting up Synapse and Databricks using ARM-based deployments; and on AWS, building serverless data platforms using Lambda and Glue provisioned via Terraform. I also have end-to-end experience building mobile and web applications, spanning backend services, database design, and system integration. And top of all these, I have a great common sense to understand people and their requirements to create 'ahaaaa' factor in the products I deliver. Over the years, I’ve worn multiple hats; DBA, Data Engineer, Backend Engineer, Web App Developer, and at times, Program Manager.

Experience

7 yrs 1 mo
Total Experience
3 yrs 6 mos
Average Tenure
4 yrs 2 mos
Current Experience

Microsoft

2 roles

Software Engineer || L62

Promoted

Nov 2023 – Present · 2 yrs 7 mos

  • ● Developing data models as a part of datawarehousing to cater for demand planning and several other supply chain metrics.
  • Innovations :
  • ● Designed and prototyped "Spark Anti Pattern analyzer" that extracts DAG of a spark job using custom event listeners identifies the anti patterns in the transformations.
  • ● Designed and prototyped a full stack app which enabled an engineer to quickly ingest new data into the data platform.
  • ● Designed a pipeline scheduler named "Symphony" to execute pipelines efficiently to save costs.
  • ● Writing a white paper "Lean Compute for Real-Time Data Engineering" which focuses on using compute optimized engines to process data pipelines without compromising on scale.
  • Access below link and add comments for providing feedback
  • https://docs.google.com/document/d/1CLWZ2KZWzRX3Ju0Z4VmL3e0NuzSInYx2fYnWeWlQlmg/edit?usp=sharing
Apache SparkData EngineeringKubernetesPythonSQL

Software Engineer || L61

Apr 2022 – Nov 2023 · 1 yr 7 mos

  • Building a streaming metric driven platform that helps supply chain management take agile responses on demand and usage of resources at a data center.
  • ● Engineered an event streaming solution using Pyspark on Synapse, processing 10 million+ events
  • per day from Azure Event Hub to Delta Lake, achieving a 80% increase in data processing
  • efficiency.
  • ● Been an SME for developing streaming systems (Replay scenarios, deduplication, SCD2, logging, DQ) are some of them.
  • ● Bootstrapped the team on performing data quality on streaming records & a data quality solution with AI infused into it.
  • ● Designed & developed the crucial template pyspark notebooks contributing to the overall setup of the platform.
  • ● Identified problems in the teammates spark jobs and optimized the spark jobs written by the team by mentoring the data engineers, enhancing their skills in Pyspark and cloud technologies.
  • ● Led the implementation of comprehensive logging and monitoring practices using Azure Log Analytics, significantly improving issue tracking and resolution speed, resulting in a 25% decrease in incident response time.
  • ●Designed and developed APIs which retrieves a million records as a part of self serve initiative (this is my favourite because I integrated the API written in javascript with a pyspark job)
  • ● few issues that I found and really enjoyed contributing to fix them :
  • https://github.com/ZJONSSON/parquetjs/pull/89/commits/a01a17b3e8e4ce8c4f0b1037cdd533da3d0520b0
  • https://github.com/Azure/azure-event-hubs-spark/issues/644
  • https://github.com/Azure/azure-event-hubs-spark/issues/648
  • https://github.com/delta-io/delta/issues/1330
  • https://github.com/Azure-Samples/Synapse/issues/155
PysparkAzureData QualityAPI DevelopmentData EngineeringStreaming Systems

Smartdrive systems

2 roles

Data Engineer

Promoted

Apr 2019 – Mar 2022 · 2 yrs 11 mos

  • > Written Apache Flink jobs for real time data processing and scaled the jobs to process 45K events/ second with a bunch of lookups for every event that was processed.
  • > Worked on optimizing the ETL lag by replacing traditional ETL pipelines with Spark jobs running on k8 pods as a part of modernization. Written many ad hoc spark jobs to copy data for various purposes.
  • > Got hands-on experience working on Elastic search. I’ve also worked on creating and maintaining Data pipelines including creating monitoring dashboards using Grafana.
  • > Tuned SQL queries for performance improvement. My favorite one is the query which was blocking itself and finally errors out due to deadlock.
  • Tech stack: Spark, S3, Redis, Cassandra, Elastic search, SCALA, JAVA, SQL, Apache-Flink, Kafka, Redis, Cassandra, SCALA, JAVA.
  • Favorite books:
  • 1. Clean Code: learned to write maintainable modularized code.
  • 2. Distributed systems by Sandy Ryza.
  • 3. Designing data-intensive applications by Martin Kelppmann.
Apache FlinkSparkElastic SearchSQLData EngineeringReal-Time Processing

Data Engineer Consultant

Jan 2019 – Apr 2019 · 3 mos

  • I worked as a freelancer here, mostly involved in writing distributed systems coding using apache-spark to optimize the ETLs and maintaining the existing ETL pipelines.
  • Learned DataWareHousing and Dimension modeling by closely observing the enterprise-level OLTP systems and data warehouses.
  • Tech stack: SQL, Python, Pyspark, Kubernetes, AWS.
Apache SparkSQLPythonData Engineering

Freelancer.com

Freelance Data and Full Stack dev

Jan 2015 – Apr 2019 · 4 yrs 3 mos · Planet Earth · Remote

  • Proven work experience as freelance data engineer, full stack engineer to several projects that I have bid online over freelancer.com, upwork.com.
  • It was an amazing experience during college days. Shipping the code to different people across the world.
  • Met so many wonderful clients from California, Australia and Japan.
  • Skills I learned : Python , Java, Spark, JavaScript, Microservice architecture.
PythonJavaSparkJavaScriptFull Stack Development

Education

IIIT

Bachelor of Engineering - BE — Computer Science

Jan 2015 – Jan 2019

Kiran Gali - Software Engineer | Stackforce