Vamsi Krishna — Product Manager

Data Engineering Manager / Data Architect with over 13 years of experience leads a team in developing and refining data pipelines using AWS EMR, Apache Spark and Databricks. This role emphasizes scalable solutions, enhanced performance, and cost efficiency within the Bigdata ecosystem, focusing on solving complex business problems and creating value across domains. Built Unified Ingestion Platform with both Streaming and Batch accommodating 300+ pipelines ingesting 10k tables processing 10B events reconciling 50TB daily. Demonstrated expertise in AWS Big Data services like EMR, S3, Athena, IAM, KMS, and Glue, optimizing workflows to handle over 15 petabytes annually, increasing efficiency and reducing costs by 25%. Incorporated Spark Structured Streaming on Azure Data Bricks to extract data from EventHub topics, pre-process, and store in a Delta Lake, increasing data throughput by 50% and reducing latency in data availability. AWS Certified Cloud Practitioner: "Certified in AWS Cloud fundamentals, equipped to leverage cloud technologies for scalable solutions." Astronomer Certification for Airflow: "Expert in Apache Airflow, skilled in designing and optimizing data workflows for efficiency." Databricks Certified Associate Developer for Apache Spark 3.0: "Proficient in Apache Spark 3.0 and skilled at building high performance data processing applications." Skills Set: AWS Cloud: AWS EMR, AWS Glue, AWS Athena, AWS S3, AWS IAM, AWS KMS, AWS Redshift Azure Cloud: Azure Databricks (ADB), Azure Data Factory (ADF), Azure Data Lake Gen2 (ADLS Gen2), Azure Key Vault (AKV), Azure Event Hub Data Engineering Tools: Spark, Scala, Python, SQL, Pyspark, Spark Structured Streaming, Hive, Kafka Devops Tools: BitBucket, Jenkins, Bamboo, GitHub SDLC: Agile, Scrum, Project management, Sprint planning IDE: PyCharm, IntelliJ, Eclipse Domains: Telecom, E-Commerce, Banking, Electrical & Electronics Email / Contact me at: vamsi.krishna107601@gmail.com / +91-7259 623 401

Stackforce AI infers this person is a Data Engineering Manager with expertise in AWS and Azure cloud solutions.

Location: Hyderabad, Telangana, India

Experience: 14 yrs 3 mos

Skills

Aws Cloud
Data Engineering
Azure Cloud

Career Highlights

Built a Unified Ingestion Platform processing 10B events daily.
Optimized AWS workflows, reducing costs by 25%.
Developed real-time analytics frameworks enhancing operational insights.

Work Experience

Paytm

Data Engineering Manager / Data Architect (3 yrs 6 mos)

Honeywell

Senior Technical Lead - Data Engineering (1 yr 4 mos)

Ericsson

Senior Data Engineer (4 yrs 8 mos)

Altran

Big Data Engineer (1 yr 6 mos)

Wipro Technologies

Big Data Engineer (3 yrs 3 mos)

Education

B.Tech at The Sri Venkateswara University College of Engineering (SVUCE), Tirupati

Vamsi Krishna

Product Manager

Hyderabad, Telangana, India14 yrs 3 mos experience

Most Likely To SwitchHighly Stable

Key Highlights

Built a Unified Ingestion Platform processing 10B events daily.
Optimized AWS workflows, reducing costs by 25%.
Developed real-time analytics frameworks enhancing operational insights.

Stackforce AI infers this person is a Data Engineering Manager with expertise in AWS and Azure cloud solutions.

Contact

Skills

Core Skills

Aws CloudData EngineeringAzure Cloud

Other Skills

AWS AthenaAWS EMRAWS GlueAWS IAMAWS KMSAWS LambdaAWS RedshiftAWS S3Amazon EC2Amazon Elastic MapReduce (EMR)Amazon RedshiftApache Spark StreamingAzure Data FactoryAzure Data LakeAzure Data Lake Gen2

About

Experience

Paytm

Data Engineering Manager / Data Architect

Sep 2022 – Present · 3 yrs 6 mos · Hyderabad, Telangana, India · Remote

Built Unified Ingestion Platform with both Streaming and Batch accommodating 200+ pipelines ingesting 20k tables processing 5B events reconciling 70TB daily.
Designed and implemented a scalable, fault-tolerant AWS data system handling petabytes, boosting retrieval speeds by 50% and enhancing reliability at peak loads.
Implemented airflow DAGs to submit PySpark and Spark Scala jobs onto AWS EMR cluster.
Built and managed scalable AWS big data solutions, growing from hundreds of terabytes to petabytes and tripling annual data volume capacity.
Developed scalable ETL pipelines on AWS EMR, processing multi-terabyte datasets daily into AWS S3, resulting in a 40% faster data processing time and enabling real-time analytics.
Demonstrated expertise in AWS Big Data services like EMR, S3, Athena, IAM, KMS, and Glue, optimizing workflows to manage over 15 petabytes annually, increasing efficiency and reducing costs by 25%.
AWS Cloud: AWS EMR, AWS Glue, AWS Athena, AWS S3, AWS IAM, AWS KMS, AWS Redshift
Data Engineering Tools: Spark, Scala, Python, SQL, Pyspark, Spark Structured Streaming, , Kafka
Devops Tools: BitBucket, Jenkins,

AWS EMRAWS GlueAWS AthenaAWS S3AWS IAMAWS KMS+10

Honeywell

Senior Technical Lead - Data Engineering

Apr 2021 – Aug 2022 · 1 yr 4 mos · Greater Bengaluru Area · Hybrid

Spearheaded the design and development of the Data Enrichment Engine utilizing the Medallion architecture pattern, enhancing data integration and analytics capabilities across business units at Honeywell, processing up to 5 terabytes of data daily.
Developed a Real-Time Streaming Analytics framework that processes over 2 million IoT sensor data points per hour, enhancing operational insights by 30% through real-time analytics like filtering and transforming.
Incorporated Spark Structured Streaming on Azure Data Bricks to extract data from Kafka/EventHub topics, pre-process, and store in a Delta Lake, increasing data throughput by 50% and reducing latency in data availability.
Deployed Prometheus and Grafana for robust monitoring and alerting, improving system uptime by 99.9% and reducing incident response time by 40%.
Defined a high-capacity data processing framework for IoT devices, handling millions of data points daily and storing in Time Series Databases and ADLS Gen2, enhancing retrieval speed and storage efficiency by 25%.
Azure Cloud: Azure Databricks (ADB), Azure Data Factory (ADF), Azure Data Lake Gen2 (ADLS Gen2), Azure Key Vault (AKV), Azure Event Hub
Data Engineering Tools: Spark, Scala, Python, SQL, Pyspark, Spark Structured Streaming, Kafka
Devops Tools: BitBucket, Bamboo,

Azure DatabricksAzure Data FactoryAzure Data Lake Gen2Azure Key VaultAzure Event HubSpark+8

Ericsson

Senior Data Engineer

Aug 2016 – Apr 2021 · 4 yrs 8 mos · Bengaluru, Karnataka, India · Hybrid

Engineered expansive big data infrastructures on AWS, accommodating data expansion from hundred terabytes to over 2 petabytes, which doubled the system's capacity to manage an annual data growth of 200%.
Directed high-efficiency ETL pipelines with AWS EMR and PySpark, processing terabytes of data daily into AWS S3, cutting transformation time by 45% and facilitating real-time analytics.
Combined Kafka and Spark Streaming to develop a real-time data processing system. This integration handled streaming data, boosting throughput by more than 50% and cutting down latency in analytics reporting by 30%
AWS Cloud: AWS EMR, AWS Glue, AWS Athena, AWS S3, AWS IAM, AWS KMS, AWS Redshift
Data Engineering Tools: Spark, Scala, Python, SQL, Pyspark, Spark Structured Streaming, , Kafka
Devops Tools: BitBucket, Jenkins,

AWS EMRAWS GlueAWS AthenaAWS S3AWS IAMAWS KMS+10

Altran

Big Data Engineer

Oct 2014 – Apr 2016 · 1 yr 6 mos · Bangalore Urban, Karnataka, India · On-site

Defined Hive tables using HiveQL, handling and analyzing more than 5 TB of data across these tables to facilitate data driven decision making in organizational projects.
Demonstrated solid experience with Big Data services including Spark, HDFS and Hive, optimizing data processing workflows that handled over 25 petabytes of data annually, which increased operational efficiency and reduced costs by 15%.
Data Engineering Tools: Spark, Scala, Python, SQL, Pyspark, Hive
Devops Tools: BitBucket, Jenkins

SparkScalaPythonSQLPysparkHive+1

Wipro technologies

Big Data Engineer

Jun 2011 – Sep 2014 · 3 yrs 3 mos · Bangalore Urban, Karnataka, India · On-site

Pioneered excellence in managing a Hive data warehouse by creating and maintaining over 50 tables, optimizing data distribution through partitioning and bucketing techniques, and enhancing query performance by 30% through HiveQL optimizations.
Migrated 20+ MapReduce programs to Spark transformations using Scala, resulting in a 40% reduction in processing time and improving job performance and scalability within the data processing workflows.
Data Engineering Tools: MapReduce, Java, Sqoop, Hive, SQL
Devops Tools: BitBucket, Jenkins,

MapReduceJavaSqoopHiveSQLData Engineering