May 2024 – Jun 2025 · 1 yr 1 mo · Remote

• Focused on building expertise in Azure Databricks, pursuing training and certification in data engineering and cloud-based data solutions.
• Delivered an SME-led demo session on CDC Pipeline.
• Successfully implemented secure key management by leveraging Databricks Secrets Scope, aligning with enterprise security and governance standards within Databricks.
• Building robust ETL pipelines and optimizing data workflows with [ Databricks | Pyspark | Spark SQL].
• Migrated and optimized SQL queries from Redshift to Databricks SQL, supporting dynamic parameters and repeated execution at scale (4,000+ variations for a single query).
• Ingested 2 TB of data from AWS S3 into Databricks using Auto Loader performed end-to-end validation including schema, row count, and null checks.
• Automated large-scale query execution with Databricks Workflows and REST APIs to simulate real-user
• workloads and benchmark performance.
• Integrated real-time Kafka streams from a MySQL-Debezium pipeline directly into the Databricks Bronze layer implemented deduplication using latest timestamps.
• Improved query performance by 2–3x using Delta Lake features like Liquid Clustering and Spark
• optimizations
• Designed an asynchronous web-scraping workflow using Selenium to extract unstructured text from 200,000+ websites, storing the data in Delta Lake for downstream ML-based article classification using Databricks MLflow.
• Developed a PDF processing pipeline to ingest 1,000+ downloaded news article PDFs, using Apache Tika to extract text content via OCR, which was then passed to pre-trained LLMs for classification and downstream analytics, with structured outputs stored in Delta Lake.
• Implemented end-to-end workflows, including data ingestion (web, PDF), unstructured text processing, and Delta Lake storage, reducing manual data processing efforts by 70%

Aug 2022 – Dec 2023 · 1 yr 4 mos · Jaipur, Rajasthan, India · On-site

• 🚀 Data Engineer | Celebal Technology PVT LTD
• With 8 months of INTERNSHIP + 9 months of training at Clebal Technology, I had the privilege of contributing to dynamic projects and honing my skills in the realm of data engineering. Throughout my tenure, I focused on developing robust solutions and gaining hands-on experience with cutting-edge technologies.
• 🌐 Key Contributions:
• Azure Data Factory (ADF) Pipeline Development: Spearheaded the design, implementation, and optimization of data pipelines using Azure Data Factory, ensuring seamless data integration and processing.
• SQL Expertise: Demonstrated proficiency in SQL, leveraging it for efficient database management, query optimization, and data retrieval.
• Pyspark Development: Applied Pyspark to enhance data processing capabilities, utilizing its powerful features for large-scale data transformations.
• Databricks Exploration: Worked extensively with Databricks, leveraging its collaborative environment to enhance productivity in big data analytics and processing.
• 💡 Proof of Concept (POC): Unity Catalog, SCD Types
• Unity Catalog POC: Successfully led a Proof of Concept for the implementation of Unity Catalog, showcasing its potential benefits and applications within our data ecosystem.
• SCD Types Implementation: Played a pivotal role in implementing Slowly Changing Dimension (SCD) types, ensuring accurate historical data representation and integrity in our data warehouse.
• 📈 Learning and Growth:
• During my 8-month internship and subsequent trainee phase, I actively engaged in continuous learning and skill development. This period allowed me to immerse myself in real-world projects, collaborate with experienced professionals, and deepen my understanding of the data engineering landscape