Aug 2025 – Present · 9 mos

• In my current role, I focus on designing, maintaining, and optimizing large-scale data extraction pipelines using Python and Ruby. My work involves building and scaling crawling infrastructure, ensuring reliability, and integrating with product requirements to deliver clean and accurate data.
• Key Responsibilities:
• Develop and maintain multiple web scraping and data pipelines leveraging Selenium, Playwright, and custom crawling workers.
• Manage and optimize infrastructure components such as Elasticsearch, RabbitMQ, Resque, and Redmine to ensure seamless data flow and system reliability.
• Collaborate with proxy vendors, evaluate compatibility, and enhance proxy integration within the platform.
• Partner with product teams to customize pipelines for client-specific needs and assess feasibility of new data sources.
• Implement and refine anti-bot and anti-CAPTCHA solutions, enabling robust data collection at scale.
• Continuously improve system performance, resource utilization, and fault tolerance across the platform.
• This role has allowed me to combine hands-on engineering with problem-solving in a production environment, ensuring our data infrastructure remains resilient, scalable, and adaptable to evolving client and market needs.

May 2023 – Aug 2025 · 2 yrs 3 mos

• As a Platform Engineer I, I contributed to the development and enhancement of scalable web scraping and data extraction systems. My focus was on building reliable pipelines, ensuring system stability, and supporting cross-functional requirements to deliver accurate, structured data to clients.
• Key Responsibilities:
• Built and integrated new websites into existing scraping infrastructure using Python, Ruby, and automation frameworks.
• Developed efficient web crawlers and pipelines leveraging tools like Selenium, Scrapy, and BeautifulSoup.
• Worked with proxies, session management, and CAPTCHA-solving techniques to ensure uninterrupted data collection.
• Participated in coding, unit testing, and code reviews to maintain high-quality standards.
• Assisted in monitoring pipelines, resolving issues, and optimizing performance for large-scale data ingestion.
• Collaborated with product and operations teams to align scraping strategies with evolving client requirements.
• This role gave me a strong foundation in data engineering, web scraping, and system optimization, while sharpening my ability to solve real-world challenges in a production environment.

Jul 2022 – Oct 2023 · 1 yr 3 mos