Crawlbot monitors Share of Voice, product content quality, and pricing across 35 online retailers in the UK, South Africa, Poland, and France. Every hour, it scrapes category pages to track which brands appear, in which positions, and whether those positions are organic or sponsored. Every night, it audits 6,000+ product pages for missing images, specs, A+ content, and videos. Built on a distributed fleet of 10 workers with anti-bot bypass for Cloudflare, Amazon, and retailer-specific defenses.
Hey everyone! I built Crawlbot because I spent years working in CE brand management and kept running into the same problem: you have products listed on 15+ retailers, and you have zero visibility into what shoppers actually see.
Is your product on page 1 at Currys? Is a competitor buying the top sponsored slot at Amazon? Did Argos drop your hero image? You only find out when sales dip, and by then it's too late.
I started with a single Python script scraping one retailer. That turned into Playwright workers, then a distributed queue system across 5 machines, then a full monitoring platform with hourly SoV tracking, nightly content audits, and price change alerts.
The hardest part was reliability. Retailers actively fight scrapers, so I had to build multi-layered anti-bot bypass, automatic retries, memory management, and a watchdog system that alerts me on Discord if anything goes down.
Today Crawlbot tracks 35 retailers across 4 countries with near-zero downtime.
Would love your feedback. Happy to answer any questions about the tech stack or the e-commerce intelligence space!
No comment highlights available yet. Please check back later!
About Crawlbot.pl on Product Hunt
“Real-time digital shelf analytics for CE brands”
Crawlbot.pl was submitted on Product Hunt and earned 3 upvotes and 1 comments, placing #183 on the daily leaderboard. Crawlbot monitors Share of Voice, product content quality, and pricing across 35 online retailers in the UK, South Africa, Poland, and France. Every hour, it scrapes category pages to track which brands appear, in which positions, and whether those positions are organic or sponsored. Every night, it audits 6,000+ product pages for missing images, specs, A+ content, and videos. Built on a distributed fleet of 10 workers with anti-bot bypass for Cloudflare, Amazon, and retailer-specific defenses.
Crawlbot.pl was featured in Analytics (171.4k followers), SaaS (41.5k followers) and E-Commerce (41.4k followers) on Product Hunt. Together, these topics include over 69.1k products, making this a competitive space to launch in.
Who hunted Crawlbot.pl?
Crawlbot.pl was hunted by Jan Trzupek. A “hunter” on Product Hunt is the community member who submits a product to the platform — uploading the images, the link, and tagging the makers behind it. Hunters typically write the first comment explaining why a product is worth attention, and their followers are notified the moment they post. Around 79% of featured launches on Product Hunt are self-hunted by their makers, but a well-known hunter still acts as a signal of quality to the rest of the community. See the full all-time top hunters leaderboard to discover who is shaping the Product Hunt ecosystem.
Want to see how Crawlbot.pl stacked up against nearby launches in real time? Check out the live launch dashboard for upvote speed charts, proximity comparisons, and more analytics.
Jan Trzupek