Context.dev

One API to scrape, enrich, and extract the internet

API

Artificial Intelligence

Data

Visit WebsiteSee on Product Hunt

@yahia_bakour3 another question for you... I'm thinking out loud here.

What happens when/if websites start blocking crawlers completely? Is that even possible?

Secondly, wouldn't it be interesting to have a platform that monetises crawlers? A small fee is paid to the website owner every time a crawler uses the website or scans any information?

Curious to hear your thoughts?

Congrats on the launch! Nice to see more infra APIs tackling the "agents need current context, not frozen training data" problem. We're solving a related version of that for documents. Curious what pushed you toward markdown as the default output format vs. structured JSON?

The agent-native angle is interesting. The part I would want very visible in the docs is the failure contract: what comes back when a page blocks rendering, the sitemap is stale, extraction is partial, or a screenshot and markdown disagree?

For agents, a clean markdown string can look more complete than it really is. I would rather get a boring status like blocked / partial / stale-source, plus the URL and capture metadata, than have the caller guess whether the context is trustworthy.

web data + enrichment in one api is the real bottlneck for agents rn 👏 well deserved #1

Sounds great. How do you handle onclick events, if data is hidden and you have to click on an element to actually see it?

@yahia_bakour3 Handling scrapers, proxies, and sitemap parsing over and over is definitely a major headache when building AI products. The agent-native aspect—where a coding agent can literally sign up and wire the API itself—sounds incredibly powerful and futuristic. How does Context.dev handle websites that have heavy anti-bot protections or complex CAPTCHAs when an agent triggers a crawl?

Really interesting approach Yahia. I'm curious—what was the hardest part to get right while building a single API that handles scraping, extraction, and enrichment reliably at scale?

The "turn any page into LLM-ready Markdown" part is the piece I keep wishing existed — I do a lot of research-heavy work and getting clean, structured text out of messy pages is always the bottleneck before anything downstream is useful. Two genuine questions: how does it hold up on JS-heavy or auth-gated pages that don't render server-side, and is scraped content cached/versioned so I can tell whether I'm reading a fresh pull or a stale one? Typed SDKs across TS/Python/Ruby is a smart touch.

the markdown output came back clean enough that i barely had to clean it up before feeding it into my agent. brand extraction on a few random sites was surprisingly on point too, definitely beats maintaining my own scrapers.

Been using this to replace a scraper that kept breaking. One API call, clean markdown back. The brand data extraction (logos, colors, fonts) is surprisingly useful for onboarding flows. Handles JS-heavy sites better than I expected. 5000+ customers is a decent trust signal. Some niche sites still struggle, but overall solid.

The tricky part on that hash is that 'material' is consumer-specific: a price flip matters to a catalog agent, a nav reshuffle doesn't, but an agent watching layout wants the reverse. If you can surface the structured diff and let the caller pick which spans count, with your materiality hash as the sensible default, you sidestep everyone fighting one baked-in definition of meaningful. Glad it's already on the roadmap.

@yahia_bakour3 Congrats on the launch! We’ve been using your APIs for almost half a year now and they’ve been really reliable, fast, and delivering great results. Quality and speed are what matter most to us, and you’ve nailed both. Thank you, Yahia, for everything!

I can already think of a ton of use case for this. Congrats on the launch Yahia!!

Have been a user for 9 months haven't had any problems.

Would recommend

How does it actually handle sites that load content dynamically with JavaScript, does it run a real browser under the hood or just hit the raw HTML and miss half the page?

Didn’t know I’d love scraping websites, extracting style guides, and pulling font data until I tried context.dev.

There’s a surprising amount of knowledge to gain from doing things like this.