Launch HN: Context.dev (YC S26) – API to get structured data from any website
www.context.devHi Hacker News, I’m Yahia. I built Context.dev (https://www.context.dev/) to make it really easy to integrate web data into your products and agents.
Here’s a demo video: https://www.tella.tv/video/build-faster-with-context-dev-api...
Since it’s an API, here are the docs: https://docs.context.dev/quickstart.
You can send us a URL and get back clean Markdown, rendered HTML, screenshots, extracted images, etc.. You can also send us a domain and get company or brand context: name, description, logos, colors, fonts, social links, screenshots, style information, and related metadata. For more custom use cases, you can send a URL plus a JSON Schema and ask us to extract structured data from the site into that shape. For example, you might ask for pricing plans, product categories, office locations, support links, integration partners, or anything else that is visible on the public site.
The goal is to give developers the output they actually want. Raw HTML is rarely the useful thing; the useful thing is usually Markdown for a model, JSON for an application, a logo for a UI, or a structured company profile for an agent.
Before, I worked at Amazon and Sunrun, and co-founded StockAlarm.io & essense.io, both of which were acquired. Also, I built knifegeek.io, which scraped pocket knives from across the internet and listed them easily. The project is outdated now (coming back soon) but back then it hit the frontpage of hacker news and people seemed to like it: https://news.ycombinator.com/item?id=34604281.
Just before Context.dev, I built Brand.dev. The idea was that your software product should automatically know about your customer if they sign up with a corporate email. The API pulled brand data such as logos, backdrops, name, description, industry, and more from the public web and surfaced it to your product to integrate as part of their onboarding experience. That’s worth doing because conversion rates on onboarding improve dramatically when you go from “enter all this info” to “confirm all this info” (and there was never any privacy concern all the information is public).
That was a nifty niche, but the more customers used it, it became obvious that “brand data” was only one slice of a larger need. People started asking for things like screenshots, structured extraction, and LLM ready data. So I expanded to Context.dev, and applied to YC (got rejected after an interview), then kept going and re-applied at which point I got in as a solo founder.
People use Context.dev in more ways than I can list, but here are some: keeping context up to date on customer websites for chatbots - building beautiful brand assets/ads for customers - enrichment flows using agent harnesses like eve.dev - crawling customer websites into chatbot knowledge bases - turning GitHub repos into branded docs sites - academic journal and PDF crawling. There are a ton more examples at https://www.context.dev/customers.
We know that many crawlers are not behaving like good citizens on the web, and the entire space has a bad reputation as a result. At the same time, customers are not usually trying to buy “scraping”. They are trying to make a support bot work, personalize onboarding, enrich CRM records, generate docs, monitor leads, or let an agent research a company. There are lots of legit use cases. We want to satisfy those while being respectful of everyone involved.
We maintain a caching layer and avoid hammering websites. Customers can configure the cache, but if we find we’re sending too many requests to a url in a certain amount of time, we step in and tone it down. Websites can opt out of our service, and we respect these requests and add them to our block list.
We focus on customers who want to build cool things for their users. Enriching onboarding is a popular use case. So is integrating context about their own websites (things like support bots), and building agents that can automatically reason about complex tasks involving the internet.
We only allow customers to use brand data to identify a specific customer on their software, you cannot use it in your own materials or to imply endorsement.
I'd love to hear your feedback about the product in the comments, thanks!
If you want to vibe something that gets you 70% of the way to this well funded startup in like 15 minutes just tell your LLM of choice to create a hook or override the web fetching skill to pipe the content through Mozilla’s readability extension that strips the DOM elements out deterministically, leaving only the content. You can then parse it however you want. Can be done entirely client side, in runtime, with a few JavaScript libraries
"the first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time."
now for my actual thoughts, if that works for anyone in production reliably, then they should at-least try to do that.
many of my customers moved from in-house solutions because the engineering time spent on maintaining something like this alone makes it quite the undertaking, the infra needed to support something like a few hundred TPS w a browser fleet would cost more than a sub to us in the majority of cases
Conversely, many of my customers abandoned many of their SaaS solutions because they can now hand roll the 10% of the SaaS solution they need in 15 minutes.
That’s not to say all SaaS is going away. But a good hunk of the single simple solution products are going to suffer
touché
the remaining 10% accounts for the other 10,000% of the development time
i see you too are an experienced dev
Readability is far from ideal. check this out: https://github.com/promptware/readweb
It's a custom pipeline I made to address its shortcomings
Hi! Congrats on the launch, I gained LOTS of great insights from your comments, particualrly the bits about diffrentiation in a crowded market.
I am building something in agentic automation space, currently it's still under development, but would love to know your journey of ideating -> building -> getting the first customer -> iterating -> and presumably getting into YC.
Am still relatively new to this (19 lol) so I got a long way, but would really appreciate any insights :)
Cheers! Wish you luck with Context.
i've yapped quite a bit in this thread already about my journey, but happy to provide any (hopefully useful) insights, you can DM me on X, mention this comment please: https://x.com/mynameisyahia
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
hi, no worries at all, it came off perfectly fine
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
What major edge cases have you found so far for the browser scraping method, its really complex like dealing with Auth, or pages that use pop ups, content blockers . etc...
if it needs auth, we don't do any type of logins at the moment, still figuring out how to do it without attracting the wrong type of customer (someone doing shady stuff)
popups / cookie banners are quite easy to get through with enough heuristics for 99%+ of cases
content-blockers are the same thing as the auth scenario, hard to figure out where to draw the line between building something people want and doing something shady
Is "residential proxy" a euphemism for botnets or is there a difference? Genuinely curious.
- worst is "extension" based traffic routing where your provider is essentially tricking end users into routing traffic through - opt-in p2p networks can be described as botnets sometimes, i try to avoid these but not as bad - good enough dISP proxies can be classified as residential quality, work quite well, although people have argued w me on their quality - managed business/enterprise residential gateways are ideal, because businesses opt in and allow the traffic - few other options
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales? > Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at hello@context.dev or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
How did you find your differentiation in a highly commoditized space? It's probably one of the most crowded spaces.
Even within YC, there are many competitors that do pretty much the same thing:
- Firecrawl
- BrowserUse
- Browserbase
- CloudCruise
- NotteLabs
- Intuned
- Expand.ai
- Reworkd
And then you have the extremely well-funded web retrieval players like Parallel and Exa.
How do you differentiate to all these?
Another thing that might interest HN: AI crawlers come with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported here on HN (and experienced myself).
Does Context respect robots.txt directives and do you disclose the identity of your crawlers via user-agent header?
the reality is that the space is huge and expanding
many that you mentioned serve different needs and it tends to not be zero sum to everyone's surprise (including mine at first)
e.g.
browseruse, browserbase = browser automation exa, parallel = search for ai (primarily)
we differentiate by prioritizing the infrastructure use-case, we encourage people to build products & features on top of our APIs and build it accordingly
some direct differentiators are: no hidden credit multipliers, world class brand data, and an api that just works
we're trying to be good samaritans on the web, and cache data whenever possible so we can avoid overloading a website, it's the main reason we don't offer "login with ai" features even though it's trivial at this point, it's quite hard to figure out how to avoid shady behavior while building something people want, i don't have a good answer yet and neither do many others
lastly, to reiterate, lots of our customers migrate from alternatives, but lots also just use multiple providers
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful.
> Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.
> Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.
[0] https://github.com/adam-s/intercept
Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally
You are smart!
Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.
> We maintain a caching layer and avoid hammering websites.
This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.
100% yes, i found out about that company recently too, it's a tempting idea for sure
Why not much traction, was the benefit more to websites than businesses/scrapers?
it counts as "reverse engineering" a website which is much worse for the website owner imo, then again im biased
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
oh wow, happy to see you here, we didn't post this anywhere so its nice to see a customer find us so quickly, thank you for the kind words.
brand data is a shockingly hard problem to get right
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier - we have world class brand data - we're focused primarily on the infra use-case, rather than gtm & everything else - anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
Interesting - I'll be sure to benchmark it at some point. We've found the best results come from blending providers depending on the task anyways.
Thanks for the quick response - and always happy to see more competition in the space. Best of luck with future features!
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
thank you!
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
This seems like a responsibly designed service, but I find it a little odd and baffling that we need such intermediaries for the average hobbyist / small project to reliably access sets of content published on the internet.
Wasn’t this the point of the web?
i think about this too, perhaps you're right, if (big if) the web was designed with the end goal in place, however it was a series of ~interesting~ mistakes & decisions that led to where we are today
the same could be said of most cybersecurity companies too
Does this respect llms.txt and robots.txt, or have you found it more effective if agents see what humans see?
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
Hey Yahia, I recently saw your startup on the YC registry and found it to be one of the most promising of the batch, so good luck with this! Do you mind if I ask how you managed to get your first users on Brand.dev?
hey, happy you like it, to be honest we had no real secret, it takes a ton of time and effort to earn developer trust
i got my first customer after 1 month, and then just worked like hell to make him happy even though it was a very large company paying me $99/mo at the time, they're still a customer too
since then, i just locked in on every customer until they were thrilled, and once we hit 50, word of mouth started to kick in, now we're at 300+ and get a referral every other day which is awesome
i think our community appreciates effort, and when someone goes the extra mile to make sure your problems are solved, you tend to remember that. it doesn't scale super well, but you'd be surprised how far it'll take you
Excellent way to run a business.
Really cool idea! I had something similar a while back. Maybe figure out a way to use MCP so the client can connect directly out of the box
ty, it's here! https://docs.context.dev/install-mcp
Awesome! Been great watching this product improve so quickly, can't wait for what's next :)
thank you, i try to ship updates often: https://docs.context.dev/changelog
Remember when the Semantic Web attempted to make the entire web machine-readable?
https://en.wikipedia.org/wiki/Semantic_Web
you just found my end goal :)
love the design... congrats
will let my designer know, it's a dev-focused product, and we're all so finely tuned to avoid slop, so design & feel was really important
new frontier models do this already
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
So basically web scraping as a service with an API on top?
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
Maybe AI-service-blocklist-as-a-service could be a YC company.
Better business model would be some sort of micropayment setup - allow humans, but make scrapers pay a hundredth of a cent for access.
Is this similar? https://blog.cloudflare.com/monetization-gateway/
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies?
AKA unusable for high value data.
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
lessgooo! context dev ftw!
thank you, im not sure who this is though, but please email me if i can help in any way
Nice, @grok how does it compare to Cloudflare that also provides a REST endpoint for structured markdown data and screenshots?
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead