We launched UnWeb in March — an API that converts messy web pages into clean, LLM-ready Markdown. We had a clear hypothesis: developers building AI pipelines are wasting context window budget on HTML noise, and they’ll pay to fix that.

Ten weeks and a few hundred API keys later, here’s what we actually learned.

The Use Cases We Didn’t Build For

We designed UnWeb for RAG pipelines — the classic “fetch a URL, get clean content, embed it, done” workflow. That’s still the most common use case. But the production usage patterns are noisier than the theory.

The second-largest segment: competitive intelligence. Developers building sales tools that track competitor websites, pricing pages, and product announcements. They don’t need embedding-ready chunks; they need parseable text that feeds into a summarization pipeline. Our output format is perfect for that, even though we never explicitly optimized for it.

Third: browser extensions. Developers building read-later tools and note-taking extensions that need clean reading content. They were using us as the content extraction layer because the output quality was higher than readability-style parsing. We hadn’t considered this at all.

The lesson: when you build a good general-purpose primitive, people will find applications you didn’t anticipate. Design your API for composability, not just your current use cases.

The Edge Cases That Mattered

Our first three months in production surfaced edge cases that test coverage couldn’t catch. A few that were worth fixing quickly:

JavaScript-rendered paywalls. Sites that show teaser content to crawlers but require login for full access. Our original behavior was to return the partial content silently. This is wrong — callers need to know they’re getting a truncated page. We added a paywall_detected flag in the response metadata.

Encoding variance. Sites in languages using extended character sets would sometimes return content with mixed encodings. The resulting Markdown had mojibake in headings. We hardened the encoding detection pipeline in the second week.

Table-heavy documentation. API reference docs with large parameter tables. Our table rendering was technically correct but produced Markdown tables that chunked badly in RAG pipelines — row boundaries weren’t always preserved. We added a structured-table mode that outputs tables as repeated key-value blocks when column count exceeds a threshold.

None of these were in our launch test suite. All of them came from production traffic within 30 days.

What “Good Enough” Actually Means for LLM Output

When we built the conversion pipeline, we aimed for “correct” Markdown. In practice, correct isn’t the right target for LLM use cases.

The useful target is: does the model extract accurate information from this? That’s a different question. A technically correct Markdown table that a model misreads due to column alignment is worse than a less-standard format the model reliably parses.

We’ve shifted our quality assessment from “does this validate?” to “does a model return the right answer when we ask questions about this content?” It’s more expensive to measure but it’s the thing that actually matters to our users.

This reorientation has changed how we think about output quality in general. For any tool that feeds into an AI pipeline, human-readable correctness and model-useful correctness can diverge. The only way to know which is which is to actually test with the model in the loop.

What’s Next: Hacker News Tomorrow

We’re submitting UnWeb to Show HN tomorrow. If you’ve been using the API and found it useful, the thread is a good place to share what you’re building.

If you’ve hit edge cases we haven’t mentioned here — sites that don’t convert cleanly, output formats that don’t work for your pipeline — the comments section is a great place to surface them. Production edge cases are exactly the kind of feedback that’s hard to get any other way.

The free tier covers evaluation-scale usage with instant API keys and no credit card required. If you’re building anything that touches web content for AI, give it a try.

UnWeb is part of the MBSoft portfolio. See mbsoftsystems.com for the full lineup.