"AI-readable Markdown" — can't we just say "Markdown"? I'm serious about this, why are we focussing on making things accessible to AI when they should just be machine-readable and accessible to human beings in the first place? No need to taint this by bringing AI into it.
> why are we focussing on making things accessible to AI
Because that’s the authors actual goal? To take a web page that looks fine to human eyes but is unintuitively not accessible to AI. That’s genuinely useful and valuable.
Sure it’s no different than converting it to markdown for human eyes. But it’s important to be clear about not just WHAT but also WHY.
How to reliably HTML to MD for any page on the internet?
I remember struggling with this in the past
How hard would it be to build an MCP that's basically a proxy for web search except it always tries to build the markdown version of the web pages instead of passing HTML?
Basically Sosumi.ai but instead of working on only for Apple docs it works for any web page (including every doc on the internet)
There are APIs such as Jina AI's reader API that do this pretty well. It doesn't produce output as clean as Sosumi for Apple docs, but it's free and does a decent job.
This is awesome and timely for me...going to give it a whirl. Thanks for building. Also, there should totally be an easter egg where clicking something somewhere plays the sound!
The AI only sees a bit of HTML plus a bunch of JS that, when executed, generates more HTML. If the AI does not run the JS it won’t see everything. During training they probably use a crawler that runs a headless browser behind the scenes to get everything a human would get.
It is for the same reasons LLM are struggling to produce something that compile in Rust? I was under the impression that most of Rust documentation was plain HTML.
I think it's safe to assume most big players have browser rendering enabled (I hope so). imo AI is struggling with a lot of languages that are not as popular as javascript, mostly because it's more niche and you don't get a lot of good examples on the web.
In my experience, coding agents seem to do a normal fetch when provided links. Which makes sense — headless browser automation is expensive, and only really necessary for interacting with a webpage.
But with these RLHF'd AIs, being confident and helpful as they are, it took me a while to realize that they couldn't actually read the Apple developer links I was giving them. Like a kid who can't read the chalkboard, but doesn't realize they need glasses.
It's expensive for sure, but probably a drop in the water compared to the cost of renting H100. Plus, it would be a massive boost in terms of data quality/quantity for them. But maybe you are right, I'm just surprised it's not the case
Once upon a time, I built a project called `swift-doc`, which eventually got Sherlocked. I think what I was most upset about was their decision to call their thing "DocC". Like, adding redundant consonants to avoid name collisions is my shtick.
Most of the docs about the language itself live on swift.org (e.g. https://docs.swift.org/swift-book/documentation/the-swift-pr...). I've had pretty good luck getting Claude to write modern Swift code by saying things like "Use Swift 6 structured concurrency instead of GCD". But I could totally see expanding sosumi to include swift.org content, too.
I think it's because, in the world of programming, people are incredibly critical. Just try putting absolutely anything out there, and you'll get certain types of people picking it apart.
I remember an early experience in my working career, when someone was sharing their sample code with a group, to demonstrate a particular concept. And one of those present picked them up on their use of magic numbers, as if that was at all relevant in the context.
I don't blame anyone for being wary of showing their work in progress. Painters often don't like their subjects trying to take a sneek peak at their work in progress, as another example.
Just saying, sites like these are also pretty great for accessibility, screen reader users in particular.
I think this one would be slightly better if it rendered that Markdown as simple HTML if accessed through a real browser, but I can imagine even this version being pretty useful.
I think it could also make the "Small web" crowd pretty happy too.
Could you share what is your motivation behind doing whole domain and web hosting ?
Personally I feel that this whole AI induced problem should even exist in the first place, but even then it is ridiculous, that you have to query some web api to solve this problem, why not just publish parsed and converted to .md set of local files and be done with it.
I agree, it'd be great if Apple provided accessible documentation in the first place. Time was, Apple published self-contained docsets that you could download and read offline.
Apple's ToS pretty explicitly forbid the kind of automation required to download everything. But even if someone did that, it'd only be a snapshot in time. And a lot can can change between OS releases.
As for the hosted web app, I wanted to provide this as a public service. I plan to open source it, so anyone can self-host instead, if they're inclined.
Yeah, pre Swift documentation was very sufficient, dense and locally explorable, since then documentation started to resemble .h copy pasta without comments.
Because that’s the authors actual goal? To take a web page that looks fine to human eyes but is unintuitively not accessible to AI. That’s genuinely useful and valuable.
Sure it’s no different than converting it to markdown for human eyes. But it’s important to be clear about not just WHAT but also WHY.
C’mon now. This isn’t controversial or even bad.
How hard would it be to build an MCP that's basically a proxy for web search except it always tries to build the markdown version of the web pages instead of passing HTML?
Basically Sosumi.ai but instead of working on only for Apple docs it works for any web page (including every doc on the internet)
https://jina.ai/reader
But with these RLHF'd AIs, being confident and helpful as they are, it took me a while to realize that they couldn't actually read the Apple developer links I was giving them. Like a kid who can't read the chalkboard, but doesn't realize they need glasses.
Yes, that is why I quit using Claude and swapped to ChatGPT about a year ago. I've had substantially less issues with GPT.
If GitHub could support .docc files, that would be great. Otherwise, I still use Jazzy Docs.
Long live Jazzy.
Thanks!
Also, Apple has started shipping docs like this, too. They are a bit hidden but you can find them here:
/Applications/Xcode-beta.app/Contents/PlugIns/IDEIntelligenceChat.framework/Versions/A/Resources/AdditionalDocumentation
Curious how it handles some of the concurrency stuff. Actors, async/await etc..
- https://developer.apple.com/documentation/swift/tasklocal
- https://sosumi.ai/documentation/swiftui/button
- https://sosumi.ai/documentation/foundation/measurement
Most of the docs about the language itself live on swift.org (e.g. https://docs.swift.org/swift-book/documentation/the-swift-pr...). I've had pretty good luck getting Claude to write modern Swift code by saying things like "Use Swift 6 structured concurrency instead of GCD". But I could totally see expanding sosumi to include swift.org content, too.
Edit: Another Swift developer life hack: for new language features, copy-pasting the Swift Evolution proposal works pretty well! https://github.com/swiftlang/swift-evolution/tree/main/propo...
It’s ok to just start coding with a public repo. Code isn’t a secret.
I remember an early experience in my working career, when someone was sharing their sample code with a group, to demonstrate a particular concept. And one of those present picked them up on their use of magic numbers, as if that was at all relevant in the context.
I don't blame anyone for being wary of showing their work in progress. Painters often don't like their subjects trying to take a sneek peak at their work in progress, as another example.
- Not wanting to get roasted
- Open source = dealing with a lot of entitlement
And the list goes on. Putting code out into the world (publicly) often sets you up for future obligation of some kind (even if it’s just saying “no”).
None of this is a stance against open source, but I understand where people are coming from.
I think this one would be slightly better if it rendered that Markdown as simple HTML if accessed through a real browser, but I can imagine even this version being pretty useful.
I think it could also make the "Small web" crowd pretty happy too.
Personally I feel that this whole AI induced problem should even exist in the first place, but even then it is ridiculous, that you have to query some web api to solve this problem, why not just publish parsed and converted to .md set of local files and be done with it.
Apple's ToS pretty explicitly forbid the kind of automation required to download everything. But even if someone did that, it'd only be a snapshot in time. And a lot can can change between OS releases.
As for the hosted web app, I wanted to provide this as a public service. I plan to open source it, so anyone can self-host instead, if they're inclined.