robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
While we're at it, it should be noted that Do Not Track was not, apparently, a joke.
It's the same as a noreply email, if you can get away with sticking your fingers in your ears and humming when someone is telling you something you don't want to hear, and you have a computer to hide behind, then it's all good.
It is ridiculous, but it is what you get when you have conflicting interests and broken legislation. The rule is that tracking has to be opt-in, so websites do it the way they are more likely to get people to opt in, and it is a cookie banner before you access the content.
Do-not-track is opt-out, not opt-in, and in fact, it is not opt-anything since browsers started to set it to "1" by default without asking. There is no law forcing advertisers to honor that.
I guess it could work the other way: if you set do-not-track to 0 (meaning "do-track"), which no browser does by default, make cookies auto-accept and do not show the banner. But then the law says that it should require no more actions to refuse consent than to consent (to counter those ridiculous "accept or uncheck 100 boxes" popups), so it would mean they would also have to honor do-not-track=1, which they don't want to.
I don't know how legislation could be unbroken. Users don't want ads, don't want tracking, they just want the service they ask for and don't want to pay for it. Service providers want exactly the opposite. Also people need services and services need users. There is no solution that will satisfy everyone.
For one, Do Not Track is on the client side and you just hope and pray that the server honors it, whereas cookie consent modals are something built by and placed in the website.
I think you can reasonably assume that if a website went through the trouble of making such a modal (for legal compliance reasons), the functionality works (also for legal compliance reasons). And, you as the client can verify whether it works, and can choose not to store them regardless.
So it did the same work that a sitemap does? Interesting.
Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
> I didn’t realize its original purpose was to manage duplicate content penalties though.
That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says:
> In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).
> It was mostly about stopping crawlers from unnecessarily consuming server resources.
Very much so.
Computation was still expensive, and http servers were bad at running cgi scripts (particularly compared to the streamlined amazing things they can be today).
SEO considerations came way way later.
They were also used, and still are, by sites that have good reasons to not want results in search engines. Lots of court files and transcripts, for instance, are hidden behind robots.txt.
I always consider "good" a bot that doesn't disguise itself and follows the robots.txt rules. I may not consider good the final intent of the bot or the company behind it, but the crawler behaviour is fundamentally good.
Especially considering the fact that it is super easy to disguise a crawler and not follow the robots conventions
Well you as the person running a website can define unilaterally what you consider good and bad. You may want bots to crawl everything, nothing or (most likely) something inbetween. Then you judge bots based on those guidelines. You know like a solicitor that rings your bell that has a text above it saying "No solicitors", certain assumptions can be made about those who ignore it.
I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others.
But I kinda like having this attitude and expectation. Makes me feel healthier.
> Trust by default, also by default, never ignoring suspicious signals.
While I absolutely love the intent of this idea, it quickly falls apart when you're dealing with systems where you only get the signals after you've already lost everything of value.
It's easy to believe, though, and most of us do it every day. For example, our commute to work is marked by the trust that other drivers will cooperate, following the rules, so that we all get to where we are going.
There are varying degrees of this through our lives, where the trust lies not in the fact that people will just follow the rules because they are rules, but because the rules set expectations, allowing everyone to (more or less) know what's going on and decide accordingly. This also makes it easier to single out the people who do not think the rules apply to them so we can avoid trusting them (and, probably, avoid them in general).
> For example, our commute to work is marked by the trust that other drivers will cooperate, following the rules, so that we all get to where we are going.
That trust comes from the knowledge that it's likely that those drivers also don't want to crash, and would rather prefer to get where they're going.
In Southern Europe, and countries with similar cultures, we don't obey rules because someone says so, we obey them when we see that is actually reasonable to do so, hence my remark regarding culture as I also experienced living in countries where everyone mostly blindly follow the rules, even if they happen to be nonsense.
Naturally I am talking about cultures where that decision has not been taken away from their citizens.
> I also experienced living in countries where everyone mostly blindly follow the rules, even if they happen to be nonsense.
The problem with that is that most people are not educated enough to judge what makes sense and what doesn’t, and the less educated you are, the more likely you are to believe you know what makes sense when you’re actually wrong. These are exactly the people that should be following the rules blindly, until they actually put in the effort to learn why those rules exist.
I believe there is a difference between education and critical thinking. One may not have a certain level of education, but could exercise a great degree of critical thinking. I think that education can help you understand the context of the problem better. But there are also plenty of people who are not asking the right questions or not asking questions - period - who have lots of education behind them. Ironically, sometimes education is the path that leads to blind trust and lack of challenging the status quo.
> the less educated you are, the more likely you are to believe you know what makes sense
It actually frightens me how true this statement is.
To reinforce my initial position about how important the rules are for setting expectations, I usually use cyclists as an example. Many follow the proposed rules, understanding they are traffic, and right of way is not automagically granted based on the choice of vehicle, having more to do with direction and the flow of said traffic.
But there's always a bad apple, a cyclist who assumes themselves to be exempt from the rules and rides against the flow of traffic, then wonders why they got clipped because a right-turning driver wasn't expecting a vehicle to be coming from the direction traffic is not supposed to come from.
In the end, it's not really about what we drive or how we get around, but whether we are self-aware enough to understand that the rules apply to us, and collectively so. Setting the expectation of what each of our behaviors will be is precisely what creates the safety that comes with following them, and only the dummies seem to be the ones who think they are exempt.
As a French, being passed by the right by Italian drivers on the highway really makes me feel the superiority of Southern Europeans judgment over my puny habit of blindly following rules. Or does it?
But yes, I do the same. I just do not come here to pretend this is virtue.
The rules in France are probably different but passing on the right is legal on Italian highways, in one circumstance: if one keeps driving on the lane on the right and somebody slower happens to be driving on the lane on the left. The rationale is that it normally happens when traffic is packed, so it's ok even if there is little traffic. Everybody keep driving straight and there is no danger.
It's not legal if somebody is following the slower car on the left and steers to the right to pass. However some drivers stick to the left at a speed slower than the limit and if they don't yield what happens is that eventually they get passed on the right.
The two cases have different names. The normal pass is "sorpasso", the other one (passing by not steering) is "superamento", which is odd but they had to find a word for it.
Not sure if it is a virtue, but standing as a pedestrians in an empty street at 3 AM waiting for a traffic light to turn green doesn't make much sense either, it isn't as if a ghost car is coming out of nowhere.
It should be a matter of judgement and not following rules just because.
I kind of agree. The rules for safety should be simple, straightforward, and protect you in the "edge cases", i.e. following while not paying 100% of attention, protect you with a malicious actor in mind aka reckless driver, etc. Ideally, in a system like that it should be a difficult and intentional behavior if one wanted to break the rules rather than to follow them.
I apologize for that. I try to mitigate my US-centricness in my comments as much as possible, understanding completely that I am speaking with a global audience, but I am definitely not perfect at it :D
I suppose the same goes if you take the tube, ride a bike, walk, etc? There's still rules in terms of behavior, flow of traffic (even foot traffic), etc, that helps set a number of expectations so everyone can decide and behave accordingly. Happy to hear different thoughts on this!
I still see the value in robots.txt and DNT as a clear, standardised way of posting a "don't do this" sign that companies could be forced to respect through legal means.
The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR.
Would robot traffic be considered tracking in light of GDPR standards? As far as I know there are no regulatory rules in relation to enforcing robots behaviors outside of robots.txt, which is more of an honor system.
DNT and GDPR was just an example. In a court case about tracking, DNT could be found to be a clear and explicit opt out. Similarly, in a case about excessive scraping or the use of scraped information, robots txt could be used as a clear and explicit signal that the site operator does not want their pages harvested. It all but certainly gets rid of the "they put it on the public web so we assumed we can scrape it, we can'task everyone for permission" argument. They can't claim it was "in good faith" if there's a widely-accepted standard for opting out.
> That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen.
The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
I created a search engine that crawled the web way back in 2003. I used a proper user agent that included my email address. I got SO many angry emails about my crawler, which played as nice as I was able to make it play. Which was pretty nice I believe. If it’s not Google people didn’t want it. That’s a good way to prevent anyone from ever competing with Google. It isn’t just about that preview for LinkedIn, it’s about making sure the web is accessible by everyone and everything that is trying to make its way. Sure, block the malicious ones. But don’t just assume that every bot is malicious by default.
I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes with an intentionality behind it. If you want to allow or disallow certain traffic, you also should answer the question why or why not. That requires understanding what each bot does. That takes time and effort.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
The most annoying thing about being a good bot owner, in my experience, is when you get complaints about it misbehaving, only to find that it was actually somebody malicious who wrote their own abusive bot, but is using your bot's user agent.
Cloudflare have some new bot verification proposals designed to fix this, with cryptographic proofs that the user-agent is who they say they are: https://blog.cloudflare.com/web-bot-auth/.
That's easy to say when it's your bot, but I've been on the other side to know that the problem isn't your bot, it's the 9000 other ones just like it, none of which will deliver traffic anywhere close to the resources consumed by scraping.
Yeah, there were times, even running a fairly busy site, that the bots would outnumber user traffic 10:1 or more, and the bots loved to endlessly troll through things like archive indexs that could be computationally (db) expensive. At one point it got so bad that I got permission to just blackhole all of .cn and .ru, since of course none of those bots even thought of half obeying robots.txt. That literally cut CPU load on the database server by more than half.
For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable for the rest of the current "free billing" cycle, i.e. until your quota gets renewed.
True. Major search engines and bots from social networks have a clear value proposition: in exchange for consuming my resources, they help drive human traffic to my site. GPTBot et al. will probably do the same, as more people use AI to replace search.
A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back.
An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.
> GPTBot et al. will probably do the same, as more people use AI to replace search.
It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.
That was my hunch. My initial post on robots.txt: https://evgeniipendragon.com/posts/i-am-disallowing-all-craw... - revolved around blocking AI models from doing that because I do not believe that it will bring more traffic to my website - it will use the content to keep people using their service. I might be proven wrong in the future, but I do not see why they would want to let go of an extra opportunity to increase retention.
Only if the GPT companies can resist the temptation of all that advertising $$$.
I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.
> but that ideal only holds between consenting adults.
If your webserver serves up the page, you've already pre-consented.
One of my retirement plans has a monthly statement available as a pdf document. We're allowed to download that. But the bot I wrote to download it once a month was having trouble, they used some fancy bot detection library to cockblock it. Wasn't allowed to use Mechanize. Why? Who the fuck knows. I'm only allowed to have that statement if I can be bothered to spend 15 minutes a month remembering how to fucking find it on their site and downloading it manually, rather than just saving a copy. Banks are even worse... they won't keep a copy of your statements longer than 6 months, but go apeshit if you try to have those automatically downloaded.
I don't ask permission or play nice anymore. Your robots.txt is ignorable, so I ignore it. I do what I want, and you're the problem not me.
It's just that people are suspicious of unknown crawlers, and rightly so.
Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.
It sucks more that Cloudflare/similar have responded to this with "if your handshake fingerprints more like curl than like Chrome/Firefox, no access for you".
There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.
Or getting a CAPTCHA from Chrome when visiting a site you've been to dozens of times (Stack Overflow). Now I just skip that content, probably in my LLM already anyway.
It's the same thing as the anti pirate ads, you only annoy legit customers, this agressive captcha campaign just makes Stackoverflow drop down even faster than it would normally by making it lower quality.
I now write all of my bots in javascript and run them from the Chrome console with CORS turned off. It seems to defeat even Google's anti-bot stuff. Of course, I need to restart Chrome every few hours because of memory leaks, but it wasn't a fun 3 days the last time I got banned from their ecosystem with my kids asking why they couldn't watch Youtube.
I guess back in 2003 people would expect an email to actually go somewhere, these days I would expect it to either go nowhere or just be part of a campaign to collect server admin emails for marketing/phishing purposes. Angry emails are always a bit much, but I wonder if they aren't sent as much anymore in general or if people just stopped posting them to point and laugh at and wonder what goes through people's minds to get so upset to send such emails.
My somewhat silly take on seeing a bunch of information like emails in a user agent string is that I don't want to know about your stupid bot. Just crawl my site with a normal user agent and if there's a problem I'll block you based on that problem. It's usually not a permanent block, and it's also usually setup with something like fail2ban so it's not usually an instant request drop. If you want to identify yourself as a bot, fine, but take a hint from googlebot and keep the user agent short with just your identifier and an optional short URL. Lots of bots respect this convention.
But I'm just now reminded of some "Palo Alto Networks" company that started dumping their garbage junk in my logs, they have the audacity to include messages in the user agent like "If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" or "find out more about our scans in [link]". I put a rule in fail2ban to see if they'd take a hint (how about your dumb bot detects that it's blocked and stops/slows on its own accord?) but I forgot about it until now, seems they're still active. We'll see if they stop after being served nothing but zipbombs for a while before I just drop every request with that UA. It's not that I mind the scans, I'd just prefer to not even know they exist.
I think a better solution would be to block all the traffic, but have a comment in robots.txt with a way to be added onto a whitelist to scrape the contents of the resource. This puts a burden of requesting the access on the owner of the bot, and if they really want that access, they can communicate it and we can work it out.
The problem with robots.txt is the reliance on identity rather purpose of the bots.
The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies.
What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival".
And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
I do like your suggestion of creating some standard that categorizes using function or purpose like you mention. This could simplify things granted that there is a way to validate the function and for spoofing to be hard to achieve. And yes - there is also legal.
I do think that I will likely need to go back and unblock a couple of other bots for this exact reason - so that it would be possible to share it and have previews in other social media. I like to take a slow and thoughtful approach to allowing this traffic as I get to learn what it is that I want and do not want.
Comments here have been a great resource to learn more about this issue and see what other people value.
This is just a problem of sharing information in band instead of out of band. The OpenGraph metadata is in band with the page content that doesn't need to be shared with OpenGraph bots. The way to separate the usage is to separate the content and metadata with some specific query using `content-type` or `HEAD` or something, then bots are free to fetch that (useless for AI bots) and you can freely forbid all bots from the actual content. Then you don't really need much of a legal framework.
I like the idea of using HEAD or OPTIONS methods and have all bots access that so that they get a high level idea of what's going on, without the access to actual content, if the owner decided to block it.
I try to stay away from negative takes here, so I’ll keep this as constructive as I can:
It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight.
I mean it was a realization for me, although I wouldn't call it profound. To your point, it was closer to obliviousness, which led me to learn more about Open Graph Protocol details and how Robots Exclusion Protocol works.
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
it's mostly that they didn't think of the page preview fetcher as a "crawler", and did not intend for their robots.txt to block it. it may not be profound but it's at the least not a completely trivial realisation. and heck, an actual human-written blog post can okay improve the average quality of the web.
The bots are called "crawlers" and "spiders", which to me evokes the image of tiny little things moving rapidly and mechanically from one place to another, leaving no niche unexplored. Spiders exploring a vast web.
Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term.
It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
> Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term.
It’s definitely not crawling as robots.txt defines the term.
:
> WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages.
You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
I agree, and I am also confused on how this got on the frontpage of all things.
It's like reading a news article of 'water is wet'.
You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
Weirdly, this is something that Apple actually gets right - the little „previews” you get when sharing links in iMessage get generated client-side; _by the sender_.
There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
The "full solution" to this, of course, is micropayments. A bot which has to pay a tenth of a cent to you every time it visits one of your pages or something else the page 404s will quickly rack up a $10 bill crawling a whole 10,000 page site. If it tries to do that every day, or every hour, that's an excellent payday for you and a very compelling reason for almost all bots to blacklist your domain name.
A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that.
The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
The problem is the robots that do follow robots.txt its all the bots that don't. Robots.txt is largely irrelevant now they don't represent most of the traffic problem. They certainly don't represent the bots that are going to hammer your site without any regard, those bots don't follow robots.txt.
I wonder whether the path in the robots.txt (or maybe a <link> tag with a bogus rel attribute) would already be enough to make evil bots follow it. That would at least avoid accidental follows due to CSS/ARIA not properly working in weird constellations
I would hope that all screen readers would respect display:none. The aria-hidden is for CYA, banning even one blind user would be quite bad optics (as is this sentence now that I think about it).
Not sure why you were downvoted. I have zero confidence that OpenAI, Anthropic, and the rest respect robots.txt however much they insist they do. It's also clear that they're laundering their traffic through residential ISP IP addresses to make detection harder. There are plenty of third-parties advertising the service, and farming it out affords the AI companies some degree of plausible deniability.
Nobody has any confidence in ai to not ddos. That's why there have been dozens of posts about how bandwidth is becoming an issue for many websites as bots continuously crawl their sites for new info.
Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day.
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
What astounds me is there are no readily available libraries crawler authors can reach for to parse robots.txt and meta robots tags, to decide what is allowed, and to work through the arcane and poorly documented priorities between the two robots lists, including what to do when they disagree, which they often do.
Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not.
Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
rel=nofollow is a bad name. It doesn’t actually forbid following the link and doesn’t serve the same purpose as robots.txt.
The problem it was trying to solve was that spammers would add links to their site anywhere that they could, and this would be treated by Google as the page the links were on endorsing the page they linked to as relevant content. rel=nofollow basically means “we do not endorse this link”. The specification makes this more clear:
> By adding rel="nofollow" to a hyperlink, a page indicates that the destination of that hyperlink should not be afforded any additional weight or ranking by user agents which perform link analysis upon web pages (e.g. search engines).
> nofollow is a bad name […] does not mean the same as robots exclusion standards
Having worked on bot detection in the past. Some really simple old fashioned attacks happened by doing the opposite of what the robots.txt file says.
While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
This reminds me of an old friend of mine who wrote long revelation posts on how he started using the "private" keyword in C++ after compiler helped him to find why a class member changed unexpectedly and how he no longer drives car with the clutch half-pressed because it burns the clutch.
This doesn't seem like a new discovery at all - this is what news publications have been dealing with ever since they went online.
You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
LinkedIn is by far the worst offender in post previews. The doctype tag must be all lowercase. The HTML document must be well-formed (the meta tags must be in an explicit <head> block, for example). You must have OG meta tags for url, title, type, and image. The url meta tag gets visited, even if it's the same address the inspector is already looking at.
Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
if you are hosting a house party that invites the entire world robots.txt is a neon sign to guide guests to where the beers are, who's cooking what kind of burgers and on what grill; rules of the house etc - you'll still have to secure your gold chains and laptop in a safe somewhere or decide to even keep them in the same house yourself
Worst offenders I come across: official government information that needs to be public, placed behind Cloudflare, preventing even their M2M feeds (RSS, Atom, ...) to be accessed
Maybe he is talking about stuff you're required by law to disclose but you don't really want to be seen too much. Like code of conduct, terms of service, retractions or public apologies.
Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.
robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
It's the same as a noreply email, if you can get away with sticking your fingers in your ears and humming when someone is telling you something you don't want to hear, and you have a computer to hide behind, then it's all good.
Do-not-track is opt-out, not opt-in, and in fact, it is not opt-anything since browsers started to set it to "1" by default without asking. There is no law forcing advertisers to honor that.
I guess it could work the other way: if you set do-not-track to 0 (meaning "do-track"), which no browser does by default, make cookies auto-accept and do not show the banner. But then the law says that it should require no more actions to refuse consent than to consent (to counter those ridiculous "accept or uncheck 100 boxes" popups), so it would mean they would also have to honor do-not-track=1, which they don't want to.
I don't know how legislation could be unbroken. Users don't want ads, don't want tracking, they just want the service they ask for and don't want to pay for it. Service providers want exactly the opposite. Also people need services and services need users. There is no solution that will satisfy everyone.
I think you can reasonably assume that if a website went through the trouble of making such a modal (for legal compliance reasons), the functionality works (also for legal compliance reasons). And, you as the client can verify whether it works, and can choose not to store them regardless.
Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says:
> In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).
— https://www.robotstxt.org/orig.html
Very much so.
Computation was still expensive, and http servers were bad at running cgi scripts (particularly compared to the streamlined amazing things they can be today).
SEO considerations came way way later.
They were also used, and still are, by sites that have good reasons to not want results in search engines. Lots of court files and transcripts, for instance, are hidden behind robots.txt.
Well, yes, the point is to tell the bots what you've decided to consider "bad" and will ban them for. So that they can avoid doing that.
Which of course only works to the degree that they're basically honest about who they are or at least incompetent at disguising themselves.
I always consider "good" a bot that doesn't disguise itself and follows the robots.txt rules. I may not consider good the final intent of the bot or the company behind it, but the crawler behaviour is fundamentally good.
Especially considering the fact that it is super easy to disguise a crawler and not follow the robots conventions
I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others.
But I kinda like having this attitude and expectation. Makes me feel healthier.
Trust by default, also by default, never ignoring suspicious signals.
Trust is not being naïve, I find the confusion of both very worrying.
While I absolutely love the intent of this idea, it quickly falls apart when you're dealing with systems where you only get the signals after you've already lost everything of value.
Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation.
EDIT: This one: https://youtu.be/mScpHTIi-kM
There are varying degrees of this through our lives, where the trust lies not in the fact that people will just follow the rules because they are rules, but because the rules set expectations, allowing everyone to (more or less) know what's going on and decide accordingly. This also makes it easier to single out the people who do not think the rules apply to them so we can avoid trusting them (and, probably, avoid them in general).
That trust comes from the knowledge that it's likely that those drivers also don't want to crash, and would rather prefer to get where they're going.
Naturally I am talking about cultures where that decision has not been taken away from their citizens.
The problem with that is that most people are not educated enough to judge what makes sense and what doesn’t, and the less educated you are, the more likely you are to believe you know what makes sense when you’re actually wrong. These are exactly the people that should be following the rules blindly, until they actually put in the effort to learn why those rules exist.
It actually frightens me how true this statement is.
To reinforce my initial position about how important the rules are for setting expectations, I usually use cyclists as an example. Many follow the proposed rules, understanding they are traffic, and right of way is not automagically granted based on the choice of vehicle, having more to do with direction and the flow of said traffic.
But there's always a bad apple, a cyclist who assumes themselves to be exempt from the rules and rides against the flow of traffic, then wonders why they got clipped because a right-turning driver wasn't expecting a vehicle to be coming from the direction traffic is not supposed to come from.
In the end, it's not really about what we drive or how we get around, but whether we are self-aware enough to understand that the rules apply to us, and collectively so. Setting the expectation of what each of our behaviors will be is precisely what creates the safety that comes with following them, and only the dummies seem to be the ones who think they are exempt.
But yes, I do the same. I just do not come here to pretend this is virtue.
It's not legal if somebody is following the slower car on the left and steers to the right to pass. However some drivers stick to the left at a speed slower than the limit and if they don't yield what happens is that eventually they get passed on the right.
The two cases have different names. The normal pass is "sorpasso", the other one (passing by not steering) is "superamento", which is odd but they had to find a word for it.
It should be a matter of judgement and not following rules just because.
I suppose the same goes if you take the tube, ride a bike, walk, etc? There's still rules in terms of behavior, flow of traffic (even foot traffic), etc, that helps set a number of expectations so everyone can decide and behave accordingly. Happy to hear different thoughts on this!
The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR.
That was a theory at least...
It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen.
The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back.
An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.
It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.
I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.
If your webserver serves up the page, you've already pre-consented.
One of my retirement plans has a monthly statement available as a pdf document. We're allowed to download that. But the bot I wrote to download it once a month was having trouble, they used some fancy bot detection library to cockblock it. Wasn't allowed to use Mechanize. Why? Who the fuck knows. I'm only allowed to have that statement if I can be bothered to spend 15 minutes a month remembering how to fucking find it on their site and downloading it manually, rather than just saving a copy. Banks are even worse... they won't keep a copy of your statements longer than 6 months, but go apeshit if you try to have those automatically downloaded.
I don't ask permission or play nice anymore. Your robots.txt is ignorable, so I ignore it. I do what I want, and you're the problem not me.
Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.
I'll bite. It seems like a poor strategy to trust by default.
That aggressive crawling to train those on everything is insane.
My somewhat silly take on seeing a bunch of information like emails in a user agent string is that I don't want to know about your stupid bot. Just crawl my site with a normal user agent and if there's a problem I'll block you based on that problem. It's usually not a permanent block, and it's also usually setup with something like fail2ban so it's not usually an instant request drop. If you want to identify yourself as a bot, fine, but take a hint from googlebot and keep the user agent short with just your identifier and an optional short URL. Lots of bots respect this convention.
But I'm just now reminded of some "Palo Alto Networks" company that started dumping their garbage junk in my logs, they have the audacity to include messages in the user agent like "If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" or "find out more about our scans in [link]". I put a rule in fail2ban to see if they'd take a hint (how about your dumb bot detects that it's blocked and stops/slows on its own accord?) but I forgot about it until now, seems they're still active. We'll see if they stop after being served nothing but zipbombs for a while before I just drop every request with that UA. It's not that I mind the scans, I'd just prefer to not even know they exist.
The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies.
What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival".
And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
I do think that I will likely need to go back and unblock a couple of other bots for this exact reason - so that it would be possible to share it and have previews in other social media. I like to take a slow and thoughtful approach to allowing this traffic as I get to learn what it is that I want and do not want.
Comments here have been a great resource to learn more about this issue and see what other people value.
Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.
It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight.
And then they posted it here for attention.
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term.
It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
It’s definitely not crawling as robots.txt defines the term. :
> WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages.
— https://www.robotstxt.org/orig.html
You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
1)
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews.
2)
From your first post,
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that.
The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
Deny /honeypot in your robots.txt
Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html
If an IP accesses that path, ban it.
Unrelated meta question but is the aria tag necessarily since display: none; should be removing the content from the flow?
Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day.
I think debian also recently spoke up about it.
As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not.
Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
rel=nofollow is a bad name. It doesn’t actually forbid following the link and doesn’t serve the same purpose as robots.txt.
The problem it was trying to solve was that spammers would add links to their site anywhere that they could, and this would be treated by Google as the page the links were on endorsing the page they linked to as relevant content. rel=nofollow basically means “we do not endorse this link”. The specification makes this more clear:
> By adding rel="nofollow" to a hyperlink, a page indicates that the destination of that hyperlink should not be afforded any additional weight or ranking by user agents which perform link analysis upon web pages (e.g. search engines).
> nofollow is a bad name […] does not mean the same as robots exclusion standards
— https://microformats.org/wiki/rel-nofollow
Shouldn't be that hard if someone WANT to be good.
While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
If you don't want people to crawl your content, don't put it online.
There are so many consequences of disallowing robots -- what about the Internet Archive for example?
I want to post online but I don't want random asshole driveby's involved.
I wish there were way less posts like this.
You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
It is amazing what people think is important these days.
Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
It used to be this ultrafake eternal job interview site, but people now seem uninhibited to go on wild political rants even there.