That’s how Konstantin Ryabitsev titles his post creepy crawlies, where he explains the scale of the problem of the “AI crawlers” in the kernel.org git repositories website. I have discussed it here a few times: the problem of the LLM crawlers, how I simplified how I self host git repos, and my attempts trying to deal with these crawlers. I haven’t reached a solution and the crawlers are still there –less so, as the blocking and some of the Javascript tricks did work–, although I haven’t tried more drastic solutions like Anubis. My problem is not at the same scale as what the people at kernel.org are experiencing: We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. That’s a lot of CPU cores! The post is a good read, because it explains how they have tried several things, from blocking the bots to using the aforementioned Anubis, until everything stops being as effective. It is an arms race: A few months later, the bots were back, solving difficulty 4. No problem, we said, let’s raise difficulty to 5. The legitimate users were more annoyed now. Difficulty 5 takes a few seconds to solve on a mobile device, and the phone gets uncomfortably warm as it’s doing the number crunching. However, it was effective and bought us a few more months of peace. Then… the bots started solving difficulty 5. Konstantin doesn’t end the post with a clear conclusion. What can we do about this? From my point of view, the obvious solution is just closing the service, like some friends have done. None of my open source projects, or the code I share on my server, is that important. Which is probably good, as it means my problem is small, and I won’t need 14 CPU cores to keep my repos online. Yet, these bots are wasting my limited resources. And for no benefit, I suspect. I don’t see all this crawling having any point. It is my code we are talking about!