Pro@programming.dev to Technology@lemmy.worldEnglish · 1 month agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comexternal-linkmessage-square223fedilinkarrow-up11.08Karrow-down19file-textcross-posted to: [email protected]
arrow-up11.07Karrow-down1external-linkCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.dev to Technology@lemmy.worldEnglish · 1 month agomessage-square223fedilinkfile-textcross-posted to: [email protected]
minus-squareT156@lemmy.worldlinkfedilinkEnglisharrow-up7·1 month agoThe website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
minus-squareacockworkorange@mander.xyzlinkfedilinkEnglisharrow-up1·1 month agoYou could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.
The website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.
You could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.