Drunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 months agoHow to combat large amounts of Ai scrapersmessage-squaremessage-square53fedilinkarrow-up1121arrow-down15file-text
arrow-up1116arrow-down1message-squareHow to combat large amounts of Ai scrapersDrunk & Root@sh.itjust.works to Selfhosted@lemmy.worldEnglish · 2 months agomessage-square53fedilinkfile-text
everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
minus-squarekcweller@feddit.nllinkfedilinkEnglisharrow-up14·2 months agoNephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.
minus-squareDrunk & Root@sh.itjust.worksOPlinkfedilinkEnglisharrow-up4·2 months agothis is really awesome but i use a vps so i dont know how they would feel about this being deployed
Nephentes that shit. Poison every scraper until they start respecting robot.txt. Purposefully use llm.txt to trap the fuckers.
this is really awesome but i use a vps so i dont know how they would feel about this being deployed