OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly trying to scrape the entire site, whose robots.txt file was not properly configured (Julie Bort/TechCrunch)

Julie Bort / TechCrunch:
OpenAI's crawlers took down e-commerce site Triplegangers by relentlessly trying to scrape the entire site, whose robots.txt file was not properly configured  —  On Saturday, Triplegangers CEO Oleksandr Tomchuk was alerted that his company's e-commerce site was down.

from Techmeme https://ift.tt/r0plCjn