Restrictions don't apply to current OpenAI models, but will affect future versions.
  • P03 LockeEnglish
    13•11 months ago

    You think Google thought about robots.txt before they developed their search engine? Nah, it’s all public Internet, and they scraped away.

    A non-zero percentage of web sites will bother to follow these instructions, but it might as well be zero.

    • ScrubblesEnglish
      8•11 months ago

      Yeah I always assumed robots.txt only told them to hide it from search results, but Google still scrapes everything they can from you. The illusion they skipped over you

      • The DoctorEnglish
        1•11 months ago

        If you look in the server logs, you can see what their spiders are grabbing.

    • The DoctorEnglish
      1•11 months ago

      Very early on, at least, their spiders respected robots.txt.

      I know there are folks that have all of the Big G in their robots.txt files on principle, might want to ask them if it works or not.