Fisheye should support robots.txt

XMLWordPrintable

    • Type: Suggestion
    • Resolution: Unresolved
    • N/A
    • Component/s: None

      Fisheye is a crawler, and so like any good crawler citizen, should:
      a) advertise a proper User-Agent string
      b) read and obey what's in robots.txt, particularly the Crawl-delay: parameter, so that administrators can stop rogue Fisheyes from killing the system.

              Assignee:
              Unassigned
              Reporter:
              Jeff Turner
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: