Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-14273

Provide Mirrors the ability to serve the robots.txt file

    XMLWordPrintable

Details

    • Suggestion
    • Resolution: Unresolved
    • None
    • Smart Mirroring
    • None
    • 0
    • We collect Bitbucket feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

    Description

      Bitbucket 5.11 introduced the ability to configure a robots.txt file for the primary Bitbucket instance (feature request: BSERV-3601)

      The robots.txt file can be saved in the $BITBUCKET_HOME/shared directory.

      Sample robots.txt request from the primary/upstream instance:

       

      Mirrors currently do not have this feature.

      Current Mirror behaviour

      When the <BITBUCKET_MIRROR_URL>/robots.txt is accessed, the request automatically gets redirected to <BITBUCKET_MIRROR_URL>/mirror

      Sample:

      The Mirror's access logs show the /robots.txt request was redirected to /mirror:

      10.255.0.1 | https | i@1UA6V0Fx402x19x0 | - | 2023-07-27 06:42:52,395 | "GET /robots.txt HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | - | - | - | - | - | - | 
10.255.0.1 | https | o@1UA6V0Fx402x19x0 | - | 2023-07-27 06:42:52,458 | "GET /robots.txt HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | 302 | 0 | 0 | - | 64 | - | 

10.255.0.1 | https | i@1UA6V0Fx402x20x0 | - | 2023-07-27 06:42:52,464 | "GET /mirror HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | - | - | - | - | - | - | 
10.255.0.1 | https | o@1UA6V0Fx402x20x0 | - | 2023-07-27 06:42:52,487 | "GET /mirror HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | 200 | 0 | 5926 | - | 23 | - |

       

      Feature request: Add the ability to configure and serve the robots.txt file for mirrors, similar to the primary instance.

      Attachments

        1. mirror-robots.txt-redirect.png
          669 kB
          JP Mariano
        2. primary-robots.txt-request.png
          61 kB
          JP Mariano

        Issue Links

          is related to

          Suggestion - BSERV-3601 robots.txt file for STASH to stop google from crawling my site

          • Closed

          Activity

            People

              Unassigned Unassigned
              jmariano@atlassian.com JP Mariano
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: