Provide Mirrors the ability to serve the robots.txt file

XMLWordPrintable

    • Type: Suggestion
    • Resolution: Unresolved
    • None
    • Component/s: Smart Mirroring
    • None
    • 0
    • 1

      Bitbucket 5.11 introduced the ability to configure a robots.txt file for the primary Bitbucket instance (feature request: BSERV-3601)

      The robots.txt file can be saved in the $BITBUCKET_HOME/shared directory.

      Sample robots.txt request from the primary/upstream instance:

       

      Mirrors currently do not have this feature.

      Current Mirror behaviour

      When the <BITBUCKET_MIRROR_URL>/robots.txt is accessed, the request automatically gets redirected to <BITBUCKET_MIRROR_URL>/mirror

      Sample:

      The Mirror's access logs show the /robots.txt request was redirected to /mirror:

      10.255.0.1 | https | i@1UA6V0Fx402x19x0 | - | 2023-07-27 06:42:52,395 | "GET /robots.txt HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | - | - | - | - | - | - | 
      10.255.0.1 | https | o@1UA6V0Fx402x19x0 | - | 2023-07-27 06:42:52,458 | "GET /robots.txt HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | 302 | 0 | 0 | - | 64 | - | 
      
      10.255.0.1 | https | i@1UA6V0Fx402x20x0 | - | 2023-07-27 06:42:52,464 | "GET /mirror HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | - | - | - | - | - | - | 
      10.255.0.1 | https | o@1UA6V0Fx402x20x0 | - | 2023-07-27 06:42:52,487 | "GET /mirror HTTP/1.1" | "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0" | 200 | 0 | 5926 | - | 23 | - | 
      

       

      Feature request: Add the ability to configure and serve the robots.txt file for mirrors, similar to the primary instance.

        1. primary-robots.txt-request.png
          61 kB
          JP Mariano
        2. mirror-robots.txt-redirect.png
          669 kB
          JP Mariano

            Assignee:
            Unassigned
            Reporter:
            JP Mariano
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: