We have several Jenkins instances in AWS eu-central-1 region (different public IPs). Today all hooks stopped working for us, we're getting NET_ERR for repo:push events.
      Bitbucket IPs are whitelisted

            [BCLOUD-17668] Hooks are not working

            @Andrii Sabitov and community, I tried to add all the cidr’s from https://ip-ranges.atlassian.com/ (count = 85) to the outbound security group in our AWS server but got error like “Number of IPs is more than acceptable limit“. Can you please correct me if I am doing anything wrong?

            Ashish Wani added a comment - @ Andrii Sabitov and community, I tried to add all the cidr’s from https://ip-ranges.atlassian.com/ (count = 85) to the outbound security group in our AWS server but got error like “Number of IPs is more than acceptable limit“. Can you please correct me if I am doing anything wrong?

            is there a way we can set webhooks from Bitbucket to Jenkins where Jenkins is running on private IP ?

            Able to expose publicly jenkins URL using Kong but not able to see option for the custom header for webhooks in Bitbucket.

            Can I know if anyone go through this before and able to solve

            suryatej yaramada added a comment - is there a way we can set webhooks from Bitbucket to Jenkins where Jenkins is running on private IP ? Able to expose publicly jenkins URL using Kong but not able to see option for the custom header for webhooks in Bitbucket. Can I know if anyone go through this before and able to solve

            mikoop added a comment -

            Question is, do all the ip ranges these need to be added to the webhook ip-whitelist (rule) and the firewall of the server?

            mikoop added a comment - Question is, do all the ip ranges these need to be added to the webhook ip-whitelist (rule) and the firewall of the server?

            My team also ran into this issue(bitbucket cloud webhooks gave the same error when trying to hit Bamboo Server instances in an AWS region). We were able to resolve it by adding the IPs found at https://ip-ranges.atlassian.com/.

            Leo Tronolone added a comment - My team also ran into this issue(bitbucket cloud webhooks gave the same error when trying to hit Bamboo Server instances in an AWS region). We were able to resolve it by adding the IPs found at https://ip-ranges.atlassian.com/ .

            @andrii_sabitov @leon_agrzegorczyk Ok, thank you for the response. As my co-worker (@BendeBruyn) already stated, for us this doesn't seem to do the trick! And it seems that the trigger works at random, but still with a RED NET_ERR status.

            Any suggestions based on the comment of Ben above?

            Michael Bom added a comment - @andrii_sabitov @leon_agrzegorczyk Ok, thank you for the response. As my co-worker (@BendeBruyn) already stated, for us this doesn't seem to do the trick! And it seems that the trigger works at random, but still with a RED NET_ERR status. Any suggestions based on the comment of Ben above?

            (Co-worker of @mbom):
            We've added áll the IP-addresses:

            #!cs
            13.55.145.74/32, 13.236.225.70/32, 13.236.240.90/32, 13.236.240.218/32, 13.237.22.210/32, 13.237.203.34/32, 34.198.210.246/32, 34.252.194.82/32, 35.160.117.30/32, 35.162.23.98/32, 35.167.86.65/32, 104.192.136.0/21, 52.8.252.137/32, 34.198.211.97/32, 34.208.237.45/32, 35.161.3.151/32, 35.164.29.75/32, 35.166.83.147/32, 52.214.35.33/32, 54.72.233.229/32, 34.192.15.175/32, 52.9.41.1/32, 54.76.3.75/32, 103.233.242.0/24, 52.8.84.222/32, 13.55.123.56/32, 13.237.238.24/32, 34.198.178.64/32, 34.208.39.80/32, 35.162.54.42/32, 52.63.74.64/32, 185.166.140.0/22, 2401:1d80::/32, 2406:da18:809:e00::/56, 2406:da1c:1e0:a200::/56, 2600:1f14:824:300::/56, 2600:1f18:2146:e300::/56, 2600:1f1c:cc5:2300::/56, 52.63.91.5/32, 2a05:d014:f99:dd00::/56, 52.215.192.128/25, 2a05:d018:34d:5800::/56, 52.51.80.244/32, 2a0a:ea00::/32, 34.198.32.85/32, 34.198.203.127/32, 13.55.180.21/32, 13.54.202.141/32, 13.52.5.0/25, 13.236.8.128/25, 18.136.214.0/25, 18.184.99.128/25, 18.234.32.128/25, 18.246.31.128/25, 34.208.209.12/32, 52.52.234.127/32, 18.205.93.0/27, 54.77.145.185/32
            

            but BitBucket still gives the NET_ERR

            In the past; when we didn't have the correct IPs whitelisted; we would get an response (from Bamboo) with something along the lines: "[some ip] isn't whitelisted". But it still would give an HTTP status code instead of NET_ERR.

            Thus; I still think that it is a problem within BitBucket.

            Ben de Bruyn added a comment - (Co-worker of @mbom): We've added áll the IP-addresses: #!cs 13.55.145.74/32, 13.236.225.70/32, 13.236.240.90/32, 13.236.240.218/32, 13.237.22.210/32, 13.237.203.34/32, 34.198.210.246/32, 34.252.194.82/32, 35.160.117.30/32, 35.162.23.98/32, 35.167.86.65/32, 104.192.136.0/21, 52.8.252.137/32, 34.198.211.97/32, 34.208.237.45/32, 35.161.3.151/32, 35.164.29.75/32, 35.166.83.147/32, 52.214.35.33/32, 54.72.233.229/32, 34.192.15.175/32, 52.9.41.1/32, 54.76.3.75/32, 103.233.242.0/24, 52.8.84.222/32, 13.55.123.56/32, 13.237.238.24/32, 34.198.178.64/32, 34.208.39.80/32, 35.162.54.42/32, 52.63.74.64/32, 185.166.140.0/22, 2401:1d80::/32, 2406:da18:809:e00::/56, 2406:da1c:1e0:a200::/56, 2600:1f14:824:300::/56, 2600:1f18:2146:e300::/56, 2600:1f1c:cc5:2300::/56, 52.63.91.5/32, 2a05:d014:f99:dd00::/56, 52.215.192.128/25, 2a05:d018:34d:5800::/56, 52.51.80.244/32, 2a0a:ea00::/32, 34.198.32.85/32, 34.198.203.127/32, 13.55.180.21/32, 13.54.202.141/32, 13.52.5.0/25, 13.236.8.128/25, 18.136.214.0/25, 18.184.99.128/25, 18.234.32.128/25, 18.246.31.128/25, 34.208.209.12/32, 52.52.234.127/32, 18.205.93.0/27, 54.77.145.185/32 but BitBucket still gives the NET_ERR In the past; when we didn't have the correct IPs whitelisted; we would get an response (from Bamboo) with something along the lines: " [some ip] isn't whitelisted". But it still would give an HTTP status code instead of NET_ERR . Thus; I still think that it is a problem within BitBucket.

            @mbom For us BitBuckets' requsts log shows green HTTP status 200 after implement list of ip ranges , just like before 1 Dec.

            Adam Grzegorczyk added a comment - @mbom For us BitBuckets' requsts log shows green HTTP status 200 after implement list of ip ranges , just like before 1 Dec.

            @mbom make sure that you added all IP addresses. Initially I added all IPv4 addresses before first IPv6 entry, but then discovered that there is another IPv4 address block below.

            andrii_sabitov added a comment - @mbom make sure that you added all IP addresses. Initially I added all IPv4 addresses before first IPv6 entry, but then discovered that there is another IPv4 address block below.

            @andrii_sabitov We are facing the same problem and also added the CIDR range from here. And after resending the request from BitBucket it still showing the NET_ERR-error. But in Bamboo we see the build is getting triggered. Did you have the same behavior? Or does BitBucket show the webhook request was sent successfully?

            Michael Bom added a comment - @andrii_sabitov We are facing the same problem and also added the CIDR range from here . And after resending the request from BitBucket it still showing the NET_ERR -error. But in Bamboo we see the build is getting triggered. Did you have the same behavior? Or does BitBucket show the webhook request was sent successfully?

            andrii_sabitov added a comment - Added IPs from https://ip-ranges.atlassian.com/

            I added IP addresses from list above, now it's working (hope these networks are under BB control).

            Just two requests to BitBucket:

            Thanks

            andrii_sabitov added a comment - I added IP addresses from list above, now it's working (hope these networks are under BB control). Just two requests to BitBucket: Can you update this What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? ? Is it possible to have a separate channel for breaking change announcements besides the Blog? It has a lot of marketing info which I don't want to read looking for valuable changes which have impact on our daily operations and infrastructure Thanks

            thanks Ryan , i was updated a wrong security group. working fine after updating correct security group

            Thaniyarasu Kannusamy added a comment - thanks Ryan , i was updated a wrong security group. working fine after updating correct security group

            rtanay added a comment -

            Likewise, adding the CIDRs from https://ip-ranges.atlassian.com/ solved the issue for us.

            rtanay added a comment - Likewise, adding the CIDRs from https://ip-ranges.atlassian.com/ solved the issue for us.

            We are still facing this issue even after updating ips into our security groups

            Thaniyarasu Kannusamy added a comment - We are still facing this issue even after updating ips into our security groups

            I wrote a simple python script, which can sync selected EC2 Security Group with a list provided by Atlassian. Maybe it will be useful for someone.

            https://github.com/agrzegorczyk-leonsoftware/atlassian-sg-updater

            I suggest to don't testing it directly on production SG

            Adam Grzegorczyk added a comment - I wrote a simple python script, which can sync selected EC2 Security Group with a list provided by Atlassian. Maybe it will be useful for someone. https://github.com/agrzegorczyk-leonsoftware/atlassian-sg-updater I suggest to don't testing it directly on production SG

            Adding these (https://ip-ranges.atlassian.com/) to our AWS security group resolved the issue for us.

            Joe Roberts added a comment - Adding these ( https://ip-ranges.atlassian.com/ ) to our AWS security group resolved the issue for us.

            devops added a comment -

            (Just tried the IP's in Adam Grzegorczyk post above in our Security Group and webhook now works )

            We're getting the same for eu-west-1. Enabling history in the webhook we see:

                HTTP status: 503
                Elapsed time: 5403ms
                Request time: 19 minutes ago (Monday, December 3rd 2018, 4:25:58 pm)
            
            Headers
            Content-Length	3215
            Via	1.1 ip-10-125-126-10.net.atlassian.com (squid)
            X-Cache	MISS from ip-10-125-126-10.net.atlassian.com
            Content-Language	en
            X-Squid-Error	ERR_CONNECT_FAIL 110
            X-Cache-Lookup	MISS from ip-10-125-126-10.net.atlassian.com:8080
            Vary	Accept-Language
            Server	squid
            Connection	keep-alive
            Date	Mon, 03 Dec 2018 16:25:58 GMT
            Content-Type	text/html;charset=utf-8
            Mime-Version	1.0
            

            devops added a comment - (Just tried the IP's in Adam Grzegorczyk post above in our Security Group and webhook now works ) We're getting the same for eu-west-1. Enabling history in the webhook we see: HTTP status: 503 Elapsed time: 5403ms Request time: 19 minutes ago (Monday, December 3rd 2018, 4:25:58 pm) Headers Content-Length 3215 Via 1.1 ip-10-125-126-10.net.atlassian.com (squid) X-Cache MISS from ip-10-125-126-10.net.atlassian.com Content-Language en X-Squid-Error ERR_CONNECT_FAIL 110 X-Cache-Lookup MISS from ip-10-125-126-10.net.atlassian.com:8080 Vary Accept-Language Server squid Connection keep-alive Date Mon, 03 Dec 2018 16:25:58 GMT Content-Type text/html;charset=utf-8 Mime-Version 1.0

            We are experiencing the same issue with our Jenkins instance in us-west-2: 529 NET_ERR

            Joe Roberts added a comment - We are experiencing the same issue with our Jenkins instance in us-west-2: 529 NET_ERR

            unfortunately I still see the issue

            andrii_sabitov added a comment - unfortunately I still see the issue

            We experiencing the same issue. In our case, this is caused by "silent" change of outbound IP ranges for webhooks by BB.
            This whitelist is not correct at the time: https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html\\
            After some digging, we found this blog post: https://bitbucket.org/blog/update-to-outgoing-webhook-ip-addresses\\
            After tests, I see, that in eu-west-1 we receiving hooks from 18.234.32.225-227.
            To be sure with incoming traffic we probably should add all range of Atlassian IPs to our security group whitelist:
            https://confluence.atlassian.com/bitbucket/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall-343343385.html\\
            https://ip-ranges.atlassian.com

            There is also post on https://stackoverflow.com/questions/53588092/recent-bitbucket-tls-update-issues-webhooks related to this issue.

            Adam Grzegorczyk added a comment - We experiencing the same issue. In our case, this is caused by "silent" change of outbound IP ranges for webhooks by BB. This whitelist is not correct at the time: https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html\\ After some digging, we found this blog post: https://bitbucket.org/blog/update-to-outgoing-webhook-ip-addresses\\ After tests, I see, that in eu-west-1 we receiving hooks from 18.234.32.225-227. To be sure with incoming traffic we probably should add all range of Atlassian IPs to our security group whitelist: https://confluence.atlassian.com/bitbucket/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall-343343385.html\\ https://ip-ranges.atlassian.com There is also post on https://stackoverflow.com/questions/53588092/recent-bitbucket-tls-update-issues-webhooks related to this issue.

            Seems fine now

            davide_bolognini_ca added a comment - Seems fine now

            We have the same problem: we have a jenkins in us-west-2. All webhooks requests are in stuck "IN PROGRESS" for hours. All ips whitelisted.

            davide_bolognini_ca added a comment - We have the same problem: we have a jenkins in us-west-2. All webhooks requests are in stuck "IN PROGRESS" for hours. All ips whitelisted.

              Unassigned Unassigned
              500511c8aeee andrii_sabitov
              Affected customers:
              3 This affects my team
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: