-
Bug
-
Resolution: Fixed
-
Medium
We have several Jenkins instances in AWS eu-central-1 region (different public IPs). Today all hooks stopped working for us, we're getting NET_ERR for repo:push events.
Bitbucket IPs are whitelisted
[BCLOUD-17668] Hooks are not working
is there a way we can set webhooks from Bitbucket to Jenkins where Jenkins is running on private IP ?
Able to expose publicly jenkins URL using Kong but not able to see option for the custom header for webhooks in Bitbucket.
Can I know if anyone go through this before and able to solve
Question is, do all the ip ranges these need to be added to the webhook ip-whitelist (rule) and the firewall of the server?
My team also ran into this issue(bitbucket cloud webhooks gave the same error when trying to hit Bamboo Server instances in an AWS region). We were able to resolve it by adding the IPs found at https://ip-ranges.atlassian.com/.
@andrii_sabitov @leon_agrzegorczyk Ok, thank you for the response. As my co-worker (@BendeBruyn) already stated, for us this doesn't seem to do the trick! And it seems that the trigger works at random, but still with a RED NET_ERR status.
Any suggestions based on the comment of Ben above?
(Co-worker of @mbom):
We've added áll the IP-addresses:
#!cs 13.55.145.74/32, 13.236.225.70/32, 13.236.240.90/32, 13.236.240.218/32, 13.237.22.210/32, 13.237.203.34/32, 34.198.210.246/32, 34.252.194.82/32, 35.160.117.30/32, 35.162.23.98/32, 35.167.86.65/32, 104.192.136.0/21, 52.8.252.137/32, 34.198.211.97/32, 34.208.237.45/32, 35.161.3.151/32, 35.164.29.75/32, 35.166.83.147/32, 52.214.35.33/32, 54.72.233.229/32, 34.192.15.175/32, 52.9.41.1/32, 54.76.3.75/32, 103.233.242.0/24, 52.8.84.222/32, 13.55.123.56/32, 13.237.238.24/32, 34.198.178.64/32, 34.208.39.80/32, 35.162.54.42/32, 52.63.74.64/32, 185.166.140.0/22, 2401:1d80::/32, 2406:da18:809:e00::/56, 2406:da1c:1e0:a200::/56, 2600:1f14:824:300::/56, 2600:1f18:2146:e300::/56, 2600:1f1c:cc5:2300::/56, 52.63.91.5/32, 2a05:d014:f99:dd00::/56, 52.215.192.128/25, 2a05:d018:34d:5800::/56, 52.51.80.244/32, 2a0a:ea00::/32, 34.198.32.85/32, 34.198.203.127/32, 13.55.180.21/32, 13.54.202.141/32, 13.52.5.0/25, 13.236.8.128/25, 18.136.214.0/25, 18.184.99.128/25, 18.234.32.128/25, 18.246.31.128/25, 34.208.209.12/32, 52.52.234.127/32, 18.205.93.0/27, 54.77.145.185/32
but BitBucket still gives the NET_ERR
In the past; when we didn't have the correct IPs whitelisted; we would get an response (from Bamboo) with something along the lines: "[some ip] isn't whitelisted". But it still would give an HTTP status code instead of NET_ERR.
Thus; I still think that it is a problem within BitBucket.
@mbom For us BitBuckets' requsts log shows green HTTP status 200 after implement list of ip ranges , just like before 1 Dec.
@mbom make sure that you added all IP addresses. Initially I added all IPv4 addresses before first IPv6 entry, but then discovered that there is another IPv4 address block below.
@andrii_sabitov We are facing the same problem and also added the CIDR range from here. And after resending the request from BitBucket it still showing the NET_ERR-error. But in Bamboo we see the build is getting triggered. Did you have the same behavior? Or does BitBucket show the webhook request was sent successfully?
I added IP addresses from list above, now it's working (hope these networks are under BB control).
Just two requests to BitBucket:
- Can you update this What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? ?
- Is it possible to have a separate channel for breaking change announcements besides the Blog? It has a lot of marketing info which I don't want to read looking for valuable changes which have impact on our daily operations and infrastructure
Thanks
thanks Ryan , i was updated a wrong security group. working fine after updating correct security group
Likewise, adding the CIDRs from https://ip-ranges.atlassian.com/ solved the issue for us.
We are still facing this issue even after updating ips into our security groups
I wrote a simple python script, which can sync selected EC2 Security Group with a list provided by Atlassian. Maybe it will be useful for someone.
https://github.com/agrzegorczyk-leonsoftware/atlassian-sg-updater
I suggest to don't testing it directly on production SG
Adding these (https://ip-ranges.atlassian.com/) to our AWS security group resolved the issue for us.
(Just tried the IP's in Adam Grzegorczyk post above in our Security Group and webhook now works )
We're getting the same for eu-west-1. Enabling history in the webhook we see:
HTTP status: 503 Elapsed time: 5403ms Request time: 19 minutes ago (Monday, December 3rd 2018, 4:25:58 pm) Headers Content-Length 3215 Via 1.1 ip-10-125-126-10.net.atlassian.com (squid) X-Cache MISS from ip-10-125-126-10.net.atlassian.com Content-Language en X-Squid-Error ERR_CONNECT_FAIL 110 X-Cache-Lookup MISS from ip-10-125-126-10.net.atlassian.com:8080 Vary Accept-Language Server squid Connection keep-alive Date Mon, 03 Dec 2018 16:25:58 GMT Content-Type text/html;charset=utf-8 Mime-Version 1.0
We are experiencing the same issue with our Jenkins instance in us-west-2: 529 NET_ERR
We experiencing the same issue. In our case, this is caused by "silent" change of outbound IP ranges for webhooks by BB.
This whitelist is not correct at the time: https://confluence.atlassian.com/bitbucket/manage-webhooks-735643732.html\\
After some digging, we found this blog post: https://bitbucket.org/blog/update-to-outgoing-webhook-ip-addresses\\
After tests, I see, that in eu-west-1 we receiving hooks from 18.234.32.225-227.
To be sure with incoming traffic we probably should add all range of Atlassian IPs to our security group whitelist:
https://confluence.atlassian.com/bitbucket/what-are-the-bitbucket-cloud-ip-addresses-i-should-use-to-configure-my-corporate-firewall-343343385.html\\
https://ip-ranges.atlassian.com
There is also post on https://stackoverflow.com/questions/53588092/recent-bitbucket-tls-update-issues-webhooks related to this issue.
We have the same problem: we have a jenkins in us-west-2. All webhooks requests are in stuck "IN PROGRESS" for hours. All ips whitelisted.
@Andrii Sabitov and community, I tried to add all the cidr’s from https://ip-ranges.atlassian.com/ (count = 85) to the outbound security group in our AWS server but got error like “Number of IPs is more than acceptable limit“. Can you please correct me if I am doing anything wrong?