-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Low
-
Affects Version/s: 8.5.5, 8.5.11, 8.5.23
-
Component/s: Server - Performance
-
20
-
Severity 2 - Major
-
16
Issue Summary
Memory leak while accessing <base-url>label/<labelname> (label search) on objects created in io.micrometer.core.instrument.ImmutableTag
This is reproducible on the Data Center: yes
Steps to Reproduce
- Use the following script to search randomly for labels
while : do curl http://127.0.0.1:8090/label/cat+dog+$RANDOM done
- Capture heap dump after some time after stopping the script
- Run following OQL in MAT
SELECT value.toString() FROM "io.micrometer.core.instrument.ImmutableTag.*" WHERE (key.toString() = "pathPattern")
- We observe the following objects in the heap. These objects are not cleared until the node is restarted

Expected Results
Objects should be collected by the Garbage Collector.
Actual Results
The objects remain in the heap till the node is restarted.

Note: Authentication is not required to exploit this method. Having an SSO doesn't seem to make any difference
This can be used in a denial-of-service attack although it takes a lot of time to fill the heap with the above method. Using longer URLs can fill up the heap more quickly. I was able to see even the following URI put in the heap(the upper limit may be governed by the size of request header tomcat accepts )
label.atom+cat+dog+18259+1052+54+mouse+24913+sbdabdjksandnsadnbsadabdmnbandsbdnbasnmdbanmbdsanmbdnsabdnasmbdabdmabdmnbsadbdmasbdmsanadmnsadsanmdbasdnmasbdnmasdbasnmdbsanmdbanmbdmnasdmnn+ahjkdshadjhajshdjkahdjkhasjdhajkhdskasjhdajskhdasjkhdjkashdjkashdksjahdasdnnmsmdhjsahdjhasjhdjksahdjkahdjskahdjkashdjakshdjkashcmnczmnxzmsjadkhasdhasjgafhjgeuwyrwiuwqwyqiryqwiuryqwsabdmnbsabdasnbdamdsbsjahdksjahdjaskhdksahdhsadhsadjasgdjhgjasdgashjgdhasjdgjashgdasjdgjagsjsgdjhgasjgdhjasgdajhgdhjsagdasjhgdajsgdjhgasdjasgdajsdgjajsgdjgasdjgasjasgdhsjdagasjdgasjdgasjgsazxcnzbcnbnmxczncxnbcbncbnznbznmbuwquyewquiweuiqqwuifbjsdbsbhbdhfjhfbsdjjfjasdkjassdhjkfkjsfjkafbskjbfsjbfsbbcsacjbksajcsbbcsbcsbkjcbjksaabjkscbcjsbcsjsbjbjcsbjskckjascscbasjccsbjabscjkbcsjsjbksbcajkbcsbsjbcsjjbsckjskcbjsbcjbscajbcscjbsjcbsjbcacjbjbsabjscbjsjbcsjbcsbjscjbcsbjcsjbcsjbscbjcsjbcsbjscbjcsbjsajcjcjbajsjhwuhkashdsajkhdjhjshakhdhhjkasdhajshajskhjakhashaskhashsskashskshskjdhjaskahdjdkahsdjksahsjakhjkasdhasjkdahdkashsajhsdjsdhsdjwiequwiuwoqieuowquqweweoiwuwe... Retained heap size:2344
The situation gets worse if jmx-monitoring is enabled as jmx starts producing metrics against each of these URI
Workaround
- Block /label/ endpoint at proxy