Memory leak while accessing <base-url>label/<labelname> (label search) on objects created in io.micrometer.core.instrument.ImmutableTag

XMLWordPrintable

    • 20
    • Severity 2 - Major
    • 16

      Issue Summary

      Memory leak while accessing <base-url>label/<labelname> (label search) on objects created in io.micrometer.core.instrument.ImmutableTag

      This is reproducible on the Data Center: yes

      Steps to Reproduce

      1. Use the following script to search randomly for labels
        while :
        do
        	curl http://127.0.0.1:8090/label/cat+dog+$RANDOM
        
        done
        

      1. Capture heap dump after some time after stopping the script
      2. Run following OQL in MAT
        SELECT value.toString() FROM "io.micrometer.core.instrument.ImmutableTag.*" WHERE (key.toString() = "pathPattern")
        
      1. We observe the following objects in the heap. These objects are not cleared until the node is restarted

      Expected Results

      Objects should be collected by the Garbage Collector.

      Actual Results

      The objects remain in the heap till the node is restarted. 

      Note: Authentication is not required to exploit this method. Having an SSO doesn't seem to make any difference

      This can be used in a denial-of-service attack although it takes a lot of time to fill the heap with the above method. Using longer URLs can fill up the heap more quickly. I was able to see even the following URI put in the heap(the upper limit may be governed by the size of request header tomcat accepts  )

      label.atom+cat+dog+18259+1052+54+mouse+24913+sbdabdjksandnsadnbsadabdmnbandsbdnbasnmdbanmbdsanmbdnsabdnasmbdabdmabdmnbsadbdmasbdmsanadmnsadsanmdbasdnmasbdnmasdbasnmdbsanmdbanmbdmnasdmnn+ahjkdshadjhajshdjkahdjkhasjdhajkhdskasjhdajskhdasjkhdjkashdjkashdksjahdasdnnmsmdhjsahdjhasjhdjksahdjkahdjskahdjkashdjakshdjkashcmnczmnxzmsjadkhasdhasjgafhjgeuwyrwiuwqwyqiryqwiuryqwsabdmnbsabdasnbdamdsbsjahdksjahdjaskhdksahdhsadhsadjasgdjhgjasdgashjgdhasjdgjashgdasjdgjagsjsgdjhgasjgdhjasgdajhgdhjsagdasjhgdajsgdjhgasdjasgdajsdgjajsgdjgasdjgasjasgdhsjdagasjdgasjdgasjgsazxcnzbcnbnmxczncxnbcbncbnznbznmbuwquyewquiweuiqqwuifbjsdbsbhbdhfjhfbsdjjfjasdkjassdhjkfkjsfjkafbskjbfsjbfsbbcsacjbksajcsbbcsbcsbkjcbjksaabjkscbcjsbcsjsbjbjcsbjskckjascscbasjccsbjabscjkbcsjsjbksbcajkbcsbsjbcsjjbsckjskcbjsbcjbscajbcscjbsjcbsjbcacjbjbsabjscbjsjbcsjbcsbjscjbcsbjcsjbcsjbscbjcsjbcsbjscbjcsbjsajcjcjbajsjhwuhkashdsajkhdjhjshakhdhhjkasdhajshajskhjakhashaskhashsskashskshskjdhjaskahdjdkahsdjksahsjakhjkasdhasjkdahdkashsajhsdjsdhsdjwiequwiuwoqieuowquqweweoiwuwe...
      
      Retained heap size:2344
      

      The situation gets worse if jmx-monitoring is enabled as jmx starts producing metrics against each of these URI

      Workaround

      • Block /label/ endpoint at proxy

        1. Screenshot 2024-06-13 at 11.35.06 PM.png
          359 kB
          Ashish Kotha
        2. Screenshot 2024-06-13 at 11.53.19 PM.png
          297 kB
          Ashish Kotha
        3. Screenshot 2024-06-14 at 8.21.49 AM.png
          513 kB
          Ashish Kotha

              Assignee:
              Anshul Chokhani (Inactive)
              Reporter:
              Ashish Kotha
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

                Created:
                Updated:
                Resolved: