Issue Summary

When adding a custom website connector to Rovo, if the site contains a link that appends to the URL infinitely, Rovo will continue to index in a continuous loop

https://support.atlassian.com/organization-administration/docs/connect-custom-website-to-rovo/

Steps to Reproduce

1. Create a website and add content to the site
2. Add a link on one of the pages that links to the same page and appends some string to the page (such as /wiki/123/)

Expected Results

Rovo stops indexing the infinite looping URLs after a certain depth

Actual Results

Rovo continuously follows the URL, resulting in a (near) infinite loop and thousands of indexed objects  

For example: https://testsite.atlassian.com/wiki/homepage/wiki/123/homepage/wiki/123/homepage/wiki/123/homepage/wiki/123/homepage/wiki/123/homepage/wiki/123

Workaround

Update the site to not include links to URLs that append infinitely