Ability to remove "forgot password" link with external user management off

We don't want the "forgot password" link on the login page to be available. I know the link goes away if you turn on external user management, but we are using LDAP and it says not to turn it on if you're using LDAP.