We need to filter out these duplicates from the results or remove these old accounts from the system.

If you go with removing them, you have to make sure you remove preference information tied to those users. Further, you will need to move that preference information over the new external entity account that will substitute for the old account.