Unable to un-follow private repositories I no longer have access to

I was previously a member of an organization that had private repositories. I "hearted" or followed a few of those. Then I left the organization and I continue to get information regarding those repositories even though I no-longer have access to them.

When I try and un-follow them, it does not work.

I assume this is because I no longer have access I can't complete the un-follow function on those repositories

Expected behavior it seems would be that when a user is removed from the organization all private repositories belonging to that organization are un-followed by the removed member.