According to this documentation, only one role-based notification can be setup for each member role. When an existing role-based notification is mapped to non-admin roles, this will prevent the creation of a new notification for custom roles as those fall under the non-admin category.

Currently, there's a pop-up error message that doesn't show a lot of information about the root cause:

We couldn’t save the role-based notification
Try again later.

It'd be good to have a clearer error message that states something like, "The member role you selected already has a role-based notification set up. Select another one."