I recently ran into an issue where I needed to limit everyone but myself, and 1 other person to add customers to a project.
A quick inquiry to the community revealed that there is no way to constrain those permissions to that degree.
My suggestion/request: please add a permission for "who can invite customers" to a project. That way I could add individual, groups, or roles to it.

Thanks,
R