LDAP connections over SSL are not pooled, meaning each operation acquires a new TCP connection as well as negotiating SSL. This significantly increases the time taken for operations like synchronisation.
For a default Crowd with no pooling configuration changes, synchronisation is single threaded, so only a single connection is made. (In testing, two connections; it's not clear if that's due to multiple threads or a single connection only being reused so many times.)
In terms of impact, a synchronisation with OpenLDAP that took 1m45 without SSL took as much as 8m15 when SSL was enabled.
Fixing this may require CWD-2790.