Switching branch does not update submodules

Let's say branch A and branch B. In branch A there's no submodule, and in branch B there is one submodule. It doesn't matter where it is or what it's called.

Switching to branch B (with submodule) updates it and all's fine and dandy.

Switching back to branch A (does NOT contain that submodule), the submodule does not get deleted from the filesystem. It's as if it's always been there.

If I then delete the submodule manually, ST does not detect a change, which is correct. So why does ST not delete a submodule that isn't supposed to be there anymore?