Disabled local agent still builds jobs

I've set up a clean 3.1.2 instance and disabled the default agent. Still, it happily builds all the jobs that I queue.