Stopping a pipeline on self hosted runners can cause steps that are yet to be assigned a runner to get stuck

Issue Summary

When stopping a pipeline whose steps run on self hosted runners, if those steps have not yet started, and are in the process of being assigned to a runner, there is the possibility of a race condition where steps that have already been stopped get assigned to a runner.

This causes the runner to get stuck, and cannot be unassigned from the step.

Steps to Reproduce

1. Start a pipeline that runs on self.hosted runners.
2. Stop the pipeline whilst its steps are still in fight and waiting to be picked up by a runnner.

Expected Results

The pipeline and its steps are stopped, without causing runners to break.

Actual Results

The self hosted runner ends up assigned to a step which can not be completed, as its already been stopped.

Workaround

The stuck runner can be deleted and re-created in the pipelines UI.