Allow truly on-demand runners and make the new APIs public

The introduction of the runner autoscaler for Kubernetes simplies automating runners, however it has the limitation that the runner needs to run at all times for the pipelines to work. What would be great is truly on-demand runners, with the ability to scale up as required and scale back to 0 when not required. As these are now running on K8S clusters, it makes the task of spinning up a container to execute steps/one task and then deleting it relatively straightforward.

Currently the runner listens to a websocket and is notified of queued steps before it then fetches the step definitions to execute whilst also updating the state of the step, If the autoscaler instead listened to the websocket, it could spawn containers to run a specific step (or steps) - the runner itself would need to be modified to allow the passing of parameters, including the step, etc The autoscaler would register and make the runner available in the frontend and when a new task is queued, it could start a copy of the runner to perform the execution.

It would be beneficial from an efficiency point of view to pass all the "low level" steps/tasks that correspond to a single step element in the pipeline.yml rather than executing them individually

Autoscaler makes use of undocumented APIs for Bitbucket Cloud (to manage runners), it would be good if these could be publicly documented to allow custom versions of the autoscaler. Similarly, would it be possible to publicly publish details of the websocket(s) and the internal APIs used by the runner (step status update, etc) this would potentially allow developers to customise the runner or autoscaler further.