It is getting more and more important to process background jobs on time (language pack generation, Gengo translations import, import language packs and products).

Recently we run into issue that Gengo job was blocked and skipped for 2 days. As a result Stash team did not get translations on time and had to perform manual import.
We do not want

Some questions:

• Should we prioritize jobs? E.g should Gengo job be more important than 'stats generation' job?
  I think that we should not. 'stats generation' job is probably less important (as the only one) but it runs only once a day. Other jobs like 'product import' and 'language pack import' can be very important depending on circumstances.
• What if new jobs appear in the queue faster than system is able to execute them?
• Is there any library which combines benefits of Quartz scheduler with the profit of our locking mechanism (exclusive runs) and queuing?
• Maybe it could be great to manually decide which jobs is most important at the moment? Like shifting the queue from TAC admin UI (similar to what Bamboo offers its plans)
• Maybe we can get rid of locking mechanism or improve it so jobs are locked only if really necessary (clash with other mutating job)??? Maybe proper handling of optimistic locking in the code will solve the problem?