- Change undocumented (and probably unused) config option container_image_cache_path default to "database/contianer_cache".
- Update container resolver framework to let plugins assert they only produce containers of a single type.
- Update mulled container resolvers to indicate they produce Docker containers only - update names also.
- Refactor mulled caching stuff to allow reuse with Singularity (where the names come from is different - but a lot of the logic is the same).
- Add a cached singularity container resolver - that searches "config.container_image_cache_path/singularity/mulled" for an image with the correct name.
Next up - similarly refactor the BuildMulledDockerContainerResolver to allow reuse and implement the BuildMulledSingularityContainerResolver.
My understanding is that unlike Docker, Singularity doesn't allow a subdirectory to be mounted rw and its parent to be mounted ro - it will just be ro. This PR adjusts the default volume handling for Singularity to intelligently fallback to mounting parent directories as rw if their child directories are listed as being mounted rw.
Ping @bgruening - is the above a description correct and does this approach work for you?
See http://singularity.lbl.gov/ for more information on Singularity and see job_conf.xml.sample_advanced for information on setting up job runners to exploit singularity.
The biggest current caveat is probably that currently these container images need to be setup manually by the admin and the paths hard-coded for each tool in job_conf.xml. There are people who do such manual setups for Docker (https://github.com/phnmnl/container-galaxy-k8s-runtime/blob/develop/config/job_conf.xml) - so it wouldn't be surprising if someone wanted to set this up for singularity as well. That said I'm sure this will be followed up by magic to fetch and convert Docker containers and leverage published singularity containers (such as mulled can now produce (https://github.com/galaxyproject/galaxy-lib/pull/64).
All job configuration parameters can be set using environment variables but currently the actual loading of the plugin cannot. This is problematic for setting up single job confs for multiple environemnts (like done with docker-galaxy-stable) when using plugins that may hang waiting for resources (such as Kubernetes for the K8 runner or a MQ for Pulsar for instance) or may require library dependencies not be setup in every environment the job configuration is deployed in.
xref https://github.com/bgruening/docker-galaxy-stable/pull/328
Rebased with advice from @bgruening.
Dannon Baker