doc: add huggingface shared data table schema to admin/data_tables

Documents the 7-column `huggingface` data table schema, column
conventions, controlled vocabularies for pipeline_tag/domain, XML
filter patterns, and an example .loc entry.

Announced in galaxyproject/galaxy-hub#3923.
This commit is contained in:
Arash
2026-04-17 12:11:25 +02:00
parent 374869d4ba
commit 543975205e
+115
View File
@@ -103,3 +103,118 @@ When a new tool is installed that uses a data table a new entry is added to
subdirectory in `tool_data_path` (in a subdirectory that has the name of the
toolshed). By default this is `tool-data/toolshed.g2.bx.psu.edu/`. Note that
these directories will also contain tool data table config files, but they are unused.
## The `huggingface` shared data table
Galaxy tools that consume pre-downloaded Hugging Face models share a single
data table named `huggingface`. Using one shared table means admins maintain
one `.loc` file and all tools benefit from every registered model entry.
### Declaring the table
Add the following block to `tool_data_table_conf.xml`:
```xml
<!-- Hugging Face models -->
<table name="huggingface" comment_char="#" allow_duplicate_entries="False">
<columns>value, name, pipeline_tag, domain, free_tag, version, path</columns>
<file path="/opt/galaxy/tool-data/huggingface.loc" />
</table>
```
Each tool ships a `tool-data/huggingface.loc.sample` that uses the same
7-column layout.
### Column reference
| # | Column | Purpose |
|---|--------|---------|
| 0 | `value` | Unique row ID across the whole table |
| 1 | `name` | Human-readable label shown in the Galaxy select widget |
| 2 | `pipeline_tag` | Model role — see controlled vocabulary below |
| 3 | `domain` | Coarse data domain — see controlled vocabulary below |
| 4 | `free_tag` | Optional narrowing tag; fallback filter when `pipeline_tag`/`domain` alone are not specific enough |
| 5 | `version` | Model version string |
| 6 | `path` | Path to the model data, a directory or a specific file, depending on the model structure |
**`value` (column 0)**
Must be globally unique across every row in `huggingface.loc`, regardless of
which tool added it. Use the Hugging Face model ID (`<owner>/<model-name>`)
directly — it is stable and unambiguous. If the same model is registered at
more than one version, append the version:
```
black-forest-labs/FLUX.1-dev
black-forest-labs/FLUX.1-dev_2
```
**`pipeline_tag` (column 2)**
Use the official [Hugging Face pipeline tag](https://huggingface.co/models).
Common values:
| Value | When to use |
|-------|-------------|
| `text-to-image` | Image generation models |
| `automatic-speech-recognition` | ASR / transcription models |
| `feature-extraction` | Sentence / document embedding models |
| `tabular-classification` | Tabular ML classifiers |
| `tabular-regression` | Tabular ML regressors |
| `text-generation` | Causal / instruction-tuned LLMs |
Do not invent synonyms for existing Hugging Face tags.
**`domain` (column 3)**
A broad category for the data type the model works with:
`image` · `text` · `audio` · `tabular` · `sequence` · `video` · `multimodal`
**`free_tag` (column 4)**
An optional short identifier used as a fallback narrowing filter when
`pipeline_tag` and `domain` alone are not specific enough. Because a model
can be consumed by multiple tools, `free_tag` must not encode a specific tool
name. Choose a short, lowercase, descriptive value and document it alongside
the tool that introduces it.
**`version` (column 5)**
The model version string. A tool declares in its XML which version(s) it
accepts, allowing multiple versions of the same model to coexist. Where
possible, rows are only added, never removed or edited.
**`path` (column 6)**
The path to the model data on the production server (maintained by admins).
Can be a directory (when the tool reads the whole Hugging Face cache layout)
or a specific file (e.g. a `.ckpt` checkpoint).
### XML filter convention
Filter primarily by `pipeline_tag` (column 2) and/or `domain` (column 3) so
only relevant model types are shown to the user. Add a `version` or
`free_tag` filter only when you need to narrow the selection further:
```xml
<param name="model" type="select" label="Model">
<options from_data_table="huggingface">
<filter type="static_value" column="2" value="<pipeline_tag>"/>
<filter type="static_value" column="3" value="<domain>"/>
<!-- optional: narrow further by version or free_tag -->
<!-- <filter type="static_value" column="5" value="<version>"/> -->
<!-- <filter type="static_value" column="4" value="<free_tag>"/> -->
</options>
</param>
```
### Example `.loc` entry
Each row is TAB-separated (7 columns):
```
# Columns: value <TAB> name <TAB> pipeline_tag <TAB> domain <TAB> free_tag <TAB> version <TAB> path
#
# Flux
black-forest-labs/FLUX.1-dev FLUX.1 [dev] text-to-image image flux 1 /data/hf_models
```