mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-09-24 16:30:27 +08:00
doc: add huggingface shared data table schema to admin/data_tables
Documents the 7-column `huggingface` data table schema, column conventions, controlled vocabularies for pipeline_tag/domain, XML filter patterns, and an example .loc entry. Announced in galaxyproject/galaxy-hub#3923.
This commit is contained in:
@@ -103,3 +103,118 @@ When a new tool is installed that uses a data table a new entry is added to
|
||||
subdirectory in `tool_data_path` (in a subdirectory that has the name of the
|
||||
toolshed). By default this is `tool-data/toolshed.g2.bx.psu.edu/`. Note that
|
||||
these directories will also contain tool data table config files, but they are unused.
|
||||
|
||||
## The `huggingface` shared data table
|
||||
|
||||
Galaxy tools that consume pre-downloaded Hugging Face models share a single
|
||||
data table named `huggingface`. Using one shared table means admins maintain
|
||||
one `.loc` file and all tools benefit from every registered model entry.
|
||||
|
||||
### Declaring the table
|
||||
|
||||
Add the following block to `tool_data_table_conf.xml`:
|
||||
|
||||
```xml
|
||||
<!-- Hugging Face models -->
|
||||
<table name="huggingface" comment_char="#" allow_duplicate_entries="False">
|
||||
<columns>value, name, pipeline_tag, domain, free_tag, version, path</columns>
|
||||
<file path="/opt/galaxy/tool-data/huggingface.loc" />
|
||||
</table>
|
||||
```
|
||||
|
||||
Each tool ships a `tool-data/huggingface.loc.sample` that uses the same
|
||||
7-column layout.
|
||||
|
||||
### Column reference
|
||||
|
||||
| # | Column | Purpose |
|
||||
|---|--------|---------|
|
||||
| 0 | `value` | Unique row ID across the whole table |
|
||||
| 1 | `name` | Human-readable label shown in the Galaxy select widget |
|
||||
| 2 | `pipeline_tag` | Model role — see controlled vocabulary below |
|
||||
| 3 | `domain` | Coarse data domain — see controlled vocabulary below |
|
||||
| 4 | `free_tag` | Optional narrowing tag; fallback filter when `pipeline_tag`/`domain` alone are not specific enough |
|
||||
| 5 | `version` | Model version string |
|
||||
| 6 | `path` | Path to the model data, a directory or a specific file, depending on the model structure |
|
||||
|
||||
**`value` (column 0)**
|
||||
|
||||
Must be globally unique across every row in `huggingface.loc`, regardless of
|
||||
which tool added it. Use the Hugging Face model ID (`<owner>/<model-name>`)
|
||||
directly — it is stable and unambiguous. If the same model is registered at
|
||||
more than one version, append the version:
|
||||
|
||||
```
|
||||
black-forest-labs/FLUX.1-dev
|
||||
black-forest-labs/FLUX.1-dev_2
|
||||
```
|
||||
|
||||
**`pipeline_tag` (column 2)**
|
||||
|
||||
Use the official [Hugging Face pipeline tag](https://huggingface.co/models).
|
||||
Common values:
|
||||
|
||||
| Value | When to use |
|
||||
|-------|-------------|
|
||||
| `text-to-image` | Image generation models |
|
||||
| `automatic-speech-recognition` | ASR / transcription models |
|
||||
| `feature-extraction` | Sentence / document embedding models |
|
||||
| `tabular-classification` | Tabular ML classifiers |
|
||||
| `tabular-regression` | Tabular ML regressors |
|
||||
| `text-generation` | Causal / instruction-tuned LLMs |
|
||||
|
||||
Do not invent synonyms for existing Hugging Face tags.
|
||||
|
||||
**`domain` (column 3)**
|
||||
|
||||
A broad category for the data type the model works with:
|
||||
`image` · `text` · `audio` · `tabular` · `sequence` · `video` · `multimodal`
|
||||
|
||||
**`free_tag` (column 4)**
|
||||
|
||||
An optional short identifier used as a fallback narrowing filter when
|
||||
`pipeline_tag` and `domain` alone are not specific enough. Because a model
|
||||
can be consumed by multiple tools, `free_tag` must not encode a specific tool
|
||||
name. Choose a short, lowercase, descriptive value and document it alongside
|
||||
the tool that introduces it.
|
||||
|
||||
**`version` (column 5)**
|
||||
|
||||
The model version string. A tool declares in its XML which version(s) it
|
||||
accepts, allowing multiple versions of the same model to coexist. Where
|
||||
possible, rows are only added, never removed or edited.
|
||||
|
||||
**`path` (column 6)**
|
||||
|
||||
The path to the model data on the production server (maintained by admins).
|
||||
Can be a directory (when the tool reads the whole Hugging Face cache layout)
|
||||
or a specific file (e.g. a `.ckpt` checkpoint).
|
||||
|
||||
### XML filter convention
|
||||
|
||||
Filter primarily by `pipeline_tag` (column 2) and/or `domain` (column 3) so
|
||||
only relevant model types are shown to the user. Add a `version` or
|
||||
`free_tag` filter only when you need to narrow the selection further:
|
||||
|
||||
```xml
|
||||
<param name="model" type="select" label="Model">
|
||||
<options from_data_table="huggingface">
|
||||
<filter type="static_value" column="2" value="<pipeline_tag>"/>
|
||||
<filter type="static_value" column="3" value="<domain>"/>
|
||||
<!-- optional: narrow further by version or free_tag -->
|
||||
<!-- <filter type="static_value" column="5" value="<version>"/> -->
|
||||
<!-- <filter type="static_value" column="4" value="<free_tag>"/> -->
|
||||
</options>
|
||||
</param>
|
||||
```
|
||||
|
||||
### Example `.loc` entry
|
||||
|
||||
Each row is TAB-separated (7 columns):
|
||||
|
||||
```
|
||||
# Columns: value <TAB> name <TAB> pipeline_tag <TAB> domain <TAB> free_tag <TAB> version <TAB> path
|
||||
#
|
||||
# Flux
|
||||
black-forest-labs/FLUX.1-dev FLUX.1 [dev] text-to-image image flux 1 /data/hf_models
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user