add some admin facing docs on data tables

doc/source/admin/data_tables.md

@@ -0,0 +1,93 @@
+# Tool data
+
+Galaxy stores tool data in the path defined by `tool_data_path` (by default `tool-data/`).
+It's possible to to separate tool data of shed installed tools by setting (`shed_tool_data_path`).
+
+Tool data consists of:
+
+1. the actual data
+2. a tool data table
+3. tool data config files
+
+## Tool data
+
+This is the actual data that is stored by default in `tool_data_path`. It may be favorable to store the
+actual tool data in a separate folder. For manually managed tool data this can be achieved by simply
+storing the data in another folder. For data that is added by data managers this can be achieved by
+setting `galaxy_data_manager_data_path`.
+
+## Tool data tables
+
+In order to make tool data usable from Galaxy tools so called tool data tables are used. 
+Those are tabular (by default tab separated) files with the extension `.loc`.
+Besides the actual paths the entries can contain, e.g. IDs, names, or other
+that can be used in tools to select reference data. The paths should be given as absolute paths,
+but can also be given relative to the Galaxy root dir.
+By default tool data tables are installed in `tool_data_path` (where also built-in tool data tables
+are stored). By setting `shed_tool_data_path` this can be separated.
+
+## Tool data table config
+
+The tool data tables that should be used in a Galaxy instance are configured
+using tool data table config files.  In addition these files contain some
+metadata.
+
+Tool data table config files are XML files listing tool data table configurations:
+
+```xml
+<tables>
+    ....
+</tables>
+```
+
+A tool data table configuration looks like this 
+
+```xml
+<table name="bwa_indexes_color" comment_char="#" allow_duplicate_entries="False">
+    <columns>value, dbkey, name, path</columns>
+    <file path="bwa_index_color.loc" />
+</table>
+```
+
+- `table`: `name`, `comment_char` (default `"#"`), `separator` (default `"\t"`), `allow_duplicate_entries` (default `True`), `empty_field_value` (default `""`)
+- `columns`: a comma separated list of column names
+- `file`: `path` (alternatively `url`, `from_config`)
+
+Tool data table definitions for tools installed from a toolshed have an additional
+element `tool_shed_repository` and sub-tags `tool_shed`
+`repository_name`, `repository_owner`, `installed_changeset_revision`, e.g.:
+
+```xml
+<table name="plasmidfinder_database" comment_char="#">
+    <columns>value, name, date, path</columns>
+    <file path="/home/berntm/projects/galaxy/tool-data/toolshed.g2.bx.psu.edu/repos/iuc/plasmidfinder/7075b7a5441b/plasmidfinder_database.loc.sample"/>
+    <tool_shed_repository>
+        <tool_shed>toolshed.g2.bx.psu.edu</tool_shed>
+        <repository_name>plasmidfinder</repository_name>
+        <repository_owner>iuc</repository_owner>
+        <installed_changeset_revision>7075b7a5441b</installed_changeset_revision>
+    </tool_shed_repository>
+</table>
+```
+
+The file path points to a data table (i.e. a `.loc` file) and can be given
+relative (to the `tool_data_path`) or absolute. If a given relative path does
+not exist also the base name is checked (many tools use something like
+`tool-data/xyz.loc` and store example `loc` files in a `tool-data/` directory in
+the tool repository).
+Currently also `.loc.sample` may be used in case the specified `.loc` is absent.
+
+Tool data table config files: 
+
+- `tool_data_table_config_path`: by default `tool_data_table_conf.xml` in Galaxy's `config/` directory.
+- `shed_tool_data_table_config`: by default `shed_tool_data_table_conf.xml` in
+Galaxy's `config/` directory. This file lists all tool data tables of tools
+installed from a toolshed. Note that the entries are versioned, i.e. there is a
+separate entry for each tool and tool version. These content of the tool data
+tables are merged when they are loaded.
+
+When a new tool is installed that uses a data table a new entry is added to
+`shed_tool_data_table_config` and a `.loc` file is placed in a versioned
+subdirectory in `tool_data_path` (in a subdirectory that has the name of the
+toolshed). By default thus is `tool-data/toolshed.g2.bx.psu.edu/`. Note that
+these directories will also contain tool data config files, but they are unused.

doc/source/admin/index.rst

@@ -20,6 +20,7 @@ This documentation is in the midst of being ported and unified based on resource
    job_metrics
    authentication
    tool_panel
+   data_tables
    mq
    dependency_resolvers
    container_resolvers