mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-10-28 02:58:18 +08:00
API design guidelines.
This commit is contained in:
John Chilton
@@ -1,70 +0,0 @@
|
||||
API Design Guidelines
|
||||
=====================
|
||||
|
||||
The following section outlines guidelines related to extending and/or modifying
|
||||
the Galaxy API. The Galaxy API has grown in an ad-hoc fashion over time by
|
||||
many contributors and so clients SHOULD NOT expect the API will conform to
|
||||
these guidelines - but developers contributing to the Galaxy API SHOULD follow
|
||||
these guidelines.
|
||||
|
||||
- API functionality should include docstring documentation for consumption
|
||||
at docs.galaxyproject.org.
|
||||
- Developers should familiarize themselves with the HTTP status code definitions
|
||||
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. The API responses
|
||||
should properly set the status code according to the result - in particular
|
||||
2XX responses should be used for successful requests, 4XX for various
|
||||
kinds of client errors, and 5XX for the errors on the server side.
|
||||
- If there is an error processing some part of the request (one item in a list
|
||||
for instance), the status code should be set to reflect the error and the
|
||||
partial result may or may not be returned depending on the controller -
|
||||
this behavior should be documented.
|
||||
- API methods should throw a finite number of exceptions
|
||||
(defined in :mod:`galaxy.exceptions`) and these should subclass
|
||||
`MessageException` and not paste/wsgi HTTP exceptions. When possible,
|
||||
the framework itself should be responsible for catching these exceptions,
|
||||
setting the status code, and building an error response.
|
||||
- Error responses should not consist of plain text strings - they should be
|
||||
dictionaries describing the error and containing the following::
|
||||
|
||||
{
|
||||
"status_code": 400,
|
||||
"err_code": 400007,
|
||||
"err_msg": "Request contained invalid parameter, action could not be completed.",
|
||||
"type": "error",
|
||||
"extra_error_info": "Extra information."
|
||||
}
|
||||
|
||||
Various error conditions (once a format has been chosen and framework to
|
||||
enforce it in place) should be spelled out in this document.
|
||||
- Backward compatibility is important and should be maintained when possible.
|
||||
If changing behavior in a non-backward compatible way please ensure one
|
||||
of the following holds - there is a strong reason to believe no consumers
|
||||
depend on a behavior, the behavior is effectively broken, or the API
|
||||
method being modified has not been part of a tagged dist release.
|
||||
|
||||
The following bullet points represent good practices more than guidelines, please
|
||||
consider them when modifying the API.
|
||||
|
||||
- Functionality should not be copied and pasted between controllers -
|
||||
consider refactoring functionality into associated classes or short of
|
||||
that into Mixins (http://en.wikipedia.org/wiki/Composition_over_inheritance)
|
||||
or into Managers (:mod:`galaxy.managers`).
|
||||
- API additions are more permanent changes to Galaxy than many other potential
|
||||
changes and so a second opinion on API changes should be sought.
|
||||
- New API functionality should include functional tests. These functional
|
||||
tests should be implemented in Python and placed in
|
||||
`lib/galaxy_test/api`.
|
||||
- Changes to reflect modifications to the API should be pushed upstream to
|
||||
the BioBlend project if possible.
|
||||
|
||||
Longer term goals/notes.
|
||||
|
||||
- It would be advantageous to have a clearer separation of anonymous and
|
||||
admin handling functionality.
|
||||
- If at some point in the future, functionality needs to be added that
|
||||
breaks backward compatibility in a significant way to a component used by
|
||||
the community - a "dev" variant of the API will be established and
|
||||
the community should be alerted and given a timeframe for when the old
|
||||
behavior will be replaced with the new behavior.
|
||||
- Consistent standards for range-based requests, batch requests, filtered
|
||||
requests, etc... should be established and documented here.
|
||||
@@ -1,186 +1,6 @@
|
||||
Quickstart
|
||||
==========
|
||||
|
||||
Log in as your user, navigate to the API Keys page in the User menu, and
|
||||
generate a new API key. Make a note of the API key, and then pull up a
|
||||
terminal. Now we'll use the display.py script in your ``galaxy/scripts/api``
|
||||
directory for a short example, which assumes your local galaxy server is
|
||||
running on the default port 8080::
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/histories
|
||||
Collection Members
|
||||
------------------
|
||||
#1: /api/histories/8c49be448cfe29bc
|
||||
name: Unnamed history
|
||||
id: 8c49be448cfe29bc
|
||||
#2: /api/histories/33b43b4e7093c91f
|
||||
name: output test
|
||||
id: 33b43b4e7093c91f
|
||||
|
||||
The result is a Collection of the histories of the user specified by the API
|
||||
key (you). To look at the details of a particular history, say #1 above, do
|
||||
the following::
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/histories/8c49be448cfe29bc
|
||||
Member Information
|
||||
------------------
|
||||
state_details: {'ok': 1, 'failed_metadata': 0, 'upload': 0, 'discarded': 0, 'running': 0, 'setting_metadata': 0, 'error': 0, 'new': 0, 'queued': 0, 'empty': 0}
|
||||
state: ok
|
||||
contents_url: /api/histories/8c49be448cfe29bc/contents
|
||||
id: 8c49be448cfe29bc
|
||||
name: Unnamed history
|
||||
|
||||
This gives detailed information about the specific member in question, in this
|
||||
case the History. To view history contents, do the following::
|
||||
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/histories/8c49be448cfe29bc/contents
|
||||
Collection Members
|
||||
------------------
|
||||
#1: /api/histories/8c49be448cfe29bc/contents/6f91353f3eb0fa4a
|
||||
name: Pasted Entry
|
||||
type: file
|
||||
id: 6f91353f3eb0fa4a
|
||||
|
||||
What we have here is another Collection of items containing all of the datasets
|
||||
in this particular history. Finally, to view details of a particular dataset
|
||||
in this collection, execute the following::
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/histories/8c49be448cfe29bc/contents/6f91353f3eb0fa4a
|
||||
Member Information
|
||||
------------------
|
||||
misc_blurb: 1 line
|
||||
name: Pasted Entry
|
||||
data_type: txt
|
||||
deleted: False
|
||||
file_name: /Users/yoplait/work/galaxy-stock/database/files/000/dataset_82.dat
|
||||
state: ok
|
||||
download_url: /datasets/6f91353f3eb0fa4a/display?to_ext=txt
|
||||
visible: True
|
||||
genome_build: ?
|
||||
model_class: HistoryDatasetAssociation
|
||||
file_size: 17
|
||||
metadata_data_lines: 1
|
||||
id: 6f91353f3eb0fa4a
|
||||
misc_info: uploaded txt file
|
||||
metadata_dbkey: ?
|
||||
|
||||
And now you've successfully used the API to request and select a history,
|
||||
browse the contents of that history, and then look at detailed information
|
||||
about a particular dataset.
|
||||
|
||||
For a more comprehensive Data Library example, set the following options in your
|
||||
galaxy.yml as well, and restart galaxy again::
|
||||
|
||||
admin_users: you@example.org
|
||||
library_import_dir: /path/to/some/directory
|
||||
|
||||
In the directory you specified for 'library_import_dir', create some
|
||||
subdirectories, and put (or symlink) files to import into Galaxy into those
|
||||
subdirectories.
|
||||
|
||||
In Galaxy, create an account that matches the address you put in 'admin_users',
|
||||
then browse to that user's preferences and generate a new API Key. Copy the
|
||||
key to your clipboard and then use these scripts::
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries
|
||||
Collection Members
|
||||
------------------
|
||||
|
||||
0 elements in collection
|
||||
|
||||
% ./library_create_library.py my_key http://localhost:8080/api/libraries api_test 'API Test Library'
|
||||
Response
|
||||
--------
|
||||
/api/libraries/f3f73e481f432006
|
||||
name: api_test
|
||||
id: f3f73e481f432006
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries
|
||||
Collection Members
|
||||
------------------
|
||||
/api/libraries/f3f73e481f432006
|
||||
name: api_test
|
||||
id: f3f73e481f432006
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries/f3f73e481f432006
|
||||
Member Information
|
||||
------------------
|
||||
synopsis: None
|
||||
contents_url: /api/libraries/f3f73e481f432006/contents
|
||||
description: API Test Library
|
||||
name: api_test
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries/f3f73e481f432006/contents
|
||||
Collection Members
|
||||
------------------
|
||||
/api/libraries/f3f73e481f432006/contents/28202595c0d2591f61ddda595d2c3670
|
||||
name: /
|
||||
type: folder
|
||||
id: 28202595c0d2591f61ddda595d2c3670
|
||||
|
||||
% ./library_create_folder.py my_key http://localhost:8080/api/libraries/f3f73e481f432006/contents 28202595c0d2591f61ddda595d2c3670 api_test_folder1 'API Test Folder 1'
|
||||
Response
|
||||
--------
|
||||
/api/libraries/f3f73e481f432006/contents/28202595c0d2591fa4f9089d2303fd89
|
||||
name: api_test_folder1
|
||||
id: 28202595c0d2591fa4f9089d2303fd89
|
||||
|
||||
% ./library_upload_from_import_dir.py my_key http://localhost:8080/api/libraries/f3f73e481f432006/contents 28202595c0d2591fa4f9089d2303fd89 bed bed hg19
|
||||
Response
|
||||
--------
|
||||
/api/libraries/f3f73e481f432006/contents/e9ef7fdb2db87d7b
|
||||
name: 2.bed
|
||||
id: e9ef7fdb2db87d7b
|
||||
/api/libraries/f3f73e481f432006/contents/3b7f6a31f80a5018
|
||||
name: 3.bed
|
||||
id: 3b7f6a31f80a5018
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries/f3f73e481f432006/contents
|
||||
Collection Members
|
||||
------------------
|
||||
/api/libraries/f3f73e481f432006/contents/28202595c0d2591f61ddda595d2c3670
|
||||
name: /
|
||||
type: folder
|
||||
id: 28202595c0d2591f61ddda595d2c3670
|
||||
/api/libraries/f3f73e481f432006/contents/28202595c0d2591fa4f9089d2303fd89
|
||||
name: /api_test_folder1
|
||||
type: folder
|
||||
id: 28202595c0d2591fa4f9089d2303fd89
|
||||
/api/libraries/f3f73e481f432006/contents/e9ef7fdb2db87d7b
|
||||
name: /api_test_folder1/2.bed
|
||||
type: file
|
||||
id: e9ef7fdb2db87d7b
|
||||
/api/libraries/f3f73e481f432006/contents/3b7f6a31f80a5018
|
||||
name: /api_test_folder1/3.bed
|
||||
type: file
|
||||
id: 3b7f6a31f80a5018
|
||||
|
||||
% ./display.py my_key http://localhost:8080/api/libraries/f3f73e481f432006/contents/e9ef7fdb2db87d7b
|
||||
Member Information
|
||||
------------------
|
||||
misc_blurb: 68 regions
|
||||
metadata_endCol: 3
|
||||
data_type: bed
|
||||
metadata_columns: 6
|
||||
metadata_nameCol: 4
|
||||
uploaded_by: nate@...
|
||||
metadata_strandCol: 6
|
||||
name: 2.bed
|
||||
genome_build: hg19
|
||||
metadata_comment_lines: None
|
||||
metadata_startCol: 2
|
||||
metadata_chromCol: 1
|
||||
file_size: 4272
|
||||
metadata_data_lines: 68
|
||||
message:
|
||||
metadata_dbkey: hg19
|
||||
misc_info: uploaded bed file
|
||||
date_uploaded: 2010-06-22T17:01:51.266119
|
||||
metadata_column_types: str, int, int, str, int, str
|
||||
|
||||
Other parameters are valid when uploading, they are the same parameters as are
|
||||
used in the web form, like 'link_data_only' and etc.
|
||||
|
||||
The request and response format should be considered alpha and are subject to change.
|
||||
|
||||
The Galaxy Training Network material has a tutorial designed
|
||||
to teach using the Galaxy API at
|
||||
https://training.galaxyproject.org/training-material/topics/dev/tutorials/bioblend-api/slides.html.
|
||||
|
||||
@@ -28,7 +28,4 @@ Overview of available API resources is available at https://galaxyproject.org/de
|
||||
|
||||
Quickstart <api/quickstart>
|
||||
|
||||
Development Guidelines <api/guidelines>
|
||||
|
||||
Documentation <api/api>
|
||||
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
API Design Guidelines
|
||||
=====================
|
||||
|
||||
The following section outlines guidelines related to extending and/or modifying
|
||||
the Galaxy API. The Galaxy API has grown in an ad-hoc fashion over time by
|
||||
many contributors and so clients SHOULD NOT expect the API will conform to
|
||||
these guidelines - but developers contributing to the Galaxy API SHOULD follow
|
||||
these guidelines.
|
||||
|
||||
Before Getting Started
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Developers should familiarize themselves with HTTP method definitions
|
||||
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9. While Galaxy
|
||||
follows these imperfectly - a serious effort should be made to use HTTP
|
||||
methods (GET, POST, PUT, DELETE, etc.) correctly and consistently. More
|
||||
information about these methods with a focus on RESTful APIs can be found
|
||||
at https://restfulapi.net/http-methods/.
|
||||
|
||||
Developers should familiarize themselves with the HTTP status code definitions
|
||||
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html. The API responses
|
||||
should properly set the status code according to the result - in particular
|
||||
2XX responses should be used for successful requests, 4XX for various
|
||||
kinds of client errors, and 5XX for the errors on the server side.
|
||||
|
||||
Developers should also familiarize themselves with RESTful API design
|
||||
https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design.
|
||||
Again Galaxy follows generic RESTful guidelines imperfectly - but a serious
|
||||
effort should be made to design the API as RESTfully and consistently as possible.
|
||||
|
||||
Documenting the API
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
An API endpoint implementation method should include high-level docstring documentation
|
||||
for consumption at docs.galaxyproject.org.
|
||||
|
||||
API functionality should consume and produce Pydantic_ `models
|
||||
<https://pydantic-docs.helpmanual.io/usage/models/>`__ with typed and annotated
|
||||
`fields <https://pydantic-docs.helpmanual.io/usage/types/>`__. This information
|
||||
will be used to generate detailed OpenAPI_ documentation for APIs in
|
||||
the near future. Typed models also improve the quality and thoroughness
|
||||
of the static checking that can be done for the Galaxy backend.
|
||||
|
||||
To further document your API and allow other developers to see how it functions,
|
||||
new API functionality should include functional tests. These functional
|
||||
tests should be implemented in Python and added to either the API test suite
|
||||
or the integration test suite. Checkout the :ref:`Writing Tests for Galaxy <./writing_tests.md>`
|
||||
documentation for more information on developing tests for Galaxy.
|
||||
|
||||
.. note::
|
||||
|
||||
The Galaxy documentation can be generated by running ``make docs-develop``
|
||||
from the Galaxy root. Galaxy can be type checked by running ``tox -e mypy``
|
||||
from the root of Galaxy after tox_ has been configured.
|
||||
|
||||
Error Handling
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
API methods should throw a finite number of exceptions
|
||||
(defined in :mod:`galaxy.exceptions`) and these should subclass
|
||||
:class:`galaxy.exceptions.MessageException` and not paste/WSGI/FastAPI HTTP exceptions.
|
||||
When possible, the framework itself should be responsible for catching these exceptions,
|
||||
setting the status code, and building an error response. Throwing these exceptions
|
||||
will ensure the API responses are consistent across frameworks and across the API.
|
||||
These exceptions allow throwers to attach arbitrary extra information.
|
||||
|
||||
Error responses should not consist of plain text strings - they should be
|
||||
dictionaries describing the error and containing the following:
|
||||
|
||||
.. code-block: json
|
||||
|
||||
{
|
||||
"status_code": 400,
|
||||
"err_code": 400007,
|
||||
"err_msg": "Request contained invalid parameter, action could not be completed.",
|
||||
"type": "error",
|
||||
"extra_error_info": "Extra information."
|
||||
}
|
||||
|
||||
The error code (``err_code``) is defined by the exception class
|
||||
and pulled from ``lig/galaxy/exceptions/error_codes.json``. This JSON
|
||||
document should be used to add new error conditions or improve
|
||||
default error messages.
|
||||
|
||||
If there is an error processing some part of the request (one item in a list
|
||||
for instance), the status code should be set to reflect the error and the
|
||||
partial result may or may not be returned depending on the controller -
|
||||
this behavior should be documented.
|
||||
|
||||
Backward Compatibility
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Backward compatibility is important and should be maintained when possible.
|
||||
If changing behavior in a non-backward compatible way please ensure one
|
||||
of the following holds - there is a strong reason to believe no consumers
|
||||
depend on a behavior, the behavior is effectively broken, or the API
|
||||
method being modified has not been part of a tagged dist release. If these
|
||||
conditions do not hold, please help the Galaxy release manager ensure
|
||||
information about the breaking API is highlighted in the Galaxy release
|
||||
notes.
|
||||
|
||||
Whenever functionality needs to be added, removed, or modified that
|
||||
breaks backward compatibility in a significant way to a component used by
|
||||
the community - a "dev" variant of the API should be established and
|
||||
the community should be alerted and given a time frame for when the old
|
||||
behavior will be replaced with the new behavior.
|
||||
|
||||
Design
|
||||
~~~~~~
|
||||
|
||||
Functionality should not be copied and pasted between controllers -
|
||||
consider refactoring functionality into associated classes or short of
|
||||
that into mixins_ or into managers (:mod:`galaxy.managers`).
|
||||
|
||||
To whatever degree is possible, the Galaxy API controller classes
|
||||
(found in :mod:`galaxy.webapps.galaxy.api`) should be as lean as
|
||||
possible. Most of the logic should be pushed into components that
|
||||
are decoupled from the actual web application and the web framework.
|
||||
The role of the controller should be simply to adapt the web request
|
||||
(tied to the web framework) to simple objects (plain Python objects,
|
||||
Pydantic_ models, or SQL Alchemy models), send these objects to a
|
||||
Python component lower in the stack, and finally to translate the
|
||||
response to a web response. This design will ensure that Galaxy
|
||||
application logic is usable in as many contexts as possible and
|
||||
decouple the core "business logic" of Galaxy from the web framework
|
||||
hosting it.
|
||||
|
||||
Going forward, we anticipate the most typical way this abstract
|
||||
advice translates to concrete code flow is that API controllers
|
||||
in :mod:`galaxy.webapps.galaxy.api` receive or produce Pydantic_
|
||||
models and pass them to manager components defined in
|
||||
:mod:`galaxy.managers`. The manager is then responsible for
|
||||
interacting with the whole backend or specific components to
|
||||
fullfil the desired request.
|
||||
|
||||
Galaxy work contexts provide a way for API requests (and other
|
||||
components) to describe what they require from the framework
|
||||
in terms of the request (examples include the Galaxy app, a user
|
||||
to act on behalf of, or a user with a defined "current history").
|
||||
These objects describing the web request are passed around rather
|
||||
haphazardly as an argument named ``trans``. To design API methods
|
||||
to deal with ``trans`` in the most statically checkable and
|
||||
rigorous way - please review the docs for :mod:`galaxy.managers.context`.
|
||||
|
||||
Examples
|
||||
~~~~~~~~
|
||||
|
||||
A very clean, isolated example of a well documented controller
|
||||
and well annotated models is the licenses endpoint -
|
||||
:mod:`galaxy.webapps.galaxy.api.licenses`. A more complicated
|
||||
endpoint that demonstrates working with a manager and the
|
||||
Galaxy model backend is the roles endpoint - :mod:`galaxy.webapps.galaxy.api.roles`.
|
||||
|
||||
BioBlend
|
||||
~~~~~~~~
|
||||
|
||||
Changes to reflect modifications to the API should be pushed upstream to
|
||||
the BioBlend_ project if possible.
|
||||
|
||||
Longer Term Goals
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- It would be advantageous to have a clearer separation of anonymous and
|
||||
admin handling functionality.
|
||||
- Consistent standards for range-based requests, batch requests, filtered
|
||||
requests, etc... should be established and documented here.
|
||||
|
||||
.. _Pydantic: https://pydantic-docs.helpmanual.io/
|
||||
.. _BioBlend: https://github.com/galaxyproject/bioblend
|
||||
.. _mixins: http://en.wikipedia.org/wiki/Composition_over_inheritance
|
||||
.. _tox: https://tox.readthedocs.io/en/latest/
|
||||
.. _OpenAPI: https://www.openapis.org
|
||||
@@ -12,6 +12,7 @@ A multi-hour long video playlist covering these slides can be found at
|
||||
:maxdepth: 1
|
||||
|
||||
schema
|
||||
api_guidelines
|
||||
interactive_environments
|
||||
build_a_job_runner
|
||||
finding_and_improving_slow_code
|
||||
|
||||
Reference in New Issue
Block a user