opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-10-28 02:58:18 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updates to help text for Human Genome Variation tools and formatHelp.html

Browse Source
This commit is contained in:
Richard Burhans
2010-09-28 12:38:52 -04:00
parent e1876ef767
commit 0cd1db87a2
15 changed files with 568 additions and 405 deletions
Download Patch File Download Diff File Expand all files Collapse all files
static/formatHelp.html
+205 -137
View File
@@ -1,51 +1,68 @@
<html>
<head><title>Galaxy data formats</title>
<head><title>Galaxy Data Formats</title>
</head>
<body>
<h2>Galaxy data formats</h2>
<h2>Galaxy Data Formats</h2>
<p>
For problems with missing queries in the tool selection boxes the most common
reason is the tool only lists history items with data formats compatible with the
tool. Some formats are subsets of others and Galaxy should also list those
with compatible subformats as well. If the query is not showing up still and you
believe it is in the correct format you can click on the pencil icon and
manually change the format. This will not edit the file just change the
metadata for the file. Some cases you will need to actually change the
file format. For example, if the file is space delimited and a
tabular file is required; then the "Convert delimiters to TAB" tool under "Text
Manipulation" can be used to reformat the file.
<br>
<h3>Dataset missing?</h3>
<p>
If you have a dataset in your history that is not appearing in the
drop-down selector for a tool, the most common reason is that it has
the wrong format. Each Galaxy dataset has an associated file format
recorded in its metadata, and tools will only list datasets from your
history that have a format compatible with that particular tool. Of
course some of these datasets might not actually contain relevant
data, or even the correct columns needed by the tool, but filtering
by format at least makes the list to select from a bit shorter.
<p>
Some of the formats are defined hierarchically, going from very
general ones like <a href="#tab">tabular</a> (which includes any text
file with tab-separated columns), to more restrictive sub-formats
like <a href="#interval">interval</a> (where three of the columns
must be the chromosome, start position, and end position), and on
to even more specific ones such as <a href="#bed">BED</a> or
<a href="#gff">GFF</a> that have additional requirements. So for
example if a tool's required input format is tabular, then all of
your history items whose format is recorded as tabular will be
listed, along with those in all sub-formats that also qualify as
tabular (interval, BED, GFF, etc.).
<p>
There are two usual methods for changing a dataset's format in
Galaxy: if the file contents are already in the required format but
the metadata is wrong (perhaps because the Auto-detect feature of the
Upload File tool guessed it incorrectly), you can fix the metadata
manually by clicking on the pencil icon beside that dataset in your
history. Or, if the file contents really are in a different format,
Galaxy provides a number of format conversion tools (e.g. in the
Text Manipulation and Convert Formats categories). For instance,
if the tool you want to run requires tabular but your columns are
delimited by spaces or commas, you can use the "Convert delimiters
to TAB" tool under Text Manipulation to reformat your data. However
if your files are in a completely unsupported format, then you need
to convert them yourself before uploading.
<p>
Some of the most commonly used formats are very similar. Start with the
basic tabular file. It has few requirements other than 1 or more columns of
data separated by tabs. Next is intervals which are tabular but they have the
added requirement that 3 of the columns must be the chromosome, start point,
and end point. There is optionally a strand and header labelling the
columns. Next is BED or GFF, which are also tabular and intervals, but with more
restrictions. BED can vary between 3 and 12 columns, with each being
precisely defined. Here
the order of the columns also matters, and only the end columns can be skipped.
Some groups of the columns have to be all there or all left off. GFF is
similar in setup but with all 9 columns required and different definitions.
See more detailed descriptions below.
<hr>
<H2>Formats</H2>
<h3>Format Descriptions</h3>
<ul>
<li><a href="#ab1">Ab1</a>
<li><a href="#axt">AXT</a>
<li><a href="#bam">BAM</a>
<li><a href="#binseq">Binseq.zip</a>
<li><a href="#bed">BED</a>
<li><a href="#bedgraph">BedGraph</a>
<li><a href="#binseq">Binseq.zip</a>
<li><a href="#fasta">FASTA</a>
<li><a href="#fastqsolexa">FastqSolexa</a>
<li><a href="#fped">fped</a>
<li><a href="#gff">GFF</a>
<li><a href="#gff3">GFF3</a>
<li><a href="#gtf">GTF</a>
<li><a href="#html">Html</a>
<li><a href="#html">HTML</a>
<li><a href="#interval">Interval</a>
<li><a href="#lav">LAV</a>
<li><a href="#lped">Lped</a>
<li><a href="#lped">lped</a>
<li><a href="#maf">MAF</a>
<li><a href="#pbed">pbed</a>
<li><a href="#psl">PSL</a>
@@ -57,20 +74,26 @@ See more detailed descriptions below.
<li><a href="#wig">Wiggle custom track</a>
<li><a href="#text">Other text type</a>
</ul>
<br>
<p>
<hr>
<strong>Ab1</strong>
<a name="ab1"/>
<p>
A binary sequence file in 'ab1' format with a '.ab1' file extension. You must manually select this 'File Format' when uploading the file.
A binary sequence file in 'ab1' format with a '.ab1' file extension.
You must manually select this file format when uploading the file.
<hr/>
<strong>AXT</strong>
<a name="axt"/>
<p>
blastz pairwise alignment format. Each alignment block in an axt file contains three lines: a summary line and 2 sequence lines. Blocks are separated from one another by blank lines. The summary line contains chromosomal position and size information about the alignment. It consists of 9 required fields.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/axt.html">Click here</a> for more information about axt format.
Used for pairwise alignment output from BLASTZ, after post-processing.
Each alignment block contains three lines: a summary line and two
sequence lines. Blocks are separated from one another by blank lines.
The summary line contains chromosomal position and size information
about the alignment, and consists of nine required fields.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/axt.html"
>More information</a>
<dl><dt>Can be converted to:
<dd><ul>
<li>FASTA<br/>
@@ -80,12 +103,13 @@ Convert Formats&rarr;AXT to LAV
</ul></dl>
<hr/>
<strong>Bam</strong>
<strong>BAM</strong>
<a name="bam"/>
<p>
A binary file compressed in the BGZF format with a '.bam' file extension.
<a href="http://samtools.sourceforge.net/SAM1.pdf">SAM</a>
format is the human readable text version of these files.
A binary file compressed in the BGZF format with a '.bam' file
extension.
<a href="http://samtools.sourceforge.net/SAM1.pdf">SAM</a> format
is the human readable text version of these files.
<dl><dt>Can be converted to:
<dd><ul>
<li>pileup<br/>
@@ -96,23 +120,21 @@ NGS: SAM Tools&rarr;Pileup-to-Interval
</ul></dl>
<hr/>
<strong>Binseq.zip</strong>
<a name="binseq"/>
<p>
A zipped archive consisting of binary sequence files in either 'ab1' or 'scf' format. All files in this archive must have the same file extension which is one of '.ab1' or '.scf'. You must manually select this 'File Format' when uploading the file.
<hr/>
<strong>BED</strong>
<a name="bed"/>
<p>
<ul>
<li> also tabular
<li> also interval
<li> also qualifies as tabular
<li> also qualifies as interval
</ul>
This describes a genomic interval, but has strict field specifications for use
in browsers.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/hgTracksHelp.html#BED">Click here</a> for field specifications.<br/>
This tab-separated format describes a genomic interval, but has
strict field specifications for use in genome browsers. BED files
can have from 3 to 12 columns, but the order of the columns matters,
and only the end ones can be omitted. Some groups of columns must
be all present or all absent.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/hgTracksHelp.html#BED"
>Field specifications</a>
<p>
Example:
<pre>
chr22 1000 5000 cloneA 960 + 1000 5000 0 2 567,488, 0,3512
@@ -129,22 +151,35 @@ Convert Formats&rarr;BED-to-GFF
<a name="bedgraph"/>
<p>
<ul>
<li> also tabular
<li> also interval
<li> also BED
<li> also qualifies as tabular
<li> also qualifies as interval
<li> also qualifies as BED
</ul>
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/bedgraph.html">BedGraph</a>
is a BED file with the name column being a float value that is displayed
as a Wiggle in tracks. Unlike wiggles this score can be retrieved in its
exact value after being loaded as a track.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/bedgraph.html"
>BedGraph</a> is a BED file with the name column being a float value
that is displayed as a wiggle score in tracks. Unlike in Wiggle
format, the exact value of this score can be retrieved after being
loaded as a track.
<hr/>
<strong>Fasta</strong>
<strong>Binseq.zip</strong>
<a name="binseq"/>
<p>
A zipped archive consisting of binary sequence files in either
'ab1' or 'scf' format. All files in this archive must have the same
file extension which is one of '.ab1' or '.scf'. You must manually
select this file format when uploading the file.
<hr/>
<strong>FASTA</strong>
<a name="fasta"/>
<p>
A sequence in
<a href="http://www.ncbi.nlm.nih.gov/blast/fasta.shtml">FASTA format</a>
consists of a single-line description, followed by lines of sequence data. The first character of the description line is a greater-than ("&gt;") symbol in the first column. All lines should be shorter than 80 characters::
<a href="http://www.ncbi.nlm.nih.gov/blast/fasta.shtml">FASTA</a>
format consists of a single-line description, followed by lines of
sequence data. The first character of the description line is a
greater-than ("&gt;") symbol. All lines should be shorter than 80
characters.
<pre>
>sequence1
atgcgtttgcgtgc
@@ -164,7 +199,8 @@ Convert Formats&rarr;FASTA-to-Tabular
<a name="fastqsolexa"/>
<p>
<a href="http://maq.sourceforge.net/fastq.shtml">FastqSolexa</a>
is the Illumina (Solexa) variant of the Fastq format, which stores sequences and quality scores in a single file
is the Illumina (Solexa) variant of the Fastq format, which stores
sequences and quality scores in a single file.
<pre>
@seq1
GACAGCTTGGTTTTTAGTGAGTTGTTCCTTTCTTT
@@ -174,9 +210,9 @@ hhhhhhhhhhhhhhhhhhhhhhhhhhPW@hhhhhh
GCAATGACGGCAGCAATAAACTCAACAGGTGCTGG
+seq2
hhhhhhhhhhhhhhYhhahhhhWhAhFhSIJGChO
</pre>
Or
<pre>
@seq1
GAATTGATCAGGACATAGGACAACTGTAGGCACCAT
+seq1
@@ -196,19 +232,22 @@ Convert Formats&rarr;FASTQ to FASTA
<strong>fped</strong>
<a name="fped"/>
<p>
Also known as the FBAT format, for use in the <a href="http://biosun1.harvard.edu/~fbat/fbat.htm">FBAT</a> program.
It consists of a pedigree file and an phenotype file.
Also known as the FBAT format, for use with the
<a href="http://biosun1.harvard.edu/~fbat/fbat.htm">FBAT</a> program.
It consists of a pedigree file and a phenotype file.
<hr/>
<strong>Gff</strong>
<strong>GFF</strong>
<a name="gff"/>
<p>
<ul>
<li> also tabular
<li> also interval
<li> also qualifies as tabular
<li> also qualifies as interval
</ul>
GFF lines have
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format3">nine required fields</a> that must be tab-separated.
GFF is a tab-separated format somewhat similar to BED, but it has
different columns and is more flexible. There are
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format3"
>nine required fields</a>.
<dl><dt>Can be converted to:
<dd><ul>
<li>BED<br/>
@@ -220,66 +259,80 @@ Convert Formats&rarr;GFF-to-BED
<a name="gff3"/>
<p>
<ul>
<li> also tabular
<li> also interval
<li> also qualifies as tabular
<li> also qualifies as interval
</ul>
The
<a href="http://www.sequenceontology.org/gff3.shtml">GFF3</a>
format addresses the most common extensions to GFF, while preserving backward compatibility with previous formats.
The <a href="http://www.sequenceontology.org/gff3.shtml">GFF3</a>
format addresses the most common extensions to GFF, while preserving
backward compatibility with previous formats.
<hr/>
<strong>GTF</strong>
<a name="gtf"/>
<p>
<ul>
<li> also tabular
<li> also interval
<li> also qualifies as tabular
<li> also qualifies as interval
</ul>
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format4">GTF</a>
is a format for describing genes and other features associated with DNA, RNA and Protein sequences.
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format4"
>GTF</a> is a format for describing genes and other features
associated with DNA, RNA, and protein sequences.
<dl><dt>Can be converted to:
<dd><ul>
<li>BED graph<br/>
<li>BedGraph<br/>
Convert Formats&rarr;GTF-to-BEDGraph
</ul></dl>
<hr/>
<strong>Html</strong>
<strong>HTML</strong>
<a name="html"/>
<p>
This format is a html web page. Click the eye icon to view the dataset in
your browser.
This format is an HTML web page. Click the eye icon next to the
dataset to view it in your browser.
<hr/>
<strong>Interval (Genomic Intervals)</strong>
<strong>Interval</strong>
<a name="interval">
<p>
<ul>
<li> also tabular
<li> also qualifies as tabular
</ul>
Required fields
This Galaxy format represents genomic intervals. It is tab-separated,
but has the added requirement that three of the columns must be the
chromosome name, start position, and end position. An optional
strand column can also be specified, and an initial header row can
be used to label the columns, which do not have to be in any special
order. Arbitrary additional columns can also be present.
<p>
Required fields:
<ul>
<li>CHROM - The name of the chromosome (e.g. chr3, chrY, chr2_random) or contig (e.g. ctgY1).
<li>START - The starting position of the feature in the chromosome or contig. The first base in a chromosome is numbered 0.
<li>END - The ending position of the feature in the chromosome or contig. The chromEnd base is not included in the display of the feature. For example, the first 100 bases of a chromosome are defined as chromStart=0, chromEnd=100, and span the bases numbered 0-99.
<li>CHROM - The name of the chromosome (e.g. chr3, chrY, chr2_random)
or contig (e.g. ctgY1).
<li>START - The starting position of the feature in the chromosome or
contig. The first base in a chromosome is numbered 0.
<li>END - The ending position of the feature in the chromosome or
contig. This base is not included in the feature. For example,
the first 100 bases of a chromosome are described as START=0,
END=100, and span the bases numbered 0-99.
</ul>
Optional
Optional:
<ul>
<li>STRAND - Defines the strand - either '+' or '-'.
<li>Headers
<li>STRAND - Defines the strand, either '+' or '-'.
<li>Header row
</ul>
Example:
<pre>
#CHROM START END STRAND NAME COMMENT
chr1 10 100 + exon myExon
chrX 1000 10050 - gene myGene
#CHROM START END STRAND NAME COMMENT
chr1 10 100 + exon myExon
chrX 1000 10050 - gene myGene
</pre>
<dl><dt>Can be converted to:
<dd><ul>
<li>BED<br/>
The exact changes needed and tools to run can vary with what fields are in
the interval file and what size BED you are converting to. In general you
will likely use Text Manipulation&rarr;Compute, Cut or Merge Columns.
The exact changes needed and tools to run will vary with what fields
are in the interval file and what type of BED you are converting to.
In general you will likely use Text Manipulation&rarr;Compute, Cut,
or Merge Columns.
</ul></dl>
<hr/>
@@ -287,7 +340,8 @@ will likely use Text Manipulation&rarr;Compute, Cut or Merge Columns.
<a name="lav"/>
<p>
<a href="http://www.bx.psu.edu/miller_lab/dist/lav_format.html">LAV</a>
is the primary output format for BLASTZ. The first line of a .lav file begins with #:lav..
is the raw pairwise alignment format that is output by BLASTZ. The
first line begins with <code>#:lav</code>.
<dl><dt>Can be converted to:
<dd><ul>
<li>BED<br/>
@@ -295,18 +349,20 @@ Convert Formats&rarr;LAV to BED
</ul></dl>
<hr/>
<strong>Lped</strong>
<strong>lped</strong>
<a name="lped"/>
<p>
This is the linkage pedigree format (separate map and ped files).
These files together describe SNPs, the map file has the position and an
identifier for the SNP and the pedigree file has the alleles.
To upload this format into Galaxy do not use auto-detect for the file
format, instead select lped. You will then be given two sections for uploading
files, one for the pedigree file and one for the map file.
For more information see <a href="http://www.broadinstitute.org/science/programs/medical-and-population-genetics/haploview/input-file-formats-0">linkage pedigree</a>
or <a href="http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#map">map</a>
or <a href="http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#ped">ped</a>.
This is the linkage pedigree format, which consists of separate
<code>map</code> and <code>ped</code> files. Together these files
describe SNPs; the map file contains the position and an identifier
for the SNP, while the pedigree file has the alleles.
To upload this format into Galaxy, do not use auto-detect for the
file format; instead select <code>lped</code>. You will then be
given two sections for uploading files, one for the pedigree file
and one for the map file. For more information, see
<a href="http://www.broadinstitute.org/science/programs/medical-and-population-genetics/haploview/input-file-formats-0">linkage pedigree</a>,
<a href="http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#map">map</a>,
and/or <a href="http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#ped">ped</a>.
<dl><dt>Can be converted to:
<dd><ul>
<li>pbed<br/>Automatic
@@ -317,16 +373,20 @@ or <a href="http://pngu.mgh.harvard.edu/~purcell/plink/data.shtml#ped">ped</a>.
<strong>MAF</strong>
<a name="maf"/>
<p>
TBA and multiz multiple alignment format. The first line of a .maf file begins with ##maf. This word is followed by white-space-separated "variable=value pairs". There should be no white space surrounding the "=".
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format5">Click here</a> for more about MAF format.
Multiple alignment format that is output by TBA and Multiz. The
first line begins with <code>##maf</code>. This word is followed by
whitespace-separated "variable=value pairs". There should be no
whitespace surrounding the "=".
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format5"
>More information</a>
<dl><dt>Can be converted to:
<dd><ul>
<li>BED<br/>
Convert Formats&rarr;Maf to BED
<li>Interval<br/>
Convert Formats&rarr;Maf to Interval
Convert Formats&rarr;MAF to BED
<li>interval<br/>
Convert Formats&rarr;MAF to Interval
<li>FASTA<br/>
Convert Formats&rarr;Maf to FASTA
Convert Formats&rarr;MAF to FASTA
</ul></dl>
<hr/>
@@ -344,7 +404,7 @@ This is the binary version of the lped file format.
<a name="psl"/>
<p>
<a href="http://main.genome-browser.bx.psu.edu/FAQ/FAQformat#format2">PSL</a>
format is for alignments, it is returned by
format is used for alignments returned by
<a href="http://genome.ucsc.edu/cgi-bin/hgBlat?command=start">BLAT</a>.
It does not include any sequence.
<hr/>
@@ -352,9 +412,10 @@ It does not include any sequence.
<strong>Scf</strong>
<a name="scf"/>
<p>
A binary sequence file in 'scf' format with a '.scf' file extension. You must manually select this 'File Format' when uploading the file.
<a href="http://staden.sourceforge.net/manual/formats_unix_2.html">Click here</a>
for more information.
A binary sequence file in 'scf' format with a '.scf' file extension.
You must manually select this file format when uploading the file.
<a href="http://staden.sourceforge.net/manual/formats_unix_2.html"
>More information</a>
<hr/>
<strong>Sff</strong>
@@ -373,58 +434,65 @@ Convert Formats&rarr;SFF converter
<strong>Table</strong>
<a name="table"/>
<p>
Text delimited into columns by something other than a tab.
Text data separated into columns by something other than tabs.
<hr/>
<strong>Tabular (tab delimited)</strong>
<strong>Tabular (tab-delimited)</strong>
<a name="tab"/>
<p>
Any data in tab delimited format (tabular)
One or more columns of text data separated by tabs.
<dl><dt>Can be converted to:
<dd><ul>
<li>FASTA<br/>
Convert Formats&rarr;Tabular-to-FASTA<br/>
Tabular file must have a title and sequence column.
The tabular file must have a title and sequence column.
<li>interval<br/>
If the tabular file has the chromosome, or is all on one chromosome, and
a position you can create an interval file. If all one chromosome use
Text Manipulation&rarr;Add column to add the chromosome. If the given position
is a 1 based position use Text Manipulation&rarr;Compute and the position
column minus 1 to get the start. Otherwise do plus 1 to get the end.
If the tabular file has the chromosome, or is all on one chromosome,
and has a position you can create an interval file (e.g. for SNPs).
If it is all on one chromosome, use Text Manipulation&rarr;Add column
to add a chromosome column. If the given position is 1-based, use
Text Manipulation&rarr;Compute with the position column minus 1 to
get the start, and use the original given column for the end.
If the given position is 0-based, use it as the start, and compute
that plus 1 to get the end.
</ul></dl>
<hr/>
<strong>Txtseq.zip</strong>
<a name="txtseqzip"/>
<p>
A zipped archive consisting of flat text sequence files. All files in this archive must have the same file extension of '.txt'. You must manually select this 'File Format' when uploading the file.
A zipped archive consisting of flat text sequence files. All files
in this archive must have the same file extension of '.txt'. You
must manually select this file format when uploading the file.
<hr/>
<strong>Wiggle custom track</strong>
<a name="wig"/>
<p>
The wiggle format is line-oriented. Wiggle data is preceded by a track definition line, which gives the type of wiggle. There are 3 different types, each
with their uses.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/wiggle.html">More information here.</a>
The wiggle format is line-oriented. Wiggle data is preceded by a
track definition line, which specifies the type of wiggle. There
are three different types, for different uses.
<a href="http://main.genome-browser.bx.psu.edu/goldenPath/help/wiggle.html"
>More information</a>
<dl><dt>Can be converted to:
<dd><ul>
<li>interval<br/>
Convert Formats&rarr;Wiggle-to-Interval<br/>
As a second step this could be converted to BED 3 or 4 by removing columns.
Text Manipulation&rarr;Cut columns from a table
As a second step this could be converted to BED-3 or BED-4 by removing
columns, using Text Manipulation&rarr;Cut columns from a table.
</ul></dl>
<hr/>
<strong>Other text type</strong>
<a name="text"/>
<p>
Any text file
Any text file.
<dl><dt>Can be converted to:
<dd><ul>
<li>tabular<br/>
If this is space or some other delimiter separated fields it can be
converted to tabular. Text Manipulations&rarr;Convert delimiters to TAB
If this has fields separated by spaces, commas, or some other
delimiter it can be converted to tabular using
Text Manipulation&rarr;Convert delimiters to TAB
</ul></dl>
<!-- blank lines so internal links will jump farther to end -->
<br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/>