This doesn't quite work because of the dynamic output collection,
which fails with:
```
galaxy.tools.parameters.output_collect DEBUG 2018-06-06 18:40:38,784 (3) Add dynamic collection datasets to history for output [reverse] (171.970 ms)
galaxy.tools.parameters.output_collect ERROR 2018-06-06 18:40:38,839 Problem gathering output collection.
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 340, in collect_dynamic_outputs
collection_builder.populate()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 88, in populate
elements = self.build_elements()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 59, in build_elements
new_elements[identifier] = element.build()
AttributeError: 'HistoryDatasetAssociation' object has no attribute 'build'
```
This seems to happen because the inner collection is wrongly detected as
nested. In general I doubt that mapping over colleciton output works
with dynamically discovered output collections.
It is jarring to users that works for collection inputs but not multiple data inputs, especially because the IUC discourages collection inputs when multiple data inputs would work.
Fix handling comparison of older hashes in Python 2 and fix the hanlding of both for Python 3.
The previous iteration gave me some errors in Python 3, I brought in and adapted @mvdbeek's Python 3 fixes to fix these:
```
======================================================================
ERROR: test_security_passwords.test_hash_and_check
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/john/workspace/galaxy-lib/.tox/py34/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/Users/john/workspace/galaxy-lib/tests/test_security_passwords.py", line 12, in test_hash_and_check
simple_pass_hash = passwords.hash_password(simple_pass)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 21, in hash_password
return hash_password_PBKDF2(password)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 44, in hash_password_PBKDF2
hashed = pbkdf2_bin(smart_str(password), salt, COST_FACTOR, KEY_LENGTH, getattr(hashlib, HASH_FUNCTION))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 78, in pbkdf2_bin
rv = u = _pseudorandom(salt + _pack_int(block))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 75, in _pseudorandom
return [ord(_) for _ in h.digest()]
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 75, in <listcomp>
return [ord(_) for _ in h.digest()]
TypeError: ord() expected string of length 1, but int found
======================================================================
ERROR: test_security_passwords.test_hash_consistent
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/john/workspace/galaxy-lib/.tox/py34/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/Users/john/workspace/galaxy-lib/tests/test_security_passwords.py", line 38, in test_hash_consistent
assert passwords.check_password(simple_pass, simple_pass_hash)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 30, in check_password
if check_password_PBKDF2(guess, hashed):
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 53, in check_password_PBKDF2
hashed_guess = pbkdf2_bin(smart_str(guess), salt, int(cost_factor), KEY_LENGTH, getattr(hashlib, hash_function))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 78, in pbkdf2_bin
rv = u = _pseudorandom(salt + _pack_int(block))
TypeError: Can't convert 'bytes' object to str implicitly
```
Added tests that all now pass in Python 3 - including generating some passwords in Python 3 and verifying they pass when checked in Python 2 and vice versa.
Update tool XSD to encourage using fully qualified inputs.
Reorganized the existing tests for consistency with other test tools I think.
(Rebased with fixes, including XSD fixes from @nsoranzo.)
- Incorrect parsing of default auto sniff flag value.
- Not respecting flag value when adding to dict of compressed sniffers.
- Not checking the right value when checking for compressed datatype instances to add to sniff order.
- Not filtering out the sniffers from being automatically added during the default stage of sniffer handling where all types with sniff are added at the end.
This last point is the most complicated. I added a hack with #6091 so that the compressed sniffers would only return True if in validation mode or explicitly enabled. I'm not convinced that approached worked and I'm definitely convinced it was a bad approach.
The better, cleaner approach is to just always let the sniffer return its truth regardless of validation mode or not and instead only add it to the list of sniffers if it should be enabled. This is what I was assuming would happen but I found this method append_to_sniff_order that was adding sniffers back in that I didn't want added.
- Rev to new container with 18.05 dependencies pre-installed.
- Enable caching of Yarn stuffs to enable
- Redo fix for git not working in node container - was mounting /etc/passwd but that wasn't compat. with Mac OS X, setting these environment variables are.
- Rearrange startup script a bit to make it load faster (wait for Selenium right before the tests not before waiting for DB).
- Fix random error in logs related to Selenium trying to read something in /root by passing -H to set the Selenium user's home directory.
Only fetch visible datasets into big, cached list of history datasets under consideration. Hidden datasets don't seem to be used by the fetcher or initial value stuff so it seems fine to exclude them. The advantage should be clear for histories with a large number of datasets hidden below a signficantly smaller number collections.
Nicola Soranzo