Fix handling comparison of older hashes in Python 2 and fix the hanlding of both for Python 3.
The previous iteration gave me some errors in Python 3, I brought in and adapted @mvdbeek's Python 3 fixes to fix these:
```
======================================================================
ERROR: test_security_passwords.test_hash_and_check
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/john/workspace/galaxy-lib/.tox/py34/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/Users/john/workspace/galaxy-lib/tests/test_security_passwords.py", line 12, in test_hash_and_check
simple_pass_hash = passwords.hash_password(simple_pass)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 21, in hash_password
return hash_password_PBKDF2(password)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 44, in hash_password_PBKDF2
hashed = pbkdf2_bin(smart_str(password), salt, COST_FACTOR, KEY_LENGTH, getattr(hashlib, HASH_FUNCTION))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 78, in pbkdf2_bin
rv = u = _pseudorandom(salt + _pack_int(block))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 75, in _pseudorandom
return [ord(_) for _ in h.digest()]
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 75, in <listcomp>
return [ord(_) for _ in h.digest()]
TypeError: ord() expected string of length 1, but int found
======================================================================
ERROR: test_security_passwords.test_hash_consistent
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/john/workspace/galaxy-lib/.tox/py34/lib/python3.4/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/Users/john/workspace/galaxy-lib/tests/test_security_passwords.py", line 38, in test_hash_consistent
assert passwords.check_password(simple_pass, simple_pass_hash)
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 30, in check_password
if check_password_PBKDF2(guess, hashed):
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 53, in check_password_PBKDF2
hashed_guess = pbkdf2_bin(smart_str(guess), salt, int(cost_factor), KEY_LENGTH, getattr(hashlib, hash_function))
File "/Users/john/workspace/galaxy-lib/galaxy/security/passwords.py", line 78, in pbkdf2_bin
rv = u = _pseudorandom(salt + _pack_int(block))
TypeError: Can't convert 'bytes' object to str implicitly
```
Added tests that all now pass in Python 3 - including generating some passwords in Python 3 and verifying they pass when checked in Python 2 and vice versa.
Update tool XSD to encourage using fully qualified inputs.
Reorganized the existing tests for consistency with other test tools I think.
(Rebased with fixes, including XSD fixes from @nsoranzo.)
- Incorrect parsing of default auto sniff flag value.
- Not respecting flag value when adding to dict of compressed sniffers.
- Not checking the right value when checking for compressed datatype instances to add to sniff order.
- Not filtering out the sniffers from being automatically added during the default stage of sniffer handling where all types with sniff are added at the end.
This last point is the most complicated. I added a hack with #6091 so that the compressed sniffers would only return True if in validation mode or explicitly enabled. I'm not convinced that approached worked and I'm definitely convinced it was a bad approach.
The better, cleaner approach is to just always let the sniffer return its truth regardless of validation mode or not and instead only add it to the list of sniffers if it should be enabled. This is what I was assuming would happen but I found this method append_to_sniff_order that was adding sniffers back in that I didn't want added.
- Rev to new container with 18.05 dependencies pre-installed.
- Enable caching of Yarn stuffs to enable
- Redo fix for git not working in node container - was mounting /etc/passwd but that wasn't compat. with Mac OS X, setting these environment variables are.
- Rearrange startup script a bit to make it load faster (wait for Selenium right before the tests not before waiting for DB).
- Fix random error in logs related to Selenium trying to read something in /root by passing -H to set the Selenium user's home directory.
Only fetch visible datasets into big, cached list of history datasets under consideration. Hidden datasets don't seem to be used by the fetcher or initial value stuff so it seems fine to exclude them. The advantage should be clear for histories with a large number of datasets hidden below a signficantly smaller number collections.
Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Just annotate auto_compressed_types="gz|bz2|gz,bz2" in datatypes_conf.xml and get for free:
- The compressed datatype with correct extension, no need to have a class definition.
- Converters from the compressed datatype to the uncompressed datatype.
- A sniffer in the correct order.
Additionally, the hacky way we were controlling whether types would be sniffed or not has been improved. This can now be set in the datatypes_conf.xml file or read from a global Galaxy configuration option for options that haven't been explicitly set on a per-datatype basis in the configuration (sniff_compressed_dynamic_datatypes_default).
This should simplify all future datatypes moving in this direction and introduces the possibility of also just loading them in dynamically as needed based on what the tools in the tool panel annotate for inputs and outputs. See a tool with an input of type "genbank.gz" should be enough to trigger this someday I hope.
Re-use test_data specification used by many of the API workflow tests and a workflow used by another test. Add abstractions and DOM element for mapping labeled test data to tool form inputs.
Nicola Soranzo