SLiMSuite 2014-04-22 now available

A new download of SLiMSuite (release 2014-04-22) is now available. As well as fixing the gopher.py error, the download page and readme have had a slight makeover, which should make them load quicker.

As part of ongoing consolidation and documentation, SeqSuite has now been incorporated into in a single SLiMSuite download. (Previously, SLiMSuite was available as a reduced set of programs and SeqSuite had the full set.) The intention is to retire the SeqSuite moniker over the coming months, although the programs themselves will still be available.

The lastest release also features a new program, SLiMFarmer, for running (Q)SLiMFinder and SLiMProb batch jobs on parallel processors. SLiMFarmer is still under development and should hopefully work with other SLiMSuite programs too but has not yet been tested.

Other miscellaneous updates are listed below.

Updates since last release:

• comparimotif_V3: Updated from Version 3.10.
→ Version 3.10: Added forking.
→ Version 3.11: Added additional overlap/matchfix checks during basic comparison to try and speed up.
→ Version 3.12: Replaced deprecated sets.Set() with set().

• gablam: Updated from Version 2.11.
→ Version 2.12: Consolidated use of BLAST V2.

• haqesac: Updated from Version 1.9.
→ Version 1.10: Added exceptions for BLAST failure.

• picsi: Updated from Version 1.1.
→ Version 1.2: Updated to BUDAPEST 2.3 and rje_mascot.

• pingu_V4: Created.
→ Version 4.0: Initial Compilation based on code from SLiMBench and PINGU 3.9 (inherited as pingu_V3).
→ Version 4.1: Adding compilation of PPI databases using new rje_xref V1.1 and older objects from PINGU V3.
→ Version 4.2: Bug fixes for use of PPISource to create PPI databases.

• qslimfinder: Updated from Version 1.6.
→ Version 1.7: Fixed "MustHave=LIST" correction of motif space.

• seqmapper: Updated from Version 2.0.
→ Version 2.1: Added catching of failure to read input sequences. Removed 'Run' from GABLAM table.

• slimbench: Updated from Version 2.0.
→ Version 2.1: Fixed memsaver=T unless in development mode (dev=T). Removed old Assessment. Tested with simbench analysis.
→ Version 2.2: Replaced searchini=LIST with searchini=FILE and moved to SimBench commands.
→ Version 2.2: Modified the FN/TN and ResNum calculations. No longer rate TP in random data as OT.

• slimfarmer: Created.
→ Version 0.0: Initial Compilation.
→ Version 1.0: Functional version using rje_qsub and rje_iridis to fork out SLiMSuite runs.
→ Version 1.1: Updated to use rje_hpc.JobFarmer and incorporate main SLiMSuite farming within SLiMFarmer class.

• slimfinder: Updated from Version 4.5.
→ Version 4.6: Minor modification to seqocc=T function. !Experimental! Added main occurrence output and modified savespace.

• slimmutant: Created.
→ Version 0.0: Initial Compilation.
→ Version 1.0: Working version with standalone functionality.

• slimprob: Updated from Version 1.0.
→ Version 1.1: Tidied import commands.
→ Version 1.2: Increased extras=X levels. Adjusted maxsize=X assessment to be post-masking.

• ned_rankbydistribution: Updated from Version 1.1.
→ Version 1.2: Replaced depracated Set module.

• rje: Updated from Version 4.8.
→ Version 4.9: Added rje.slimsuite, which determines the slimsuite home directory from rje.py file path.
→ Version 4.10: Added osx=T/F option for Mac-specific running options.

• rje_blast_V2: Updated from Version 2.4.
→ Version 2.5: Minor modifications for SLiMCore UPC generation.
→ Version 2.6: Minor bug fixes.

• rje_db: Updated from Version 1.2.
→ Version 1.3: Minor modifications for SLiMCore FUPC development.
→ Version 1.4: Added list checking with addEmptyTable.

• rje_dismatrix_V2: Updated from Version 2.9.
→ Version 2.10: Minor modifications for SLiMCore UPC.

• rje_genemap: Updated from Version 1.4.
→ Version 1.5: Minor tweak of expected HGNC input following change to downloads.

• rje_hpc: Created.
→ Version 1.0: Initial Compilation based on rje_iridis V1.10.

• rje_iridis: Updated from Version 1.9.
→ Version 1.10: Modified freemem setting to run on Katana. Made rsh optional. Removed defunct IRIDIS3 option.

• rje_obj: Updated from Version 1.3.
→ Version 1.4: Added sourceDataFile() method from SLiMBench for wider use.
→ Version 1.5: Added 'basestr' and 'basefile' cmdlist types.
→ Version 1.6: Added osx=T/F option for Mac-specific running options.

• rje_qsub: Updated from Version 1.4.
→ Version 1.5: Added emailing of job stats after run. Added vmem limit.

• rje_seq: Updated from Version 3.17.
→ Version 3.18: Minor BLAST+ bug fixes. Added exceptions to readBLAST failure.

• rje_seqlist: Updated from Version 1.3.
→ Version 1.4: Added dna2prot reformat function.

• rje_slimcore: Updated from Version 1.12.
→ Version 1.13: Modified the savespace settings to reduce numbers of files. targz file now uses RunID not Build Info.
→ Version 1.14: Started adding code for Fragmented UPC (FUPC) clustering.

• rje_slimlist: Updated from Version 1.2.
→ Version 1.3: Added auto-download of ELM data.

• rje_uniprot: Updated from Version 3.14.
→ Version 3.14: Added dblist=LIST and dbsplit=T/F for additional DB link output control. Set unipath default to url.
→ Version 3.15: Added extraction of taxonomic groups. Add UniFormat to improve pure downloads.
→ Version 3.16: Added WBGene ID's from WormBase as one of the recognised DB XRef to parse.
→ Version 3.17: Efficiency tweak to URL-based extraction of acclist.
→ Version 3.18: Minor modification to database parsing.

• rje_xref: Updated from Version 1.0.
→ Version 1.1: Added output of ID lists to text files. Major reworking. Tested with HPRD and HGNC.

New downloads and fixed webpages

New releases of SeqSuite and SLiMSuite are now available. The webpages have now hopefully been fixed too, including the broken Manual links. (A bit of trouble parsing some the docstrings had messed up the HTML, in case you care!) Please report any more anomalies.

There are not many major updates since the last release. The biggest are that SLiMFinder (and QSLiMFinder) now produce a single *.occ.csv containing motif instances for all datasets, in addition to the old dataset-specific files. This is to make the output more consistent with SLiMProb although do note that some of the column headers are different. The new file contains the same data as the old dataset-specific *.occ.csv files plus two additional columns: Dataset and RunID. (These match the main *.csv output.)

Dataset-specific results files have also been cleaned up a little for (Q)SLiMFinder and SLiMProb (i.e. the SLiMCore Class in libraries/rje_slimcore) to make the targz=T/F and savespace=X options a little more useful and consistent. This will be the subject of another post shortly.

Other miscellaneous updates are listed below.

Updates since last release:

• comparimotif_V3: Updated from Version 3.10.
→ Version 3.10: Added forking.
→ Version 3.11: Added additional overlap/matchfix checks during basic comparison to try and speed up.

• qslimfinder: Updated from Version 1.6.
→ Version 1.7: Fixed "MustHave=LIST" correction of motif space.

• slimfinder: Updated from Version 4.5.
→ Version 4.6: Minor modification to seqocc=T function. !Experimental! Added main occurrence output and modified savespace.

• rje_pydocs: Updated from Version 2.8.
→ Version 2.8: Added docsource=PATH : Input path for Python Module documentation (manuals etc.) ['../docs/']
→ Version 2.9: Attempts to fix some broken links and sort out manuals confusion

• rje_slimcore: Updated from Version 1.12.
→ Version 1.13: Modified the savespace settings to reduce numbers of files. targz file now uses RunID not Build Info.

• rje_uniprot: Updated from Version 3.14.
→ Version 3.14: Added dblist=LIST and dbsplit=T/F for additional DB link output control. Set unipath default to url.
→ Version 3.15: Added extraction of taxonomic groups. Add UniFormat to improve pure downloads.

New Software Release

New releases of SLiMSuite and SeqSuite are now available. Please note that RJESuite has now been discontinued - for simplicity, all of the extra gubbins is now part of the SeqSuite release. SLiMSuite still represents a cut-down version that focuses on Short Linear Motif analysis tools.

There have been a number of updates since the last release, which will be the focus of future posts. The biggest change since the last release is the implementation of BLAST+ as the default in place of BLAST for most tools. The old BLAST can still be invoked using the oldblast=T switch. In addition to blastpath=PATH, a new blast+path=PATH parameter will need to be set.

Apart from some file organisation tweaks, the other major change is that CompariMotif now has a memsaver=T mode, which will process very large motif lists much quicker and avoid memory issues. The XGMML output is not (yet) available in this mode. For multi-processor CPUs and large searchdb motif lists, CompariMotif now also supported forking (forks=X).

Documentation is in the process of having an overhaul and is still lagging behind as a result. Please ask if anything is unclear and that section of documentation will be prioritised.

Updates since last release:

• aphid: Updated from Version 2.0.
→ Version 2.1: Reduced import commands.

• budapest: Updated from Version 2.1.
→ Version 2.2: Removed unrequired rje_dismatrix import.
→ Version 2.3: Updated to use rje_blast_V2. Needs further updates for BLAST+. Deleted obsolete OLDreadMascot() method.

• comparimotif_V3: Updated from Version 3.9.
→ Version 3.10: Added MemSaver option, which will read and process input motifs (not searchdb) one motif at a time.
→ Version 3.10: Added forking.

• fiesta: Updated from Version 1.5.
→ Version 1.6: Removed HAQESAC import (uses MultiHAQ).
→ Version 1.7: Updated to use rje_blast_V2. Needs work to make function with BLAST+.

• gablam: Updated from Version 2.10.
→ Version 2.11: Altered to use BLAST+ and rje_blast_V2.

• gasp: Updated from Version 1.3.
→ Version 1.4: Minor tweaks to imports.

• gfessa: Updated from Version 1.2.
→ Version 1.3: Tidied module imports.
→ Version 1.4: Switched to rje_blast_V2. More work needed for BLAST+.

• haqesac: Updated from Version 1.8.
→ Version 1.9: Added rje_blast_V2 implementation and BLAST+. Use oldblast=T for old BLAST.

• peptcluster: Updated from Version 1.3.
→ Version 1.4: Bug fixes for end of sequence characters and different length peptides.

• picsi: Updated from Version 1.0.
→ Version 1.1: Updated to blast_V2 and BLAST+.

• pingu: Updated from Version 3.8.
→ Version 3.9: Tidied imports.

• qslimfinder: Updated from Version 1.5.
→ Version 1.6: Removed excess module imports.

• slimbench: Updated from Version 1.8.
→ Version 1.9: Added memsaver option. Replaced SLiMSearch with SLiMProb. Altered default IO paths.
→ Version 1.9: Removed 3DID again: new ELM interaction_domains file has position-specific PPI details.
→ Version 2.0: Major overhaul of input options to standardise/clarify. Implemented auto-downloads and PPI datasets.

• slimprob: Updated from Version 1.0.
→ Version 1.1: Tidied import commands.

• slimsuite: Created.
→ Version 0.0: Initial Compilation with downloadelm function.

• rje_pydocs: Updated from Version 2.6.
→ Version 2.7: Added rje_ppi output for module links.
→ Version 2.8: Added parsing of commandline options from docstring and cmdRead calls.
→ Version 2.8: Added docsource=PATH : Input path for Python Module documentation (manuals etc.) ['../docs/']

• rje: Updated from Version 4.6.
→ Version 4.7: Added self.warn list and self.warnLog() functions to Log object. Modified i=-1 quitchoice to raise not quit.
→ Version 4.8: Added perc cmdtype = float that is multiplied by 100.0 if < 1.0. Removed server option from iniCmds().

• rje_ancseq: Updated from Version 1.2.
→ Version 1.3: Changed "biproblem" error handling in gaspProbs()

• rje_blast_V1: Updated from Version 1.14.
→ Version 1.15: Added OldBLAST/Legacy option to Object for compatibility with rje_blast_V2. (Always True!)

• rje_blast_V2: Updated from Version 2.1.
→ Version 2.2: Added gablamData() to return old-style GABLAM dictionary from table.
→ Version 2.3: Added blastCluster() method to return UPC clustering and GABLAM distance matrix from a file.
→ Version 2.4: Scrapped BLAST "Run" field to simplify code - keep a single run per BLASTRun object.

• rje_db: Updated from Version 1.0.
→ Version 1.1: Added sortedEntries() function.
→ Version 1.2: Added Table.hasField(field). Add openTable(), readEntry() and readSet() methods.

• rje_forker: Created.
→ Version 0.0: Initial Compilation.

• rje_iridis: Updated from Version 1.8.
→ Version 1.9: Added scanning of legacy folder - moving GOPHER_V2!

• rje_obj: Updated from Version 1.0.
→ Version 1.1: Added rje_zen import and self.zen() to call rje_zen.Zen().wisdom().
→ Version 1.2: Added warnLog functions.
→ Version 1.3: Added perc cmdtype = float that is multiplied by 100.0 if < 1.0. Also added cmdtype = date for YYYY-MM-DD.

• rje_ppi: Updated from Version 2.7.
→ Version 2.8: Tweaked Spring Layout. Stores original Hub and Spoke Field.

• rje_seq: Updated from Version 3.16.
→ Version 3.17: Updated to use BLAST+ and rje_blast_V2

• rje_sequence: Updated from Version 2.2.
→ Version 2.3: Added alternative self.info keys for sequence (for UniProt splice variants). Added SpliceVar dict.

• rje_slimcore: Updated from Version 1.10.
→ Version 1.11: Tidied some of the module imports.
→ Version 1.12: Upgraded BLAST to BLAST+. Can use old BLAST with oldblast=T.

• rje_slimlist: Updated from Version 1.1.
→ Version 1.2: Added some extra functions for CompariMotif Memsaver mode

• rje_tree: Updated from Version 2.9.
→ Version 2.10: Added cleanup of *.r.csv file following R-based PNG generation.

• rje_uniprot: Updated from Version 3.13.
→ Version 3.14: Added direct retrieval of UniProt entries from URL, including full proteomes. Updated output file naming.
→ Version 3.14: Added dblist=LIST and dbsplit=T/F for additional DB link output control. Set unipath default to url.

• rje_xml: Updated from Version 0.1.
→ Version 0.2: Added parsing from URL.

• rje_xref: Updated from Version 0.0.
→ Version 1.0: Added xfrom and xto fields and xMap() function for mapping from one ID set to another.

Availability, Installation and Setup

SLiMSuite and Seqsuite are currently available from http://bioware.soton.ac.uk as three packages:

1. SLiMSuite contains software for Short Linear Motif (SLiM) analysis.
2. SeqSuite contains all of the SLiMSuite programs plus some additional sequence analysis programs.
3. RJESuite contains SLiMSuite, SeqSuite and a bunch of other miscellaneous utilities and bits and bobs.

In future, it is envisaged that a single Git repository will contain all the relevant code and documentation.

All three packages have the same basic installation, directory structure and setup requirements. For basic functionality, no other setup should be necessary beyond downloading and unzipping the package in the desired directory if Python is installed on your system. Some programs will need to use external components or accessory applications, which may need additional installation.

If you do not have Python, you can download it free from www.python.org at http://www.python.org/download/. The modules are written in Python 2.x and most have been tested with 2.7. The Python website has good information about how to download and install Python but if you have any problems, please get in touch and I will help if I can.

All the required files should have been provided in the download zip file. The Python Modules are open source and may be changed if desired, although please give me credit for any useful bits you pillage. I cannot accept any responsibility if you make changes and the program stops working, however! If you want some help understanding the way the modules and classes are set up so you can edit them, just contact me.

Directory Structure

Once unzipped, the download will unpack a top level seqsuite/ or slimsuite/ directory with the following subdirectories:

data/ contains example data for testing programs. (Currently under development.)

docs/ contains documentation.

extras/ contains accessory programs that are not part of the main program suite.

legacy/ contains superseded programs that are no longer supported. (Currently under development.)

libraries/ contains all the python libraries used by the main tools (and extras), some of which have standalone functionality.

settings/ contains INI files set default options.

tools/ contains the main program suite.

NOTE: It is recommended that analyses are performed outside these directories for ease of reinstallation.

Third party software

Many of the tools make use of third party software. Where possible, instructions will be provided for obtaining these programs but a quick Google is usually sufficient - wherever possible, third party software is free for academic use and (ideally) open source.

When third party software is used, SeqSuite will also need to the path to the program, or suite of programs. This will be covered more in the Command-line Options section but BLAST and clustalw deserve a special mention as examples because many of the programs use these as default programs for certain functions.

BLAST is actually a suite of programs and the path containing these executables should be provided using `blastpath=PATH/', e.g.:

blastpath=/usr/ncbi/bin/

For BLAST, do not give the full path to the program (e.g. blastpath=/usr/ncbi/bin/blastp). BLAST cannot be replaced easily by other programs. BLAST has now largely been superseded by BLAST+, which needs its own path parameter:

blast+path=PATH

Some programs are still restricted to BLAST at the moment and other tools can be made to use the BLAST with the oldblast=T switch.

Clustalw is a useful standalone program that is used as a default for alignments and trees in the absence of newer (better) programs. For this, and other single executables, the full path to the program is given:

clustalw=/usr/bioware/clustalw1.83/clustalw

In these situations, a different program with the same input and output can be substituted.

NOTE: Remember to set the relevant paths in an appropriate *.ini file in settings/. Where possible, error messages will identify issues with third party software but due to a lack of testing on a diversity of systems, this is not always possible. If a program crashes, please check the *.log file for signs that there may be a problem with the installation and/or path given for third party programs, such as BLAST.

Upgrading

At present, each upgrade is distributed as a separate package. You can check the current version by the date in the name of the distribution file (in ISO 8601 standard, YYYY-MM-DD format). Plans are afoot to switch to a Git repository, which will make upgrades easier.

New Software Release

New releases of SeqSuite, SLiMSuite and RJESuite are now available.

The biggest change since the last release is the renaming of SLiMSearch to SLiMProb. This is to avoid confusion between the old SLiMSearch 1.x (now SLiMProb) and the newer SLiMSearch 2.x webserver, which has a different range of functions.

Updates since last release:

• cpppred: Created.

• gopher: Updated from Version 3.1.
→ Version 3.2: Minor tweak to prevent unwanted directory generation for programs using existing GOPHER alignments.
→ Version 3.3: Added rje_blast_V2 to use BLAST+. Run with legacy=T to stick with old NCBI BLAST. Started utilising rje_seqlist.

• pepbindpred: Created.

• slimprob: Created.
→ Version 1.0: SLiMProb 1.0 based on SLiMSearch 1.7. Altered output files to be *.csv and *.occ.csv.

• file_monster: Updated from Version 2.0.
→ Version 2.1: Added dirsum function.

• rje: Updated from Version 4.5.
→ Version 4.6: Added dev and warn options.

• rje_blast_V2: Created.
→ Version 2.0: Initial Compilation from rje_blast_V1 V1.14.
→ Version 2.1: Tweaking code to work with GOPHER 3.x - removing self.info etc. Added blastObj() method.

• rje_db: Updated from Version 0.4.
→ Version 0.5: Initial coding of index mode. (Not yet fully functional.)
→ Version 1.0: Working, so upgraded to version 1.0!

• rje_obj: Updated from Version 0.0.
→ Version 1.0: Fully working version, so upgraded to 1.0. Added dev and warn options.

• rje_seq: Updated from Version 3.15.
→ Version 3.16: Added BLAST+ path and seqFromBlastDBCmd()

• rje_slimcalc: Updated from Version 0.5.
→ Version 0.6: Minor tweak to avoid unwanted GOPHER directory generation.
→ Version 0.7: Added RLC to "All" conscore running.

• rje_slimcore: Updated from Version 1.9.
→ Version 1.10: Bypass UPC generation for single sequences.

Documentation is still in the process of development. BLAST+ implementation is ongoing - please get in touch if this is something you need.

New Software Release

New releases of SeqSuite, SLiMSuite and RJESuite are now available.

Updates since last release:

• budapest: Updated from Version 2.0.
→ Version 2.1: Improved handling of iTRAQ data using rje_mascot V1.2.

• comparimotif_V3: Updated from Version 3.8.
→ Version 3.8: Changed scoring of overlapping ambiguities - uses IC of all possible ambiguities. Added "Ugly" match type.
→ Version 3.9: Added xgformat=T/F : Whether to use default CompariMotif formatting or leave blank for e.g. Cytoscape [True]

• happi: Updated from Version 1.1.
→ Version 1.2: Added addclass and refined output for Host-Pathogen PPI analysis.

• pingu: Updated from Version 3.7.
→ Version 3.8: Hopefully fixed issue of Fasta file generation log output writing to wrong log file.

• qslimfinder: Updated from Version 1.4.
→ Version 1.4: Added qexact=T/F option for calculating Exact Query motif space (True) or estimating from dimers (False).
→ Version 1.5: Implemented SigV calculation. Modified extras setting.

• seqmapper: Updated from Version 1.2.
→ Version 2.0: Reworked with new Object format, new BLAST(+) module and new seqlist module.

• slimbench: Updated from Version 1.5.
→ Version 1.6: Added "simonly" to datatype - calculates both SN and FPR from "sim" data (ignores "ran") to check query bias.
→ Version 1.7: Added Benchmarking of ELM datasets without queries.
→ Version 1.8: Added Benchmarking dataset generation from PPI data and 3DID.

• slimfinder: Updated from Version 4.4.
→ Version 4.5: Minor modifications to fix sigV and sigPrime bugs. Modified extras setting. Added palindrome setting for DNA motifs.

• file_monster: Updated from Version 1.6.
→ Version 2.0: Major reworking with new object making use of rje_db tables etc. Old functions to be ported with time.

• rje_dbase: Updated from Version 2.2.
→ Version 2.3: Added construction of EnsEMBL TaxaDB sets during TaxaDB construction.

• rje_seqgen: Updated from Version 1.6.
→ Version 1.7: Modified/fixed ESTgen function to work for protein sequences.

• ned_rankbydistribution: Updated from Version 1.0.

• rje: Updated from Version 4.4.
→ Version 4.5: Modified randomString() and added stringShuffle() methods.

• rje_blast_V1: Created.
→ Version 0.0: Initial Working Compilation.
→ Version 0.1: No Out Object in Objects
→ Version 1.0: Corrected to work with blastn (and blastp)
→ Version 1.1: Added special calling for Cerberus
→ Version 1.2: Added GABLAM and GABLAMO to BlastHit
→ Version 1.3: Added GABLAM calculation upon reading BLAST results and clearing Alignment sequences to save memory
→ Version 1.4: Tidied up the module with improved logging and progress reporting. Added dbCleanup.
→ Version 1.5: Added checking for multiple hits with same name and modified BLAST_Run.hitToSeq()
→ Version 1.6: Added nucleotide vs protein searches to GABLAM
→ Version 1.7: Added nucleotide vs nucleotide searches to GABLAM
→ Version 1.8: Added local alignment summary output to ReadBLAST()
→ Version 1.9: Added BLAST -C
→ Version 1.10: Added BLAST -g
→ Version 1.11: Added gablamfrag=X : Length of gaps between mapped residue for fragmenting local hits [100]
→ Version 1.12: Altered checkDB and cleanupDB to spot index files split over multiple files (*.00.p* etc.)
→ Version 1.13: Added localcut=X : Cut-off length for local alignments contributing to global GABLAM stats) [0]
→ Version 1.14: Added blast.checkProg(qtype,stype) to check whether blastp setting matches sequence formats.

• rje_blast_V2: Created.
→ Version 2.0: Initial Compilation from rje_blast_V1 V1.14.

• rje_db: Updated from Version 0.3.
→ Version 0.4: Improved use of AutoID and added Table.autoID() method.

• rje_ensembl: Updated from Version 2.8.
→ Version 2.9: Reduced DNA chromosome downloads. Updated some species data. Added "known_by_projection" handling.

• rje_genbank: Updated from Version 0.2.
→ Version 0.3: Added reloading of features.

• rje_hmm_V1: Created.
→ Version 0.0: Initial Working Compilation.
→ Version 1.0: Working version with multiple HMM capacity
→ Version 1.1: Added hmmpfam option
→ Version 1.2: Cleaned up and debugged for rje_ensembl.ensDat()

• rje_hmm_V2: Created.
→ Version 2.0: Initial HMMER3.0 version based on Version 1.2 and RJE_BLAST 2.0.

• rje_markov: Updated from Version 2.1.

• rje_mascot: Updated from Version 1.0.
→ Version 1.1: Fixed bugs for reading in data with unmatched peptides and iTRAQ data.
→ Version 1.2: Added

• rje_menu: Updated from Version 0.2.
→ Version 0.3: Modified to work with new object types.

• rje_ppi: Updated from Version 2.5.
→ Version 2.6: Added addPPI(hub,spoke,evidence) method. Added nodelist option.
→ Version 2.7: Added tabout=T/F Output PPI data as Node and Edge tables [False]

• rje_seqlist: Updated from Version 1.1.
→ Version 1.2: Added seqshuffle option for randomising sequences.

• rje_uniprot: Updated from Version 3.12.
→ Version 3.13: Minor bug fix for link table output.

• rje_xref: Created.
→ Version 0.0: Initial Compilation.

New SLiMSuite, SeqSuite and RJESuite downloads available

Just in time for Christmas, new releases of all the downloads are available at the Edwards Lab software page. Documentation is still lagging behind but will hopefully catch up (along with a bit of an overhaul of this blog). Questions welcome in the meantime.

In addition to QSLiMFinder 1.4, the biggest change this release is probably the upgrade of GOPHER. Version 3.x features improved organisation of output files for queries from different species in addition to a capacity to have several different multiple alignment programs run on the same orthologue sets. See the website for more info.

Updates since last release:

• gopher: Created.
→ Version 3.0: See archived GOPHER 1.9 and gopher_V2 2.9 for history and obselete options.
→ Version 3.0: Added organise=T/F and gopherdir=PATH for improved file organisation. Tightened savespace.
→ Version 3.0: Added compfilter=T/F for improved complexity filter and composition statistics control for *initial* BLAST.
→ Version 3.0: Changed default tree extension to *.nwk for compatibility with MEGA. Deleted _phosAlign() method.
→ Version 3.0: Added orthology ID option and alignment program to customise output further.
→ Version 3.1: Added full reciprocal best hit method. (fullrbh=T/F)

• gopher_V2: Updated from Version 2.8.
→ Version 2.9: Deleted oldStigg() method. Added simple Reciprocal Best Hit orthology prediction.

• qslimfinder: Updated from Version 1.2.
→ Version 1.3: Updated the output for Max/Min filtering and the pickup options.
→ Version 1.4: Added additional dictionary and list to store Query dimers and SLiMs for motif space calculations.
→ Version 1.4: Added qexact=T/F option for calculating Exact Query motif space (True) or estimating from dimers (False).

• slimfinder: Updated from Version 4.2.
→ Version 4.3: Updated the output for Max/Min filtering and the pickup options. Removed TempMaxSetting.
→ Version 4.4: Modified to work with GOPHER V3.0.

• rje: Updated from Version 4.3.
→ Version 4.4: Added lineFromIndex(target,file,re_index='^(\S+)\s',sortunique=False,xreplace=True).

• rje_seq: Updated from Version 3.13.
→ Version 3.14: Added CLUSTAL Omega alignment program ['clustalo']
→ Version 3.15: Added PAGAN alignment program ['pagan'] and (hopefully) fixed minor Windows fastacmd bug.

• rje_sequence: Updated from Version 2.1.
→ Version 2.2: Added more yeast species.

• rje_slimcalc: Updated from Version 0.4.
→ Version 0.5: Altered to use GOPHER V3 and handle nested alignment directories.

• rje_slimlist: Updated from Version 1.0.
→ Version 1.1: Modified to work with GOPHER V3.0 for alignments.

New SLiMSuite, SeqSuite and RJESuite releases are now available

New releases of SLiMSuite, SeqSuite and RJESuite are now available from the Edwards Lab software page.

Please note that the documentation (particularly the manuals) are still lagging a bit behind, so do report anything that does not make sense. The default settings also need to be verified as there is a chance that some of these may have inadvertently changed over the years. (The same core code is now used for the webservers, which often have different defaults.) Checking these along with updating and checking the servers themselves are ongoing priorities.

A full list of updated modules is given below. As well as SLiMMaker now handling end of sequence characters, the biggest changes this release are updates to CompariMotif to (3.7) output unmatched input motifs and (3.8) improve handling of partially overlapping ambiguous positions (e.g. [AGS] and [ST]). The motivation behind both these changes is the ongoing benchmarking (and preparation for publication) of QSLiMFinder and the creation of SLiMBench for benchmarking motif prediction methods. A QSLiMFinder section has been added to the SLiMFinder Manual (section 5.4). SLiMBench is still a work in progress and will be documented in a later release.

Updates since last release:

• comparimotif_V3: Updated from Version 3.6.
→ Version 3.7: Added coreIC and output of unmatched motifs.
→ Version 3.8: Added overlaps=T/F : Whether to include overlapping ambiguities (e.g. [KR] vs [HK]) as match [True]
→ Version 3.8: Changed scoring of overlapping ambiguities - uses IC of all possible ambiguities. Added "Ugly" match type.

• slimbench: Created.
→ Version 0.0: Initial Compilation.
→ Version 0.1: Functional version with benchmarking dataset generation.
→ Version 1.0: Consolidation of "working" version with additional basic benchmarking analysis.
→ Version 1.1: Added simulated dataset construction and benchmarking.
→ Version 1.2: Added MinIC filtering to benchmark assessment. Sorted beginning/end of line for reduced ELMs.
→ Version 1.3: Made SimCount a list rather than Integer. Sorted CompariMotif assessment issue.
→ Version 1.4: Added ICCut and SLiMLenCut as lists and output columns.
→ Version 1.5: Added Summary Results output table. Removed PropRes.

• slimmaker: Updated from Version 1.0.
→ Version 1.1: Modified to work with end of line characters.

• slimsearch: Updated from Version 1.5.
→ Version 1.6: Minor tweaks to Log output. Add option for UPC number in occ output.

• rje: Updated from Version 4.1.
→ Version 4.2: Modified INI reading across the board to look in ../settings/ and look for defaults.ini as well as rje.ini.
→ Version 4.2: Enabled handing on -ini FILE in addition to ini=FILE.
→ Version 4.3: Added ilist and nlist types to cmdRead for objects. (Lists of integers and floats). Add ratio() function.

• rje_blast: Updated from Version 1.13.
→ Version 1.14: Added blast.checkProg(qtype,stype) to check whether blastp setting matches sequence formats.

• rje_db: Created.
→ Version 0.0: Initial Compilation.
→ Version 0.1: Added merge tables option.
→ Version 0.2: Miscellaneous updates to various methods.
→ Version 0.3: Minor doc tweaks and added keepFields().

• rje_seq: Updated from Version 3.12.
→ Version 3.13: Updated sequence type checking for use with GABLAM 2.10.

• rje_seqlist: Created.
→ Version 0.0: Initial Compilation. Based on rje_seq 3.10.
→ Version 0.1: Added basic species filtering and sequence output.
→ Version 0.2: Added upper case filtering.
→ Version 0.3: Added accnum filtering and sequence renaming.
→ Version 0.4: Added sequence redundancy filtering.
→ Version 0.5: Added newgene=X for sequence renaming (newgene_spcode__newaccXXX). NewAcc no longer fixed Upper Case.
→ Version 1.0: Upgraded to "ready" Version 1.0. Added concatenate=T and split=X options for sequence concatenation.
→ Version 1.0: Added reading of sequence type from rje_seq.py and mixed=T/F.
→ Version 1.1: Added shortName() and modified SeqDict.

• rje_sequence: Updated from Version 2.0.
→ Version 2.1: Added re_unirefprot = re.compile('^([A-Za-z0-9\-]+)\s+([A-Za-z0-9]+)_([A-Za-z0-9]+)\s+')

• rje_slim: Updated from Version 1.5.
→ Version 1.6: Fixed splitting bug introduced by lower case motifs.

• rje_slimcore: Updated from Version 1.8.
→ Version 1.9: Minor modifications to Log output. Updated motifSeq() function to output unmasked sequences.

• rje_slimlist: Updated from Version 0.6.
→ Version 1.0: Functional module with lower case motif splitting fixed and ? -> .{0,1} replacement.

• rje_zen: Updated from Version 1.0.
→ Version 1.1: Added a few more words here and there.

Using SLiMFinder to discover "local motifs" in protein sequences

The makers of the highly successful MEME Suite have another tool out:

DLocalMotif: A discriminative approach for discovering local motifs in protein sequences

I've not had a chance to go over it in detail but it looks like it could be pretty useful, especially for subcellular targeting motifs. There is one thing that rankles me slightly, though. They define a "local motif" as

"patterns in DNA or protein sequences that occur in a short sequence interval relative to a sequence anchor or landmark."

They then go on to say:

"We believe that DLocalMotif is the only tool for discovering local motifs in protein sequences."

This is just a quick post to point out that SLiMFinder will happily find "local motifs" in protein sequences using the start and end of the sequence as an anchor or landmark. I think it is more limited than DLocalMotif as it is restricted to SLiMs that are very proximal to the sequence termini but it features the usual SLiMChance probability calculations and corrections for evolutionary relationships. (Even without restricting to searches relative to anchor points, SLiMFinder is very successful at finding the KDEL motif and C-terminal PDZ ligand motifs.) The max distance from the termini can be set by maxwild=X up to a limit of 9aa.

If you want to restrict yourself to just N- or C-terminal motifs, use the musthave=LIST option:

musthave="^" for N-terminal motifs.

musthave="$" for C-terminal motifs.

musthave="^,$" for both.

If you want to anchor the motifs internally, this can be done too with a bit of imagination. Just insert an non-standard amino acid character (e.g. Z) at the anchor position, set the expanded alphabet using alphabet=LIST and then force the motif to have the new symbol using musthave=X, e.g.:

alphabet="A,C,D,E,F,G,H,I,K,L,M,N,P,Q,R,S,T,V,W,Y,Z" musthave=Z

I must confess that I have never tried this but it should work and I am happy to help iron out any wrinkles.

(BTW, SLiMFinder also has an experimental feature for using a negative dataset (negatives=FILE if anyone wants to try it out.)

SLiMMaker now handles end of sequence characters

SLiMMaker is a simple tool for generating regular expression motifs from aligned peptide sequences. It was originally made for making new SLiM definitions based on a set of aligned occurrences and therefore the ends of these peptides are typically not the actual ends of sequences. Sometimes, however, such as in the KDEL ER retrieval motif, they are at (or near) the end (or beginning) of a protein and you might want that taken into consideration when generating a motif.

The SLiMMaker website will now accept beginning (^) and end ($) of seuqence characters. The peptides still need to be aligned, so if using them you should fill in any non-conforming peptides with an X. If the peptides are of different lengths and the end of lines character ends up appearing within an ambiguous position with regular amino acids (e.g. [$AGS]), SLiMMaker will truncate the regular expression at that point (not including that position) - otherwise it just gets too messy!

This upgrade will appear in the next distributions of SLiMSuite and SeqSuite. As always, if it is not clear then just try it out with some test sequences. (And report any odd behaviour.)

SLiMPrints paper published

The SLiMPrints paper is now available online at Nucleic Acids Research (doi: 10.1093/nar/gks854; PMID: 22977176).

SLiMPrints: conservation-based discovery of functional motif fingerprints in intrinsically disordered protein regions

Davey NE, Cowan JL, Shields DC, Gibson TJ, Coldwell MJ, Edwards RJ.

Large portions of higher eukaryotic proteomes are intrinsically disordered, and abundant evidence suggests that these unstructured regions of proteins are rich in regulatory interaction interfaces. A major class of disordered interaction interfaces are the compact and degenerate modules known as short linear motifs (SLiMs). As a result of the difficulties associated with the experimental identification and validation of SLiMs, our understanding of these modules is limited, advocating the use of computational methods to focus experimental discovery. This article evaluates the use of evolutionary conservation as a discriminatory technique for motif discovery. A statistical framework is introduced to assess the significance of relatively conserved residues, quantifying the likelihood a residue will have a particular level of conservation given the conservation of the surrounding residues. The framework is expanded to assess the significance of groupings of conserved residues, a metric that forms the basis of SLiMPrints (short linear motif fingerprints), a de novo motif discovery tool. SLiMPrints identifies relatively overconstrained proximal groupings of residues within intrinsically disordered regions, indicative of putatively functional motifs. Finally, the human proteome is analysed to create a set of highly conserved putative motif instances, including a novel site on translation initiation factor eIF2A that may regulate translation through binding of eIF4E.

Server available at bioware.ucd.ie.

New software downloads now available

Updated versions of all packages are now available for download. Unfortunately, due to limited time availability, the manuals are getting a little out of date with respect to all the available functions but the readme pages contain all the latest options and defaults. Please contact me if you find any bugs and/or want specific documentation improved. It is on the (long) list of things to try and get done over the summer!

Note that in a slight modification of previous releases, zip downloads now contain the creation date in the name (e.g. rjesuite.2012-07-26.tar.gz) and will be archived.

SLiMSuite servers and programs

An emerging field of biology is the role of intrinsically disordered regions in protein function and, specifically, protein-protein interactions (PPI) [1-2]. Of particular interest, Short, Linear Motifs (SLiMs) playing a vital role in disorder-mediated PPI, acting as ligands for molecular signalling, post-translational modifications and subcellular targeting [3]. SLiMs have extremely compact protein interaction interfaces, generally encoded by less than 4 major affinity-/specificity-determining residues within a stretch of 2-10 residues [4]. Their small size enables high functional density and evolutionary plasticity, which is frequently exploited by rapidly evolving pathogens that use them to hijack cellular processes [5]. These same features also make experimental discovery a challenge and considerable attention has therefore been given to computational methods for SLiM prediction and analysis [6].

A number of these tools have been developed by the Edwards and Shields labs [7-11] and made available as part of the SLiMSuite package and online as webservers (http://bioware.ucd.ie) [9-10,12-14], with two new tools, SLiMPrints and QSLiMFinder, currently in preparation for submission, and SLiMMaker to be added soon. The main tools that form the SLiMSuite package/servers are as follows:

• SLiMFinder [8,13]: de novo SLiM prediction based on a statistical model of over-represented motifs in unrelated proteins.
  
• SLiMDisc [7,12]: de novo SLiM prediction based on heuristic ranking of over-represented motifs in unrelated proteins.
  
• SLiMPred [11]: de novo SLiM/MoRF prediction in single proteins based machine learning of motif attributes.
  
• SLiMSearch [10]: biological context (disorder & conservation) for searches of pre-defined motifs with under- and over-representation statistics, correcting for evolutionary relationships.
  
• SLiMSearch 2.0 [14]: biological context (disorder & conservation) and ranking for proteome-wide searches of pre-defined motifs.
  
• SLiMPrints (in prep.): de novo SLiM/MoRF prediction in single proteins from statistical clustering of conserved disordered residues.
  
• QSLiMFinder (server coming soon): Query-based variant of SLiMFinder with increased sensitivity and specificity.
  
• CompariMotif [9]: Motif-motif comparison tool.
  
• SLiMMaker (coming soon): Simple tool for converting aligned peptides or SLiM occurrences into a regular expression motif.
  
• GOPHER [12]: Automated orthologue prediction and alignment algorithm. Used for conservation-based masking (SLiMFinder/SLiMSearch) and prediction (SLiMPrints).
  
• GABLAM [7] (server coming soon): BLAST-based protein similarity scoring and clustering. Used for SLiMFinder and SLiMSearch adjustments for evolutionary relationships.
  

Personnel (and funding applications) permitting, a number of improvements for these resources are planned, including updates to the underlying databases for proteome-wide predictions (SLiMSearch 1.0 & 2.0), conservation analyses (SLiMSearch 1.0 & 2.0, SLiMPrints, GOPHER) and SLiM comparisons (CompariMotif). We also intend to improve the integration of different tools, allowing seamless continuation of analyses. Motif predictions ((Q)SLiMFinder/SLiMPrints/SLiMPred) will be able to be searched directly against known motifs (CompariMotif) or proteomes (SLiMSearch); GOPHER alignments will be accessible for SLiMPrints analyses and even SLiMSearch/(Q)SLiMFinder input; outputs of motif occurrences ((Q)SLiMFinder/SLiMSearch) can be used to redefine motifs using SLiMMaker etc. If you have any other suggestions for improvements, please let us know.


References:

[1] Tompa P (2011) Unstructural biology coming of age. Curr Opin Struct Biol 21: 419; [2] Babu MM et al. (2011) Intrinsically disordered proteins: regulation and disease. Curr Opin Struct Biol 21:432; [3] Diella F et al. (2008) Understanding eukaryotic linear motifs and their role in cell signaling and regulation. Front Biosci 13:6580; [4] Davey NE et al. (2012) Attributes of short linear motifs. Mol Biosyst 8:268; [5] Davey NE, Trave G & Gibson TJ (2011) How viruses hijack cell regulation. Trends Biochem Sci 36:159; [6] Davey NE, Edwards RJ & Shields DC (2010) Computational identification and analysis of protein short linear motifs. Front Biosci 15:801; [7] Davey NE, Shields DC & Edwards RJ (2006): SLiMDisc: short, linear motif discovery, correcting for common evolutionary descent. Nucleic Acids Res. 34:3546; [8] Edwards RJ, Davey NE & Shields DC (2007): SLiMFinder: A probabilistic method for identifying over-represented, convergently evolved, short linear motifs in proteins. PLoS ONE 2:e967; [9] Edwards RJ, Davey NE & Shields DC (2008): CompariMotif: Quick and easy comparisons of sequence motifs. Bioinformatics 24:1307; [10] Davey NE et al. (2010): SLiMSearch: a webserver for finding novel occurrences of short linear motifs in proteins, incorporating sequence context. Lecture Notes in Bioinformatics 6282:50; [11] Mooney C et al. (2012): Prediction of short linear protein binding regions. J Mol Biol 415:193; [12] Davey NE, Edwards RJ & Shields DC (2007): The SLiMDisc server: short, linear motif discovery in proteins. Nuc Acids Res 35:W455; [13] Davey NE et al. (2010): SLiMFinder: a web server to find novel, significantly over-represented, short protein motifs. Nuc Acids Res 38:W534; [14] Davey NE et al. (2011): SLiMSearch 2.0: biological context for short linear motifs in proteins. Nuc Acids Res 39:W56.

SLiMMaker: regular expressions from aligned peptide sequences

SLiMMaker has a fairly simple function of reading in a set of sequences and generating a regular expression motif from them. It is designed with protein sequences in mind but should work for DNA sequences too. Input sequences can be in fasta format or just plain text (with no sequence headers) and should be aligned already. Gapped positions will be ignored (treated as Xs) and variable length wildcards are not returned.

SLiMMaker considers each column of the input in turn and compresses it into a regular expression element according to some simple rules, screening out rare amino acids and converting particularly degenerate positions into wildcards. Each amino acid in the column that occurs at least X times (as defined by minseq=X) is considered for the regular expression definition for that position. The full set of amino acids meeting this criterion is then assessed for whether to keep it as a defined position, or convert into a wildcard.

First, if the number of different amino acids meeting this criterion is zero or above a second threshold (maxaa=X), the position is defined as a wildcard. Second, the proportion of input sequences matching the amino acid set is compared to a minimum frequency criterion (minfreq=X). Failing to meet this minimum frequency will again result in a wildcard. Otherwise, the amino acid set is added to the SLiM definition as either a fixed position (if only one amino acid met the minseq criterion) or as a degenerate position. Finally, leading and trailing wildcards are removed.

By default, each defined position in a motif will contain amino acids that (a) occur in at least three sequences each, (b) have a combined frequency of >=75%, and (c) have 5 or fewer different amino acids (that occur in 3+ sequences).

Note. The final motif only contains defined positions that match a given frequency of the input (75% by default). Because positions are considered independently, however, the final motif might occur in fewer than 75% of the input sequences. Results will indicate the coverage of the input data but SLiMSearch can be used to check the occurrence stats more thoroughly.

Citation: SLiMMaker is part of the ongoing benchmarking of QSLiMFinder, which should be submitted for publication soon. In the meantime, please cite the SLiMMaker URL: http://bioware.soton.ac.uk/slimmaker.html.

Availability: SLiMMaker is available on request and will shortly be part of the SLiMSuite package.