python
Table of Content
About
Python is a dynamic object-oriented scripting language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools, comes with extensive standard libraries, and can be learned in a few days.
Versions and Availability
▶ Display Softenv Keys for python on all clusters
Machine | Version | Softenv Key |
---|---|---|
supermike2 | 2.7.3 | +Python-2.7.3-gcc-4.4.6 |
supermike2 | anaconda-3-4.0.0 | +Python-3.5.1-anaconda-3.4.0 |
▶ Softenv FAQ?
The information here is applicable to LSU HPC and LONI systems.
Shells
A user may choose between using /bin/bash and /bin/tcsh. Details about each shell follows.
/bin/bash
System resource file: /etc/profile
When one access the shell, the following user files are read in if they exist (in order):
- ~/.bash_profile (anything sent to STDOUT or STDERR will cause things like rsync to break)
- ~/.bashrc (interactive login only)
- ~/.profile
When a user logs out of an interactive session, the file ~/.bash_logout is executed if it exists.
The default value of the environmental variable, PATH, is set automatically using SoftEnv. See below for more information.
/bin/tcsh
The file ~/.cshrc is used to customize the user's environment if his login shell is /bin/tcsh.
Softenv
SoftEnv is a utility that is supposed to help users manage complex user environments with potentially conflicting application versions and libraries.
System Default Path
When a user logs in, the system /etc/profile or /etc/csh.cshrc (depending on login shell, and mirrored from csm:/cfmroot/etc/profile) calls /usr/local/packages/softenv-1.6.2/bin/use.softenv.sh to set up the default path via the SoftEnv database.
SoftEnv looks for a user's ~/.soft file and updates the variables and paths accordingly.
Viewing Available Packages
The command softenv will provide a list of available packages. The listing will look something like:
$ softenv These are the macros available: * @default These are the keywords explicitly available: +amber-8 Applications: 'Amber', version: 8 Amber is a +apache-ant-1.6.5 Ant, Java based XML make system version: 1.6. +charm-5.9 Applications: 'Charm++', version: 5.9 Charm++ +default this is the default environment...nukes /etc/ +essl-4.2 Libraries: 'ESSL', version: 4.2 ESSL is a sta +gaussian-03 Applications: 'Gaussian', version: 03 Gaussia ... some stuff deleted ...
Managing SoftEnv
The file ~/.soft in the user's home directory is where the different packages are managed. Add the +keyword into your .soft file. For instance, ff one wants to add the Amber Molecular Dynamics package into their environment, the end of the .soft file should look like this:
+amber-8
@default
To update the environment after modifying this file, one simply uses the resoft command:
% resoft
The command soft can be used to manipulate the environment from the command line. It takes the form:
$ soft add/delete +keyword
Using this method of adding or removing keywords requires the user to pay attention to possible order dependencies. That is, best results require the user to remove keywords in the reverse order in which they were added. It is handy to test out individual keys, but can lead to trouble if changing multiple keys. Changing the .soft file and issuing the resoft is the recommended way of dealing with multiple changes.
▶ Display Module Names for python on all clusters.
Machine | Version | Module |
---|---|---|
qb2 | 2.7.10-anaconda | python/2.7.10-anaconda |
qb2 | 2.7.12-anaconda-tensorflow | python/2.7.12-anaconda-tensorflow |
qb2 | 2.7.7-anaconda | python/2.7.7-anaconda |
qb2 | 3.5.2-anaconda-tensorflow | python/3.5.2-anaconda-tensorflow |
smic | 2.7.10-mkl-mic | python/2.7.10-mkl-mic |
smic | 2.7.13-anaconda-tensorflow | python/2.7.13-anaconda-tensorflow |
smic | 2.7.7 | python/2.7.7/GCC-4.9.0 |
smic | 2.7.7-anaconda | python/2.7.7-anaconda |
philip | 2.7.10-anaconda | python/2.7.10-anaconda |
philip | 2.7.7 | python/2.7.7/GCC-4.9.0 |
▶ Module FAQ?
The information here is applicable to LSU HPC and LONI systems.
Shells
A user may choose between using /bin/bash and /bin/tcsh. Details about each shell follows.
/bin/bash
System resource file: /etc/profile
When one access the shell, the following user files are read in if they exist (in order):
- ~/.bash_profile (anything sent to STDOUT or STDERR will cause things like rsync to break)
- ~/.bashrc (interactive login only)
- ~/.profile
When a user logs out of an interactive session, the file ~/.bash_logout is executed if it exists.
The default value of the environmental variable, PATH, is set automatically using SoftEnv. See below for more information.
/bin/tcsh
The file ~/.cshrc is used to customize the user's environment if his login shell is /bin/tcsh.
Modules
Modules is a utility which helps users manage the complex business of setting up their shell environment in the face of potentially conflicting application versions and libraries.
Default Setup
When a user logs in, the system looks for a file named .modules in their home directory. This file contains module commands to set up the initial shell environment.
Viewing Available Modules
The command
$ module avail
displays a list of all the modules available. The list will look something like:
--- some stuff deleted --- velvet/1.2.10/INTEL-14.0.2 vmatch/2.2.2 ---------------- /usr/local/packages/Modules/modulefiles/admin ----------------- EasyBuild/1.11.1 GCC/4.9.0 INTEL-140-MPICH/3.1.1 EasyBuild/1.13.0 INTEL/14.0.2 INTEL-140-MVAPICH2/2.0 --- some stuff deleted ---
The module names take the form appname/version/compiler, providing the application name, the version, and information about how it was compiled (if needed).
Managing Modules
Besides avail, there are other basic module commands to use for manipulating the environment. These include:
add/load mod1 mod2 ... modn . . . Add modules rm/unload mod1 mod2 ... modn . . Remove modules switch/swap mod . . . . . . . . . Switch or swap one module for another display/show . . . . . . . . . . List modules loaded in the environment avail . . . . . . . . . . . . . . List available module names whatis mod1 mod2 ... modn . . . . Describe listed modules
The -h option to module will list all available commands.
Module is currently available only on SuperMIC.
Usage
The Python language can be used like most other scripting languages. Issuing the command alone, python, starts a interactive pythonic shell session. If a Python program is contained in a file command-line execution is possible:
$ python [options] [-c cmd | -m mod | file | -] [arg [args ...]]
Use python --help for quick help, and see the documentation in the resource section for language details.
▶ Executable Scripts FAQ?
Executable Scripts
All shells and most interpreters (i.e. Python, Tcl, gawk, R, etc.) can be used to create executable scripts. This is accomplished through the magic of the shell #! line, which is a special comment line. The line, which must be the first line of the script, specifies the path to the program that can interpret the main contents of the file, which are command lines the language understands. Once the script is written, set it executable with chmod. A few examples should serve to illustrate the general idea.
Executable Bash Script
A simple script to echo out Hello World and do a long ls listing.
#!/bin/bash echo "Hello World!" ls -l
If the above is in a file named hello.sh, it can be made executable for the user (i.e. see man chmod for other possibilities) and run with:
$ chmod u+x hello.sh $ ./hello.sh
Other Scripts
Here are some line usable with other scripting languages. In most cases, options allowed by the command used can be provided on the #! line. Note: that the paths should be verified and not use blindly. In particular command shells tend to reside in different directories.
gawk: #!/bin/gawk tcsh: #!/usr/bin/tcsh tcl: #!/usr/local/bin/tcl
▶ QSub FAQ?
Portable Batch System: qsub
qsub
All HPC@LSU clusters use the Portable Batch System (PBS) for production processing. Jobs are submitted to PBS using the qsub command. A PBS job file is basically a shell script which also contains directives for PBS.
Usage
$ qsub job_script
Where job_script is the name of the file containing the script.
PBS Directives
PBS directives take the form:
#PBS -X value
Where X is one of many single letter options, and value is the desired setting. All PBS directives must appear before any active shell statement.
Example Job Script
#!/bin/bash # # Use "workq" as the job queue, and specify the allocation code. # #PBS -q workq #PBS -A your_allocation_code # # Assuming you want to run 16 processes, and each node supports 4 processes, # you need to ask for a total of 4 nodes. The number of processes per node # will vary from machine to machine, so double-check that your have the right # values before submitting the job. # #PBS -l nodes=4:ppn=4 # # Set the maximum wall-clock time. In this case, 10 minutes. # #PBS -l walltime=00:10:00 # # Specify the name of a file which will receive all standard output, # and merge standard error with standard output. # #PBS -o /scratch/myName/parallel/output #PBS -j oe # # Give the job a name so it can be easily tracked with qstat. # #PBS -N MyParJob # # That is it for PBS instructions. The rest of the file is a shell script. # # PLEASE ADOPT THE EXECUTION SCHEME USED HERE IN YOUR OWN PBS SCRIPTS: # # 1. Copy the necessary files from your home directory to your scratch directory. # 2. Execute in your scratch directory. # 3. Copy any necessary files back to your home directory. # Let's mark the time things get started. date # Set some handy environment variables. export HOME_DIR=/home/$USER/parallel export WORK_DIR=/scratch/myName/parallel # Set a variable that will be used to tell MPI how many processes will be run. # This makes sure MPI gets the same information provided to PBS above. export NPROCS=`wc -l $PBS_NODEFILE |gawk '//{print $1}'` # Copy the files, jump to WORK_DIR, and execute! The program is named "hydro". cp $HOME_DIR/hydro $WORK_DIR cd $WORK_DIR mpirun -machinefile $PBS_NODEFILE -np $NPROCS $WORK_DIR/hydro # Mark the time processing ends. date # And we're out'a here! exit 0
Resources
Last modified: August 21 2017 10:47:37.