Mason at Indiana University

Username:
Password:

On this page:

System overview

Mason (mason.indiana.edu) at Indiana University is a large memory computer cluster configured to support data-intensive, high-performance computing tasks for researchers using genome assembly software (particularly software suitable for assembly of data from next-generation sequencers), large-scale phylogenetic software, or other genome analysis applications that require large amounts of computer memory. At IU, Mason accounts are available to IU faculty, postdoctoral fellows, research staff, and students involved in genome research. IU educators providing instruction on genome analysis software, and developers of such software, are also welcome to use Mason. IU has also made Mason available to genome researchers from the National Science Foundation's Extreme Science and Engineering Discovery Environment (XSEDE) project.

Mason consists of 16 Hewlett Packard DL580 servers, each containing four Intel Xeon L7555 8-core processors and 512 GB of RAM. The total RAM in the system is 8 TB. Each server chassis has a 10-gigabit Ethernet connection to the other research systems at IU and the XSEDE network (XSEDENet).

The Mason nodes run Red Hat Enterprise Linux 6.0. Job management is provided by the TORQUE resource manager (for more, see Running your applications below) in combination with the Moab job scheduler. Mason employs the Modules system to simplify application and environment configuration (for more, see Computing environment below).

Note: The scheduled monthly maintenance window for Mason is the first Tuesday of each month, 7am-7pm.

System information

System configuration	Aggregate information	Per-node information
Machine type	High-performance computing (data-intensive computing)
Operating system	Red Hat Enterprise Linux 6
Memory model	Distributed
Processor cores	512	32
CPUs	64 Intel Xeon L7555 8-core processors	4 Intel Xeon L7555 8-core processors
Nodes	16 Hewlett Packard DL580 servers
RAM	8 TB	512 GB
Network	10-gigabit Ethernet per node
Local storage	8 TB	500 GB
Computational systems details	Total	Per node
Processing capability	Rmax = 4,302 gigaflops	Rmax = 239 gigaflops
Benchmark data	HPL gigaflops 3129.75	HPL gigaflops 222.22
Power usage	0.000153 teraflops per watt	0.000173 teraflops per watt
Login nodes	Rmax = 478 gigaflops	Rmax = 239 gigaflops
Homogeneous compute nodes	Rmax = 3,824 gigaflops	Rmax = 239 gigaflops
Heterogeneous compute nodes (GPU or other accelerator)	Rmax = 3,824 gigaflops

File systems (storage for IU users)

File system	Description	Total disk space	Quotas	Backups
`/N/u/username/Mason`	User home directories reside on a network-attached storage (NAS) device. The path to your home directory on Mason includes your Network ID username (`username`).	8 TB	10 GB; shared, if applicable, between user's accounts on Big Red, Quarry, and the Research Database Complex (RDC)	The home directory file system on Mason makes daily "hourly" and "nightly" snapshots, and saves them to hidden `.snapshot` directories. To recover a file, go to the directory where the file existed, and then enter its `.snapshot` directory. Additionally, the High Performance Systems (HPS) group makes weekly backups. If you don't find your file among the snapshots, email the HPS group.
`/scratch`	Local scratch space	450 GB	n/a	Local scratch space is not intended for permanent storage of data, and is not backed up. Files are automatically deleted once they are 14 days old.
`/N/dc/scratch/username`	Shared scratch space is hosted on the IU Data Capacitor. The path to your scratch space on the Data Capacitor contains your Network ID username (`username`).	427 TB	n/a	Shared scratch space is not intended for permanent storage of data, and is not backed up. Files in scratch space may be purged if they have not been accessed for more than 60 days.

IU graduate students, faculty, and staff who need more than 10 GB of permanent storage can apply for accounts on the Research File System (RFS) and the Scholarly Data Archive (SDA). See At IU, how can I apply for an account on the SDA or RFS?

Note: Indiana University will soon replace its current Data Capacitor with Data Capacitor II, a high-speed, high-capacity storage facility for very large data sets. With 5 PB of storage, Data Capacitor II will support big data applications used in computational research. IU partnered with DataDirect Networks, Inc. (DDN) to develop Data Capacitor II, which is scheduled to be installed in the IU Data Center in spring 2013. For more about Data Capacitor II, see the November 8, 2012, press release. If you have questions about how the change to Data Capacitor II will affect your research, email the High Performance File Systems group.

System access

Access policy

Although Mason is an IU resource dedicated to genome analysis research, access to the cluster is not restricted to IU researchers:

At IU, students, faculty, and staff can request accounts on Mason via the Account Management Service (AMS); see At IU, if I already have some computing accounts, how do I get others?
NSF-funded life sciences researchers at other institutions can apply to the National Center for Genome Analysis Support (NCGAS) allocations committee to request access to Mason. To request an allocation, submit the NCGAS Allocations Request Form. If you have questions, email NCGAS.
Access to Mason is also available to Extreme Science and Engineering Discovery Environment (XSEDE) researchers through the normal XSEDE allocation process.

For information about your responsibilities as a user of this resource, see:

IU and NCGAS users: What are my responsibilities as a computer user at IU?
XSEDE users: What are my responsibilities as an XSEDE user?

Logging into Mason

IU users use their IU Network IDs to log into Mason.

Note: Mason login nodes use the IU Active Directory Service for user authentication. As a result, local passwords/passphrases are not supported. For information about changing your ADS passphrase, see At IU, how do I change my Network ID passphrase? For helpful information regarding secure passphrases, see Passwords and passphrases.

Researchers with NCGAS allocations authenticate with the credentials associated with their NCGAS allocations.

Researchers with XSEDE allocations authenticate with their XSEDE-wide logins.

Use of public key authentication is also permitted on Mason. For more, see In SSH and SSH2 for Unix, how do I set up public key authentication?

Methods of access

For IU and NCGAS users, interactive access to Mason is provided via SSH only. Use SSH2 clients to connect to mason.indiana.edu , which resolves to either h1.mason.indiana.edu or h2.mason.indiana.edu , Mason's two login nodes.

XSEDE users are encouraged to use GSI-SSH from the XSEDE User Portal or one of the desktop clients. SSH access via public key authentication is also permitted. For more, see Access Resources on the XSEDE User Portal.

File transfer is supported via SCP and SFTP. For more, see At IU, what SSH/SFTP clients are supported and where can I get them?

Available software

Software installed on Mason is made available to users via Modules, an environment management system that lets you easily and dynamically add software packages to your user environment. For a list of software modules available on Mason, see Mason Modules in the IU Cyberinfrastructure Gateway.

For more on Modules, see Modules below.

For more on the IU Cyberinfrastructure Gateway, see What is the IU Cyberinfrastructure Gateway?

Note: Users can install software in their home directories on Mason and request the installation of software for use by all users on the system. Only faculty or staff can request software. If students require software packages on Mason, their advisors must request them. For more, see At IU, what is the policy about installing software on Mason?

Computing environment

Unix shell

The shell is the primary method of interacting with the Mason cluster. The command line interface provided by the shell lets users run built-in commands, utilities installed on the system, and even short ad hoc programs.

Mason supports the Bourne-again (bash) and TC (tcsh) shells. New user accounts are assigned the bash shell by default. For more on bash, see the Bash Reference Manual and the Bash (Unix shell) Wikipedia page.

To change your shell on Mason, use the changeshell command.

Note: Running chsh (instead of changeshell) changes your shell only on the node on which you run it, and leaves the other nodes of the cluster unchanged; changeshell prompts you with the shells available on the system, and changes your login shell system-wide within 15 minutes.

Environment variables

The shell uses environment variables primarily to modify shell behavior and the operation of certain commands. A good example is the PATH variable.

When the shell parses a command you have entered (i.e., after you hit Enter or Return), it interprets certain words you've typed as program files that should be executed. The shell then searches various directories on the system to locate these files. The PATH variable determines which directories are searched, and the order in which they are searched. In the bash shell, the PATH variable is a string of directories separated by colons (e.g., /bin:/usr/bin:/usr/local/bin). The shell searches for an executable file in the /bin directory, then the /usr/bin directory, and finally the /usr/local/bin directory. If files of the same name (e.g., foo) exist in all three directories, /bin/foo will be run, because the shell will find it first.

To display and change the values of environment variables:

Shell	Display a value	Change a value
`bash`	`echo $VARNAME`	`export VARNAME=VALUE`
`tcsh`	`echo $VARNAME`	`setenv VARNAME VALUE`

Startup scripts

Shells offer much flexibility in terms of startup configuration. On login, bash by default reads and executes commands from the following directories (and in this order):

/etc/profile ~/.bash_profile ~/.bashrc

Note: The ~ (tilde) represents your home directory (e.g., ~/.bash_profile is the .bash_profile file in your home directory).

On logout, the shell reads and executes ~/.bash_logout. For more on bash startup files, see the "Bash Startup Files" section of the Bash Reference Manual.

On login, the tcsh shell reads and executes commands from the following directories (also in this order):

/etc/csh.cshrc /etc/csh.login ~/.tcshrc (if it exists, otherwise ~/.cshrc) ~/.history ~/.login ~/.cshdirs

In practice, on Mason, only the first two files exist. You may create the others, and add commands and variables to them as you see fit.

Modules

Mason use the Modules package to provide a convenient method for dynamically modifying your environment.

Some common Modules commands include:

Command	Action
`module avail`	List all software packages available on the system.
`module avail package`	List all versions of `package` available on the system, for example: module avail openmpi
`module list`	List all packages currently loaded in your environment.
`module load package/version`	Add the specified `version` of the `package` to your environment, for example: module load intel/11.1 To load the default version of the `package`, use: module load package
`module unload package`	Remove the specified `package` from your environment.
`module swap package_A package_B`	Swap the loaded package (`package_A`) with another package (`package_B`). This is synonymous with: module switch package_A package_B
`module show package`	Shows what changes will be made to your environment (e.g., paths to libraries and executables) by loading the specified `package`. This is synonymous with: module display package

For more about the Modules package, see the module manual page and the modulefile manual page.

For information about using Modules on IU research systems, see On IU's Mason and Quarry clusters, how do I use Modules to manage my software environment? For information about using Modules on XSEDE digital services, see On XSEDE, how do I manage my software environment using Modules?

Transferring your files to Mason

Presently, Mason supports only SCP and SFTP for transferring files. SCP is a command line utility included with OpenSSH. Basic use is:

scp username@host1:file1 username@host2:file2

For example, to copy foo.txt from the current directory on your computer to your home directory on Mason, use (replacing username with your username): scp foo.txt username@mason.indiana.edu:foo.txt

You may specify absolute paths or paths relative to your home directory:

scp foo.txt username@mason.indiana.edu:some/path/for/data/foo.txt

You also may leave the destination filename unspecified, in which case it will become the same as the source filename. For more, see In Unix, how do I use SCP to securely transfer files between two computers?

SFTP clients provide file access, transfer, and management, and offer functionality similar to FTP clients. For example, using a command-line SFTP client (e.g., from a Linux or Mac OS X workstation), you could transfer files as follows:

$ sftp username@mason.indiana.edu username@mason.indiana.edu's password: Connected to mason.indiana.edu. sftp> ls -l -rw------- 1 username group 113 May 19 2011 loadit.pbs.e897 -rw------- 1 username group 695 May 19 2011 loadit.pbs.o897 -rw-r--r-- 1 username group 693 May 19 2011 local_limits sftp> put foo.txt Uploading foo.txt to /N/hd00/username/Mason/foo.txt foo.txt 100% 43MB 76.9KB/s 09:39 sftp> exit $

For more, see What is SFTP, and how do I use an SFTP client to transfer files?

Application development

Programming models

Mason is designed to support codes that have extremely large memory requirements. As these codes typically do not implement a distributed memory model, Mason is geared toward a serial or shared-memory parallel programming paradigm. However, Mason can support distributed memory parallelism.

Compiling

The GNU Compiler Collection (GCC) is added by default to your user environment on Mason. The Intel and Portland Group (PGI) compiler collections, and the Open MPI and MPICH wrapper compilers, are also available.

Recommended optimization options are -O3 and -xHost (the -xHost option will optimize based on the processor of the current host).

For the GCC compilers, the mtune=native and march=native options are recommend to generate instructions for the machine and CPU type.

Examples

Serial programs:

To compile the C program simple.c :
- With the GCC compiler: gcc -O2 -o -mtune=native -march=native simple simple.c
- With the Intel compiler: icc -o simple simple.c
To compile the Fortran program simple.f :
- With the GCC compiler: g77 -o simple simple.f
- With the Intel compiler: ifort -O2 -o simple -lm simple.f

Parallel programs:

To compile the C program simple.c with the MPI wrapper script: mpicc -o simple simple.c
To compile the Fortran program simple.f with the MPI wrapper script: mpif90 -o simple -O2 simple.f
To use the GCC C compiler to compile simple.c to run in parallel using OpenMP: gcc -O2 -fopenmp -o simple simple.c
To use the Intel Fortran compiler to compile simple.f to run in parallel using OpenMP: ifort -openmp -o simple -lm simple.f

Libraries

Both the Intel Math Kernel Library (MKL) and the AMD Core Math Library (ACML) are available on Mason.

Debugging

Both the Intel Debugger (IDB) and the GNU Project Debugger (GDB) are available on Mason.

For information about using the IDB, see the Intel IDB page.

For information about using the GDB, see the GNU GDB page. For an example, see Step-by-step example for using GDB within Emacs to debug a C or C++ program.

Running your applications

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated without warning. If you require more than 20 minutes of CPU time, use the TORQUE qsub command to submit a batch job (see Submitting a job below).

Short definitions

A job is an instance of an application you wish to run.
A queue is a pool of compute resources that accepts job to run, and executes them according to a fairshare policy.
A job scheduler is the application responsible for scheduling jobs.

Queues

BATCH: The batch queue is the default, general-purpose queue on Mason. The default walltime is one hour; the maximum limit is two weeks. If your job requires more than two weeks of walltime, email the High Performance Systems group for assistance.
SHARED: Mason's shared queue is for jobs that require relatively little memory and brief walltime. It allows jobs from multiple users to run simultaneously on a node. To use the shared queue either add the -q shared flag to your qsub command, or include #PBS -q shared in your job script.

Queue policies

The Moab job scheduler uses fairshare scheduling to track usage and prioritize jobs. For information on fairshare scheduling and using Moab to check the status of your jobs on Mason, see What is Moab? For a summary of commands, see Common Moab scheduler commands.

Submitting a job

To submit a job to run on Mason, use the qsub command. If the command exits successfully, it will return a job ID, for example:

[jdoe@Mason]$ qsub job.script 123456.m1.mason [jdoe@Mason]$

If you need attribute values different from the defaults, but less than the maximum allowed, specify these either in the job script using TORQUE directives, or on the command line with the -l switch. For example, to submit a job that needs more than the default 60 minutes of walltime, use:

qsub -l walltime=10:00:00 job.script

Jobs on Mason default to a per-job virtual memory resource of 8 MB. So, for example, to submit a job that needs 100 GB of virtual memory, use:

qsub -l nodes=1:ppn=4,vmem=100gb job.script

Note: Command-line arguments override directives in the job script, and you may specify many attributes on the command line, either as comma-separated options following the -l switch, or each with its own -l switch. The following two commands are equivalent:

qsub -l nodes=1:ppn=16,vmem=1024mb job.script qsub -l nodes=1:ppn=16 -l vmem=1024mb job.script

Useful qsub options include:

Option	Action
`-a <date_time>`	Execute the job only after specified date and time.
`-I`	Run the job interactively. (Interactive jobs are forced to not re-runnable.)
`-m e`	Mail a job summary report when the job terminates.
`-q <queue name>`	Specify the destination queue for the job. (Not applicable on Mason.)
`-r [y\|n]`	Declare whether the job is re-runnable. Use the argument `n` if the job is not re-runnable. The default value is `y` (re-runnable).
`-V`	Export all environment variables in your current environment to the job.

For more about the qsub command in TORQUE, see the Adaptive Computing qsub page and the qsub manual page.

Monitoring a job

To monitor the status of a queued or running job, use the TORQUE qstat command. Useful qstat options include:

Option	Action
`-a`	Display all jobs.
`-f`	Write a full status display to standard output.
`-n`	List the nodes allocated to a job.
`-r`	Display jobs that are running.
`-u user1@host,user2@host`	Display jobs owned by specified users.

For more about the qstat command in TORQUE, see the Adaptive Computing qstat page and the qstat manual page.

Deleting a job

To delete a queued or running job, use the qdel command.

Occasionally, a node will become unresponsive and unable to respond to the TORQUE server's requests to kill a job. In such cases, try using qdel -W <delay> to override the delay between SIGTERM and SIGKILL signals (for <delay>, specify a value in seconds).

For more about the qdel command in TORQUE, see the Adaptive Computing qdel page and the qdel manual page.

Reference

Intel Xeon Processor L7555 specifications
DDR3 memory technology (an HP white paper)

Support

For IU and NCGAS users

Support for IU and NCGAS users is provided by the UITS High Performance Systems (HPS) and Scientific Applications and Performance Tuning (SciAPT) groups, and by the National Center for Genome Analysis Support (NCGAS):

If you have system-specific questions about Mason, email the HPS group.
If you have questions about compilers, programming, scientific/numerical libraries, or debuggers on Mason, email the SciAPT group.
If you need help installing software packages in your home directory on Mason, email NCGAS.

For XSEDE users

XSEDE users with questions about hardware or software on Mason should contact the XSEDE Help Desk. For more, see How do I get help with XSEDE?

For more about XSEDE compute, advanced visualization, storage, and special purpose systems, see the Resources Overview, Systems Monitor, and User Guides. For scheduled maintenance windows, outages, and other announcements related to XSEDE digital services, see User News.

This document was developed with support from National Science Foundation (NSF) grant OCI-1053575. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.

This is document bbhh in domains all and xsede-all.
Last modified on May 10, 2013.