Big Red usage policies

Username:
Password:

On this page:

Accounts

Access to Big Red is provided to all Indiana University faculty and graduate students, and faculty-sponsored undergraduates and staff. Instructional use is limited to courses that have been approved by the Director for Research Technologies.

Home directories

Home directories reside on the RAC storage system and have a default quota of 10GB. If you need additional permanent storage space, apply for an account on IU's Massive Data Storage Service (MDSS).

CPU limits and batch jobs

User processes on the login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated with no warning. You may run interactive jobs requiring more than 20 minutes of CPU time on nodes b509, b510, b511, and b512. There is a 24-hour CPU time limit on all user processes running on these nodes. If you require more than 24 hours of CPU time, submit a batch job to the LoadLeveler batch queuing system with the llsubmit command.

Computational resources

Each queue has properties defined in order to maximize job throughput and minimize time spent waiting in queue. Submit jobs to the queue most appropriate to the specific job requirements. If no queue is specified, jobs will go into the DEBUG queue.

Note: The total number of idle jobs you may have over all queues is 128. In the DEBUG queue, you may have only 1 idle job.

SERIAL queue properties

Nodes: 760
Maximum nodes per job: 1 (4 cores)
Maximum nodes per user: 64
Maximum wall time: 48 hours (2 days)
Maximum CPU time: 192 hours (48 hours * 4 cores)
Maximum running jobs in queue: Unlimited
Maximum idle jobs in queue: Up to 128 (see "Note" above)

NORMAL queue properties

Nodes: 392
Maximum nodes per job: 256 (1,024 cores)
Maximum nodes per user: 392
Maximum wall time: 48 hours (2 days)
Maximum CPU time: 48 hours (2 days) per core per job
Maximum running jobs in queue: Unlimited
Maximum idle jobs in queue: Up to 128 (see "Note" above)

Note: The NORMAL queue was formerly known as BIG. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

LONG queue properties

Nodes: 368
Maximum nodes per job: 32 (128 cores)
Maximum nodes per user: 64
Maximum wall time: 336 hours (14 days)
Maximum CPU time: 336 hours (14 days) per core per job
Maximum running jobs in queue: Unlimited
Maximum idle jobs in queue: Up to 128 (see "Note" at the top of this section)

Note: The LONG queue was formerly known as MED. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

DEBUG queue properties

Nodes: 4
Maximum nodes per job: 4 (16 cores)
Maximum nodes per user: 4
Maximum wall time: 15 minutes
Maximum CPU time: 15 minutes per core per job
Maximum running jobs in queue: 1
Maximum idle jobs in queue: 1

Note: The DEBUG queue was formerly known as FAST. An alias for the old name is in place for job submissions, but when you use commands like llclass and showq -w class=XXXXX, you must use the new name.

IEDC queue properties (group restricted access)

Nodes: 512
Maximum nodes per job: 256
Maximum nodes per user: 256
Maximum wall time: 336 hours (14 days)
Maximum CPU time: 344064 hours (14 days) per core per job
Maximum running jobs in queue: Unlimited
Maximum idle jobs in queue: Up to 128 (see "Note" above)

Important: Default wall time in all queues is two hours; request more if you need more. The default limit on running jobs is 16 per user. Please contact the administrators if you feel you need a higher limit.

Mail usage

UITS does not provide a production mail service on the Big Red cluster; however, LoadLeveler communicates via email. Mail forwarding should be configured during account creation, but if you are receiving mail on the cluster, you may need to set up a .forward file for your account. For help, see How do I forward my mail from a Unix account?

If you plan to run a large number of jobs through Big Red in a relatively short period of time and will be forwarding notifications to an external mail server, make sure the external mail server can handle the load. If you are not absolutely certain that the external email server can handle a large volume of mail, include this line in your jobs:

# @notification = never

Scheduled downtime

Big Red maintenance windows will be scheduled for the first Tuesday of each month. A window will be used only if necessary; an announcement regarding the status of maintenance will be posted the preceding Friday morning on both the Big Red Message of the Day and the Scheduled Downtime page.

Questions and comments

The policies described in this document were established with the goal of providing a stable, powerful, and efficient research computing environment for IU faculty, graduate students, and staff. If you have any questions or comments about these policies, UITS welcomes your input. Please contact High Performance Systems.

This is document avgm in domain all.
Last modified on February 03, 2009.