Available access to allocated and short-term storage capacity on IU's research systems

On this page:


Allocated storage and its uses

Important:

Before storing data on any of Indiana University's research computing or storage systems, make sure you understand the information in Types of sensitive institutional data appropriate for UITS Research Technologies services.

Make sure you do not include sensitive institutional data as part of a file's filename or pathname.

Indiana University provides multiple storage options to store, share, and collaborate on files on IU research computing resources. As a member of the IU research community, you have access to allocated storage on the following Research Technologies systems.

Allocated resource Description Default capacity Default file limit
Home directory space on IU research supercomputers
  • Mounted on IU's research supercomputers
  • Allocated with research supercomputer accounts
  • Disk-based, replicated, and highly secure
  • Relatively slow I/O throughput
  • For more, see below.
100 GB 800,000 files
Scholarly Data Archive (SDA)
  • HPSS data archive
  • Tape-based, long-term storage
  • Allocations are available to IU graduate students, faculty, and staff (undergraduate students and sponsored affiliates need either faculty or staff sponsors to request accounts)
  • For more, see below.
50 TB 25,000 files
Slate
  • Persistent storage for scholarly data
  • Lustre-based, high-throughput, parallel file system
  • Disk-based and highly secure
  • No purge policy
  • Data stored on Slate are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

  • Allocations are available to all IU research supercomputer users
  • For more, see below.
800 GB 6.4 million objects
Slate-Project
  • Centralized project space supporting extreme, data-intensive, performance-demanding workflows running on IU research supercomputers
  • Lustre-based, high-throughput, parallel file system
  • Disk-based and highly secure
  • No purge policy
  • Data stored on Slate-Project are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

  • Allocated with approval from UITS High Performance File Systems (HPFS) allocations committee
  • For more, see below.
Up to 15 TB No file limit
Slate-Scratch
  • Centralized project space supporting extreme, data-intensive, performance-demanding workflows running on IU research supercomputers
  • Lustre-based, high-throughput, parallel file system
  • Hybrid of Flash and disk-based, highly secure
  • 30-day purge policy
  • Data stored on Slate-Scratch are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

  • Allocations are automatically provisioned with the creation of a supercomputer account
  • For more, see below.
Up to 100 TB 10 million
Geode-Project
  • Fee-based project space
  • Disk-based, replicated, and highly secure
  • Mounted on IU research supercomputers
  • Mountable/accessible from campus computers
  • A memo of understanding (MOU) with UITS Research Technologies is required
  • For more, see below.
No default (depends on MOU) 800,000 files per every 100 GB (up to a maximum of 80 million for a "standard" allocation)

Home directories

By default, you are allotted 100 GB of home directory disk storage (with a maximum file limit of 800,000 files) when you create an account on any of IU's research supercomputers.

To request an account on an Indiana University research system, see Get additional IU computing accounts. Account availability depends on your eligibility.

Home directories are neither intended for storing large data sets nor capable of handling data-intensive computational I/O; other Research Technologies systems are provided for those purposes. Home directories are relatively small by design, as they are intended for low-capacity, low-performance uses (such as storing parameter or configuration files and compiling binaries) only.

When in support of research activities, an increase in capacity (along with a proportional increase in the file limit) may be granted. If you need additional home directory disk space, and other allocated storage options do not meet your needs, email the UITS Research Storage team (store-admin@iu.edu).

Scholarly Data Archive

An account on IU's Scholarly Data Archive (SDA) provides up to 50 TB of long-term, tape-based, fully-replicated archival storage. A maximum file limit of 25,000 files is enforced for new accounts.

The Indiana University Scholarly Data Archive (SDA) provides extensive capacity (approximately 79 PB of tape overall) for storing and accessing research data. The SDA is a distributed storage service co-located at IU data centers in Bloomington and Indianapolis, providing IU researchers with large-scale archival or near-line data storage, arranged in large files, with two copies of data made by default (for disaster recovery).

For eligibility requirements, see the "Research system accounts (all campuses)" section in Computing accounts at IU.

After submitting your account request, UITS will notify you via email when your account is ready for use.

Note:
In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access data containing protected health information (PHI) using a group (or departmental) account. To ensure accountability and maintain appropriate levels of access control, all users must use an individual login for all work involving PHI.

When in support of research activities, extensions beyond 50 TB may be granted for a nominal charge; for more, see Fee-based research storage below.

Slate

The default quota allotment is 800 GB per user. Upon request, your quota may be increased to a maximum of 1.6 TB. To request more space on Slate, contact the UITS High Performance File Systems (HPFS) group using the Research Technologies contact form (from the "Choose an area to direct your question to" drop-down, select High performance storage). Additionally, an inode quota (sometimes called "file quota") limits the number of objects a single user can create to 6.4 million.

Slate is a centralized, high performance Lustre file system designed for the persistent storage of scholarly data to meet the needs of data-intensive workflows and analytics running on Indiana University's research supercomputers. Space on Slate is not subject to a purge policy.

Data stored on Slate are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

For more, see About Slate high performance storage for research computation at IU.

Space on Slate is available to all IU research supercomputer users. To create a Slate account, follow the instructions in Get additional IU computing accounts.

Slate-Project

Slate-Project allocations are available to IU researchers who need shared/project spaces or more storage capacity than the 1.6 TB available per user on Slate. Requests for fewer than 15 TB are granted without fee.

The Slate-Project high performance file system is a centralized storage environment supporting extreme, data-intensive, performance-demanding (Big Data) workflows running on IU's research supercomputers.

Data stored on Slate-Project are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

For more, see About Slate-Project high performance project space at IU.

To get an allocation on the Slate-Project file system, fill out and submit the Slate-Project request form. Once your request is approved, you will be able to create a Slate-Project account using the instructions in Get additional IU computing accounts.

Slate-Project allocations of 15 TB or more are available from UITS Research Technologies as a direct-bill service; for more, see Fee-based research storage below.

Geode-Project

Geode-Project is a fee-based Research Technologies service providing disk-based persistent storage allocations to research projects using Indiana University's research supercomputers. Geode-Project allocations are hosted on Geode, which is co-located at the IU data centers in Bloomington and Indianapolis. Files stored in Geode-Project space are replicated, by default, at each data center. Project space users can access Geode-Project directly from all IU research supercomputers and remotely from personal workstations connected to the IU campus network.

Geode-Project allocations require a memo of understanding (MOU) with UITS Research Technologies; for more, see Fee-based research storage below.

Unallocated temporary storage and its uses

As a member of the IU research community, you have access to unallocated temporary storage in local and shared scratch space managed by UITS Research Technologies. Scratch space is intended only for temporary storage of computational data. Data are not backed up. Files are regularly purged once they reach a certain age, and cannot be recovered. Users are responsible for moving data they want to keep to permanent storage solutions.

Local scratch space

The following amounts of local scratch space are accessible to users with accounts on the specified systems:

System Local scratch space available
Quartz 1.7 TB accessible at /tmp

Files in local scratch directories are automatically deleted once they are 10 days old; purged files cannot be recovered.

Shared scratch space on Slate-Scratch

Shared scratch space is hosted on the Slate-Scratch file system.

Slate-Scratch is a large-capacity, high-throughput, high-bandwidth Lustre-based file system designed for the temporary storage of computational data to meet the needs of data-intensive workflows and analytics running on Indiana University's research supercomputers.

Slate-Scratch directories are created automatically for all users with accounts on IU's research supercomputers. If you have an account on an IU research supercomputer, your Slate-Scratch directory is mounted at /N/scratch/username (replace username with your IU username).

Each IU research supercomputer user is allowed to store up to 100 TB of data on Slate-Scratch. An inode quota limits the number of files and directories a single user can create to 10 million.

Space on Slate-Scratch is not intended for permanent storage. Files in scratch space will be purged if they have not been accessed for more than 30 days. Users are responsible for archiving their data. To archive scratch space data, move files to the Scholarly Data Archive (SDA); see Access the SDA at IU.

Data stored on Slate-Scratch are not backed up in any way, shape, or form by Research Technologies or any other entity in UITS.

For more, see:

Fee-based research storage

Schools, departments, labs, and individual investigators with storage needs beyond those met by baseline (no fee) services may arrange direct-bill agreements with UITS Research Technologies for fee-based storage on Research Technologies systems:

To find fee information for Research Technologies services, see the "IU Internal Research Technologies Services" section of UITS Rates for Direct-Bill Services. If you are interested in fee-based storage services, contact UITS Research Technologies to request a meeting.

Access your allocated research storage space(s)

Home directories

Provided you have the required account(s), you can access your home directory space from any of IU's research supercomputers (in the following examples, replace username with the appropriate individual or group IU username):

System Home directory path
Big Red 200 /N/u/username/BigRed200
Quartz /N/u/username/Quartz

From your home directory space on one system, you can access your home directory space on another (provided you have accounts on both systems). For example, if you (username) have accounts on Quartz and Big Red 200, you can access your home directory on Big Red 200 from your home directory on Quartz; on the command line, enter:

cd ../BigRed200

As a result, the command prompt will change to reflect your present location within the file system; for example:

[username@h2 ~]$ cd ../BigRed200
[username@h2 BigRed200]$

Slate accounts

Your Slate account is accessible from each IU research supercomputer at /N/slate/username (replace username with your IU username).

Slate-Project space

Your Slate-Project space is accessible from each IU research supercomputer at /N/project/project_name (replace project_name with the name of your project).

SDA accounts

Methods available for transferring data to and from the Indiana University Scholarly Data Archive (SDA) include secure FTP (SFTP), secure copy (SCP), GridFTP (via the IU Globus Web App), and Hierarchical Storage Interface (HSI). For instructions, see:

Important:

Files containing PHI must be encrypted when they are stored (at rest) and when they are transferred between networked systems (in transit). To ensure that files containing PHI are encrypted when they are stored, encrypt them before transferring them to storage. To ensure that files containing PHI remain encrypted during transit, use SFTP/SCP or the IU Globus Web App. For more, see Recommended tools for encrypting data containing HIPAA-regulated PHI.

This is document avkm in the Knowledge Base.
Last modified on 2024-01-25 09:48:47.