Getting started on DARWIN

DARWIN (Delaware Advanced Research Workforce and Innovation Network) is a big data and high performance computing system designed to catalyze Delaware research and education funded by a $1.4 million grant from the National Science Foundation (NSF). This award will establish the DARWIN computing system as an XSEDE Level 2 Service Provider in Delaware contributing 20% of DARWIN's resources to XSEDE: Extreme Science and Engineering Discovery Environment now transitioned to ACCESS: Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support as of September 1, 2022. DARWIN has 105 compute nodes with a total of 6672 cores, 22 GPUs, 100TB of memory, and 1.2PB of disk storage. See compute nodes and storage for complete details on architecture.

Figure 1. Fish-eye front view of DARWIN in the computing center

Configuration

The DARWIN cluster is being set up to be very similar to the existing Caviness cluster, and will be familiar to those currently using Caviness. However DARWIN is a NSF funded HPC resource available via committee reviewed allocation request process similar to ACCESS allocations.

An HPC system always has one or more public-facing systems known as login nodes. The login nodes are supplemented by many compute nodes which are connected by a private network. One or more head nodes run programs that manage and facilitate the functioning of the cluster. (In some clusters, the head node functionality is present on the login nodes.) Each compute node typically has several multi-core processors that share memory. Finally, all the nodes share one or more filesystems over a high-speed network.

Figure 2. DARWIN Configuration

Login nodes

Login (head) nodes are the gateway into the cluster and are shared by all cluster users. Their computing environment is a full standard variant of Linux configured for scientific applications. This includes command documentation (man pages), scripting tools, compiler suites, debugging/profiling tools, and application software. In addition, the login nodes have several tools to help you move files between the HPC filesystems and your local machine, other clusters, and web-based services.

Login nodes should be used to set up and submit job workflows and to compile programs. You should generally use compute nodes to run or debug application software or your own executables.

If your work requires highly interactive graphics and animations, these are best done on your local workstation rather than on the cluster. Use the cluster to generate files containing the graphics information, and download them from the HPC system to your local system for visualization.

When you use SSH to connect to darwin.hpc.udel.edu your computer will choose one of the login (head) nodes at random. The default command line prompt clearly indicates to which login node you have connected: for example, [bjones@login00.darwin ~]$ is shown for account bjones when connected to login node login00.darwin.hpc.udel.edu.

Only use SSH to connect to a specific login node if you have existing processes present on it. For example, if you used the screen or tmux utility to preserve your session after logout.

Compute nodes

There are many compute nodes with different configurations. Each node consists of multi-core processors (CPUs), memory, and local disk space. Nodes can have different OS versions or OS configurations, but this document assumes all the compute nodes have the same OS and almost the same configuration. Some nodes may have more cores, more memory, GPUs, or more disk.

All compute nodes are now available and configured for use. Each compute node has 64 cores, so the compute resources available are the following:

The standard Linux on the compute nodes are configured to support just the running of your jobs, particularly parallel jobs. For example, there are no man pages on the compute nodes. All compute nodes will have full development headers and libraries.

Commercial applications, and normally your programs, will use a layer of abstraction called a programming model. Consult the cluster specific documentation for advanced techniques to take advantage of the low level architecture.

Storage

Home filesystem

Each DARWIN user receives a home directory (/home/<uid>) that will remain the same during and after the early access period. This storage has slower access with a limit of 20 GiB. It should be used for personal software installs and shell configuration files.

Lustre high-performance filesystem

Lustre is designed to use parallel I/O techniques to reduce file-access time. The Lustre filesystems in use at UD are composed of many physical disks using RAID technologies to give resilience, data integrity, and parallelism at multiple levels. There is approximately 1.1 PiB of Lustre storage available on DARWIN. It uses high-bandwidth interconnects such as Mellanox HDR100. Lustre should be used for storing input files, supporting data files, work files, and output files associated with computational tasks run on the cluster.

• Each allocation will be assigned a workgroup storage in the Lustre directory (/lustre/«workgroup»/).

• Each workgroup storage will have a users directory (/lustre/«workgroup»/users/«uid») for each user of the workgroup to be used as a personal directory for running jobs and storing larger amounts of data.

• Each workgroup storage will have a software and VALET directory (/lustre/«workgroup»/sw/ and /lustre/«workgroup»/sw/valet) all allow users of the workgroup to install software and create VALET package files that need to be shared by others in the workgroup.

• There will be a quota limit set based on the amount of storage approved for your allocation for the workgroup storage.

While all filesystems on the DARWIN cluster utilize hardware redundancies to protect data, there is no backup or replication and no recovery available for the home or Lustre filesystems.

Local filesystems

Each node has an internal, locally connected disk. Its capacity is measured in terabytes. Each compute node on DARWIN has a 1.75 TiB SSD local scratch filesystem disk. Part of the local disk is used for system tasks such memory management, which might include cache memory and virtual memory. This remainder of the disk is ideal for applications that need a moderate amount of scratch storage for the duration of a job's run. That portion is referred to as the node scratch filesystem.

Each node scratch filesystem disk is only accessible by the node in which it is physically installed. The job scheduling system creates a temporary directory associated with each running job on this filesystem. When your job terminates, the job scheduler automatically erases that directory and its contents.

Software

There will not be a full set of software during early access and testing, but we will be continually installing and updating software. Installation priority will go to compilers, system libraries, and highly utilized software packages. Please DO let us know if there are packages that you would like to use on DARWIN, as that will help us prioritize user needs, but understand that we may not be able to install software requests in a timely manner.

Users will be able compile and install software packages in their home or workgroup directories. There will be very limited support for helping with user compiled installs or debugging during early access. Please reference basic software building and management to get started with software installations utilizing VALET (versus Modules) as suggested and used by IT RCI staff on our HPC systems.

Please review the following documents if you are planning to compile and install your own software

• High Performance Computing (HPC) Tuning Guide for AMD EPYC™ 7002 Series Processors guide for getting started tuning AMD 2nd Gen EPYC™ Processor based systems for HPC workloads. This is not an all-inclusive guide and some items may have similar, but different, names in specific OEM systems (e.g. OEM-specific BIOS settings). Every HPC workload varies in its performance characteristics. While this guide is a good starting point, you are encouraged to perform your own performance testing for additional tuning. This guide also provides suggestions on which items should be the focus of additional, application-specific tuning (November 2020).

• HPC Tuning Guide for AMD EPYC™ Processors guide intended for vendors, system integrators, resellers, system managers and developers who are interested in EPYC system configuration details. There is also a discussion on the AMD EPYC software development environment, and we include four appendices on how to install and run the HPL, HPCG, DGEMM, and STREAM benchmarks. The results produced are ‘good' but are not necessarily exhaustively tested across a variety of compilers with their optimization flags (December 2018).

• AMD EPYC™ 7xx2-series Processors Compiler Options Quick Reference Guide, however we do not have the AOCC compiler (with Flang - Fortran Front-End) installed on DARWIN.

Scheduler

DARWIN will being using the Slurm scheduler like Caviness, and is the most common scheduler among ACCESS resources. Slurm on DARWIN is configured as fair share with each user being given equal shares to access the current HPC resources available on DARWIN.

Queues (Partitions)

Partitions have been created to align with allocation requests moving forward based on different node types. There will be no default partition, and must only specify one partition at a time. It is not possible to specify multiple partitions using Slurm to span different node types.

Run Jobs

In order to schedule any job (interactively or batch) on the DARWIN cluster, you must set your workgroup to define your cluster group. Each research group has been assigned a unique workgroup. Each research group should have received this information in a welcome email. For example,

will enter the workgroup for it_css. You will know if you are in your workgroup based on the change in your bash prompt. See the following example for user bjones

Now we can use salloc or sbatch as long as a partition is specified as well to submit an interactive or batch job respectively. See DARWIN Run Jobs, Schedule Jobs and Managing Jobs wiki pages for more help about Slurm including how to specify resources and check on the status of your jobs.

All resulting executables (created via your own compilation) and other applications (commercial or open-source) should only be run on the compute nodes.

It is a good idea to periodically check in /opt/shared/templates/slurm/ for updated or new templates to use as job scripts to run generic or specific applications designed to provide the best performance on DARWIN.

Help

ACCESS allocations

To report a problem or provide feedback, submit a help desk ticket on the ACCESS Portal and complete the form selecting darwin.udel.xsede.org as the system and your problem details in the description field to route your question more quickly to the research support team. Provide enough details (including full paths of batch script files, log files, or important input/output files) that our consultants can begin to work on your problem without having to ask you basic initial questions.

Ask or tell the HPC community

hpc-ask is a Google group established to stimulate interactions within UD's broader HPC community and is based on members helping members. This is a great venue to post a question about HPC, start a discussion, or share an upcoming event with the community. Anyone may request membership. Messages are sent as a daily summary to all group members. This list is archived, public, and searchable by anyone.

Publication and Grant Writing Resources

Please refer to the NSF award information for a proposal or requesting allocations on DARWIN. We require all allocation recipients to acknowledge their allocation awards using the following standard text: “This research was supported in part through the use of DARWIN computing system: DARWIN – A Resource for Computational and Data-intensive Research at the University of Delaware and in the Delaware Region, Rudolf Eigenmann, Benjamin E. Bagozzi, Arthi Jayaraman, William Totten, and Cathy H. Wu, University of Delaware, 2021, URL: https://udspace.udel.edu/handle/19716/29071″

ACCESS Allocations

A PI may request allocations on DARWIN via Access. See the Access Allocations page for details on how to do so. If an allocation on DARWIN is granted, a PI may use the ACCESS Allocations portal to add or remove accounts for an active allocation on DARWIN as long as the person you want to add has an ACCESS user portal account. If the person doesn’t have an ACCESS user portal account, then they need to visit the ACCESS User Registration to create one. The person will need to share their ACCESS user portal account with the PI to be added. Please keep in mind it may take up to 10 business days to process an account request on DARWIN for ACCESS users.

Accounts

An ACCESS username will be assigned having the form xsedeuuid. The uid is based on a unique, 4-digit numerical identifier assigned to you. An email with the subject [darwin-users] New DARWIN ACCESS (XSEDE) account information will be sent to the ACCESS user once their account is ready on DARWIN. Please keep in mind it may take up to 10 business days to process an account request on DARWIN for ACCESS users. Passwords are not set for ACCESS accounts on DARWIN, so you must set a password using the password reset web application at https://idp.hpc.udel.edu/access-password-reset/.

The application starts by directing the client to the CILogon authentication system where the “ACCESS CI (XSEDE)” provider should be selected. If successful (and the client has an account on DARWIN), the application next asks for an email address to which a verification email should be sent; the form is pre-populated with the email address on-record on DARWIN for the client's account. The client has 15 minutes to follow the link in that email message to choose a new password. The form displays information regarding the desired length and qualifications of a valid password. If the new password is acceptable, the client's DARWIN password is set and SSH access via password should become available immediately.

ACCESS users on DARWIN can use the password reset web application to reset a forgotten password, too.

See connecting to DARWIN for more details.

For example,

Groups

The allocation groups of which you are a member determine which computing nodes, job queues, and storage resources you may use. Each group has a unique descriptive group name (gname). There are two categories of group names: class and workgroup.

The class category: All users belong to the group named everyone.

The workgroup category: Each workgroup has a unique group name (e.g., xg-tra180011) assigned for each allocation. The PI and users are members of that allocation group (workgroup). To see the usernames of all members of the workgroup, type the hpc-group-info -a allocation_workgroup command.

Use the command groups to see all of your groups. The example below is for user xsedeu1201

For example, the command below will display the complete information about the workgroup xg-tra180011 and its members.

The output of this command represents (description=PI), along with every member in the workgroup and their account information (Username, Full Name, Email Address).

Connecting to DARWIN

Secure Shell program (SSH)

Use a secure shell program/client (SSH) to connect to the cluster and a secure file transfer program to move files to and from the cluster.

There are many suitable secure clients for Windows, Mac OS X, and UNIX/Linux. We recommend MobaXterm or PuTTY and Xming for Windows users. Macintosh and UNIX/Linux users can use their pre-installed SSH and X11 software. (Newer versions of Mac OS X may not have a current version of X11 installed. See the Apple web site for X11 installation instructions.)

IT strongly recommends that you configure your clients as described in the online X-windows (X11) and SSH documents (Windows / Linux/MacOSX). If you need help generating or uploading your SSH keys, please see the Managing SSH Keys page for ACCESS recommendations on how to do so.

Your HPC home directory has a .ssh directory. Do not manually erase or modify the files that were initially created by the system. They facilitate communication between the login (head) node and the compute nodes. Only use standard ssh commands to add keys to the files in the .ssh directory.

Please refer to Windows and Mac/Linux related sections for specific details using the command line on your local computer:

• Connecting to DARWIN from Windows

• Connecting to DARWIN from Mac/Linux

Logging on to DARWIN

You need a DARWIN account to access the login node.

To learn about launching GUI applications on DARWIN, refer to Schedule Jobs page.

ACCESS users with an allocation award on DARWIN will not be able to login until their password is set by using the password reset web application at https://idp.hpc.udel.edu/access-password-reset/.

The application starts by directing the client to the CILogon authentication system where the “ACCESS CI (XSEDE)” provider should be selected. If successful (and the client has an account on DARWIN), the application next asks for an email address to which a verification email should be sent; the form is pre-populated with the email address on-record on DARWIN for the client's account. The client has 15 minutes to follow the link in that email message to choose a new password. The form displays information regarding the desired length and qualifications of a valid password. If the new password is acceptable, the client's DARWIN password is set and SSH access via password should become available immediately.

ACCESS users on DARWIN can use the password reset web application to reset a forgotten password, too.

Once a password has been set, you may login to DARWIN by using:

or if you need you to use X-Windows requiring X11 forwarding (e.g., for a Jupyter Notebook or applications that generate graphical output), then use

where XXXX is your unique uid. The standard methods documented for adding a public key on DARWIN will only work once a password has been set for your ACCESS DARWIN account using the password reset web application. If need help setting up SSH, please see the Generating SSH Keys page and/or Uploading Your SSH Key page.

Once you are logged into DARWIN, your account is configured as a member of an allocation workgroup which determines access to your HPC resources on DARWIN. Setting your allocation workgroup is required in order to submit jobs to the DARWIN cluster. For example, the bjones account is a member of the it_css workgroup. To start a shell in the it_css workgroup, type:

Consult the following pages for detailed instructions for using DARWIN.

File Systems

Home

The 13.5 TiB file system uses 960 GiB enterprise class SSD drives in a triple-parity RAID configuration for high reliability and availability. The file system is accessible to all nodes via IPoIB on the 100 Gbit/s InfiniBand network.

Storage

Each user has 20 GB of disk storage reserved for personal use on the home file system. Users' home directories are in /home (e.g., /home/1005), and the directory name is put in the environment variable $HOME at login.

High-Performance Lustre

Lustre is designed to use parallel I/O techniques to reduce file-access time. The Lustre file systems in use at UD are composed of many physical disks using RAID technologies to give resilience, data integrity, and parallelism at multiple levels. There is approximately 1.1 PiB of Lustre storage available on DARWIN. It uses high-bandwidth interconnects such as Mellanox HDR100. Lustre should be used for storing input files, supporting data files, work files, and output files associated with computational tasks run on the cluster.

Consult All About Lustre for more detailed information.

Workgroup Storage

Allocation workgroup storage is available on a high-performance Lustre-based file system having almost 1.1 PB of usable space. Users should have a basic understanding of the concepts of Lustre to take full advantage of this file system. The default stripe count is set to 1 and the default striping is a single stripe distributed across all available OSTs on Lustre. See Lustre Best Practices from Nasa.

Each allocation will have at least 1 TiB of shared (workgroup) storage in the /lustre/ directory identified by the «allocation_workgroup» (e.g., /lustre/it_css) accessbile by all users in the allocation workgroup, and is referred to as your workgroup directory ($WORKDIR), if the allocation workgroup has been set.

Each user in the allocation workgroup will have a /lustre/«workgroup»/users/«uid» directory to be used as a personal workgroup storage directory for running jobs, storing larger amounts of data, input files, supporting data files, work files, output files and source code. It can be referred to as $WORKDIR_USERS, if the allocation workgroup has been set.

Each allocation will also have a /lustre/«workgroup»/sw directory to allow users to install software to be shared for the allocation workgroup. It can be referred to as $WORKDIR_SW, if the allocation workgroup has been set. In addition a /lustre/«workgroup»/sw/valet) directory is also provided to store VALET package files to shared for the allocation workgroup.

Please see workgroup for complete details on environment variables.

Note: A full file system inhibits use for everyone preventing jobs from running.

Local/Node File System

Temporary Storage

Each compute node has its own 2 TB local hard drive, which is needed for time-critical tasks such as managing virtual memory. The system usage of the local disk is kept as small as possible to allow some local disk for your applications, running on the node.

Quotas and Usage

To help users maintain awareness of quotas and their usage on the /home file system, the command my_quotas is available to display a list of all quota-controlled file systems on which the user has storage space.

For example, the following shows the amount of storage available and in-use for user bjones in workgroup it_css for their home and workgroup directory.

Home

Each user's home directory has a hard quota limit of 20 GB. To check usage, use

The example below displays the usage for the home directory (/home/1201) for the account bjones as 7.2 GB used out of 20 GB which matches the above example provide by my_quotas command.

Workgroup

All of Lustre is available for allocation workgroup storage. To check Lustre usage for all users, use df -h /lustre.

The example below shows 25 TB is in use out of 954 TB of usable Lustre storage.

To see your allocation workgroup usage, please use the my_quotas command. Again the the following example shows the amount of storage available and in-use for user bjones in allocation workgroup it_css for their home and allocation workgroup directories.

Node

The node temporary storage is mounted on /tmp for all nodes. There is no quota, and if you exceed the physical size of the disk you will get disk failure messages. To check the usage of your disk, use the df -h command on the compute node where your job is running.

We strongly recommend that you refer to the node scratch by using the environment variable, $TMPDIR, which is defined by Slurm when using salloc or srunor sbatch.

For example, the command

shows size, used and available space in M, G or T units.

This node r1n00 has a 2 TB disk, with only 41 MB used, so 1.8 TB is available for your job.

There is a physical disk installed on each node that is used for time critical tasks, such as swapping memory. Most of the compute nodes are configured with a 2 TB disk, however, the /tmp file system will never have the total disk. Larger memory nodes will need to use more of the disk for swap space.

Recovering Files

While all file systems on the DARWIN cluster utilize hardware redundancies to protect data, there is no backup or replication and no recovery available for the home or Lustre file systems. All backups are the responsibility of the user. DARWIN's systems administrators are not liable for any lost data.

Usage Recommendations

Home directory: Use your home directory to store private files. Application software you use will often store its configuration, history and cache files in your home directory. Generally, keep this directory free and use it for files needed to configure your environment. For example, add symbolic links in your home directory to point to files in any of the other directory.

Workgroup directory: Use the personal allocation workgroup directory for running jobs, storing larger amounts of data, input files, supporting data files, work files, output files and source code in $WORKDIR_USERS as an extension of your home direcory. It is also appropriate to use the software allocation workgroup directory to build applications for everyone in your allocation group in $WORKDIR_SW as well as create a VALET package for your fellow researchers to access applications you want to share in $WORKDIR_SW/valet.

Node scratch directory: Use the node scratch directory for temporary files. The job scheduler software (Slurm) creates a temporary directory in /tmp specifically for each job's temporary files. This is done on each node assigned to the job. When the job is complete, the subdirectory and its contents are deleted. This process automatically frees up the local scratch storage that others may need. Files in node scratch directories are not available to the head node, or other compute nodes.

Transferring Files

Be careful about modifications you make to your startup files (e.g. .bash*). Commands that produce output such as VALET or workgroup commands may cause your file transfer command or application to fail. Log into the cluster with ssh to check what happens during login, and modify your startup files accordingly to remove any commands which are producing output and try again. See computing environment startup and logout scripts for help.

Common Clients

You can move data to and from the cluster using the following supported clients:

Table. Command-line Clients

If you prefer a non-command-line interface, then consult this table for GUI clients.

Table. Graphical-User-Interface Clients

For Windows clients: editing files on Windows desktops and then transferring them back to the cluster, you may find that your file becomes "corrupt" during file transfer process. The symptoms are very subtle because the file appears to be okay, but in fact contains CRLF line terminators. This causes problems when reading the file on a Linux cluster and generates very strange errors. Some examples might be a file used for submitting a batch job such as submit.qs and one you have used before and know is correct, will no longer work. Or an input file used for ABAQUS like tissue.inp which has worked many times before produces an error like Abaqus Error: Command line option "input" must have a value..

Use the "file" utility to check for CRLF line terminators and dos2unix to fix it, like this below

Copying Files to the Cluster

To copy a file over an SSH connection from a Mac/UNIX/Linux system to any of the cluster's file systems, type the generic command:

scp [options] local_filename HPC_username@HPC_hostname:HPC_filename

Begin the HPC_filename with a "/" to indicate the full path name. Otherwise the name is relative to your home directory on the HPC cluster.

Use scp -r to copy an entire directory, for example:

copies the fuelcell directory in your local current working directory into the /lustre/it_css/users/1201/projects directory on the DARWIN cluster. The /lustre/it_css/users/1201/projects directory on the DARWIN cluster must exist, and bjones must have write access to it.

Copying files from the cluster

To copy a file over an SSH connection to a Mac/UNIX/Linux system from any of the cluster's files systems type the generic command:

scp [options] HPC_username@HPC_hostname:HPC_filename local_filename

Begin the HPC_filename with a "/" to indicate the full path name. Otherwise, the name is relative to your home directory.

Use scp -r to copy the entire directory. For example:

will copy the directory fuelcell on the DARWIN cluster into a new fuelcell directory in your local system's current working directory. (Note the final period in the command.)

Copying Files Between Clusters

You can use GUI applications to transfer small files to and from your PC as a way to transfer between clusters, however this is highly inefficient for large files due to multiple transfers and slower disk speeds. As a result, you do not benefit from the arcfour encoding.

The command tools work the same on any Unix cluster. To copy a file over an SSH connection, first logon the file cluster1 and then use the scp command to copy files to cluster1. Use the generic commands:

ssh [options] HPC_username1@HPC_hostname1
scp [options] HPC_filename1 HPC_username2@HPC_hostname2:HPC_filename2

Login to HPC_hostname1 and in the scp command begin both HPC_filename1 and HPC_filename2 with a "/" to indicate the full path name. The clusters will most likely have different full path names.

Use ssh -A to enable agent forwarding and scp -r to copy the entire directory.¹⁾ For example:

will copy the directory fuelcell from Farber to a new fuelcell directory on DARWIN.

If you are using PuTTY, skip the ssh step and connect to the cluster you want to copy from.

Application Development

There are three 64-bit compiler suites on DARWIN with Fortran, C and C++:

In addition, multiple versions of OpenJDK are available for compiling java applications on the login node.

DARWIN is based on AMD EPYC processors, please review the following documents if you are planning to compile and install your own software

• High Performance Computing (HPC) Tuning Guide for AMD EPYC™ 7002 Series Processors guide for getting started tuning AMD 2nd Gen EPYC™ Processor based systems for HPC workloads. This is not an all-inclusive guide and some items may have similar, but different, names in specific OEM systems (e.g. OEM-specific BIOS settings). Every HPC workload varies in its performance characteristics. While this guide is a good starting point, you are encouraged to perform your own performance testing for additional tuning. This guide also provides suggestions on which items should be the focus of additional, application-specific tuning (November 2020).

• HPC Tuning Guide for AMD EPYC™ Processors guide intended for vendors, system integrators, resellers, system managers and developers who are interested in EPYC system configuration details. There is also a discussion on the AMD EPYC software development environment, and we include four appendices on how to install and run the HPL, HPCG, DGEMM, and STREAM benchmarks. The results produced are ‘good' but are not necessarily exhaustively tested across a variety of compilers with their optimization flags (December 2018).

• AMD EPYC™ 7xx2-series Processors Compiler Options Quick Reference Guide, however we do not have the AOCC compiler (with Flang - Fortran Front-End) installed on DARWIN.

Computing Environment

UNIX Shell

The UNIX shell is the interface to the UNIX operating system. The HPC cluster allows use of the enhanced Bourne shell bash, the enhanced C shell tcsh, and the enhanced Korn shell zsh. IT will primarily support bash, the default shell.

For most Linux systems, the sh shell is the bash shell and the csh shell is the tcsh shell. The remainder of this document will use only bash commands.

Environment Variables

Environment variables store dynamic system values that affect the user environment. For example, the PATH environment variable tells the operating system where to look for executables. Many UNIX commands and tools, such as the compilers, debuggers, profilers, editors, and applications with graphical user interfaces, often look at environment variables for information they need to function. The man pages for these programs typically have an ENVIRONMENT VARIABLES section with a list of variable names which tells how the program uses the values.

This is why we encourage users to use VALET to modify your environment versus explicitly setting environment variables.

In bash, a variable must be exported to be used as an environment variable. By convention, environment variables are all uppercase. You can display a list of currently set environment variables by typing

The "echo" and "export" commands will display and set environment variables.

You can display specific environment variables by typing for example:

The variable FFLAGS will have the value -g -Wall in the shell and exported to programs run from this shell.

Spaces are important. Do not put spaces around the equal sign. If the value has spaces, enclose the value in quotes.

If you see instructions that refer the setenv command, replace it with the export bash command. Make sure you use equal signs, with no spaces. The setenv "csh" command uses spaces instead of one equal.

Startup and Logout Scripts

All UNIX systems set up a default environment and provide users with the ability to execute additional UNIX commands to alter the environment. These commands are automatically sourced (executed) by your shell and define the normal and environmental variables, command aliases, and functions you need. Additionally, there is a final system-wide startup file that automatically makes global environment changes that IT sets for all users.

You can modify the default environment by adding lines at the end of the ~/.bash_profile file and the ~/.bashrc file. These modifications affect shells started on the login node and the compute nodes. In general we recommend that you should not modify these files especially when software documentation refers to changing the PATH environment variable, instead use VALET to load the software.

• The ~/.bash_profile file's commands are executed once at login. Add commands to this file to set your login environment and to run startup programs.

• The ~/.bashrc file's commands are executed by each new shell you start (spawn). Add lines to this file to create aliases and bash functions. Commands such as "xterm" and "workgroup" automatically start a new shell and execute commands in the ~/.bashrc file. The "salloc" command starts a shell on a compute node and will execute the ~/.bashrc file from your home directory, but it does not execute the commands in the ~/.bash_profile file.

You may modify the IT-supplied ~/.bash_udit file to be able to use several IT-supplied aliases (commands) and environment settings related to your workgroup and work directory . Edit .bash_udit and follow the directions in the file to activate these options. This is the ONLY way you should set your default workgroup at login. DO NOT add the workgroup command to your .bashrc or .bash_profile as this will likely prevent you from logging in and will cause file transfer programs like WinSCP, sftp or Fetch to break.

Exiting the login session or typing the "logout" command executes your ~/.bash_logout file and terminates your session. Add commands to ~/.bash_logout that you want to execute at logout.

To restore the .bash_profile, .bashrc, .bash_udit and .bash_logout files in your home directory to their original state, type from the login node:

Where to put startup commands: You can put bash commands in either ~/.bashrc or ~/.bash_profile. Again we do not recommend modifying these files unless you really know what you are doing. Here are general suggestions:

• Even if you have favorite commands from other systems, start by using the supplied files and only modify .bash_udit for customization.

• Add essential commands that you fully understand, and keep it simple. Quoting rules can be complicated.

• Do not depend on the order of command execution. Do not assume your environment, set in .bash_profile, will be available when the commands in .bashrc are executed.

• Do not include commands that spawn new shells, such as workgroup.

• Be very careful of commands that may produce output. If you must, only execute them after a test to make sure there is a terminal to receive the output. Keep in mind using any commands that produce output may break other applications like file transfer (sftp, scp, WinSCP).

• Do not include VALET commands as they produce output and will be a part of every job submitted which could cause conflicts with other applications you are trying to run in your job script.

• Keep a session open on the cluster, so when you make a change that prevents you from logging on you can reverse the last change, or copy the original files from /opt/shared/templates/homedir/ to start over.

Using workgroup and Directories

There are some key environment variables that are set for you, and are important for your work on any cluster. They are used to find directories for your projects. These environment variables are set on initial connection to a cluster, and will be changed if you

• set your workgroup (allocation group allocation_group name) with the "workgroup" command,

• change to your project directory with the "cd" command,

• connect to the compute node resources with "salloc" (or "sbatch") command specifying a single partition your allocation workgroup has access based on resources requested for your allocation.

Connecting to Login Node

The system's initialization scripts set the values of some environment variables to help use the file systems.

The initialization scripts also set the standard prompt with your login name and a shortened host name. For example, if your hostname is darwin.hpc.udel.edu and your login_name is bjones, then the standard prompt will be

[bjones@login00.darwin ~]$

Clusters may be configured to have multiple login nodes, with one common name for connecting. For example, on the DARWIN cluster, the hostname may be set to login00 or login01, and the standard prompt and window title bar will indicate which login node on darwin.

Setting Workgroup

To use the compute node resources for a particular allocation group (workgroup), you need to use the "workgroup" command.

For example,

starts a new shell for the workgroup it_css, and sets the environment variables:

Use specific environment variables such as $WORKDIR_USERS when referring to your allocation workgroup user directory and $WORKDIR_SW when referring to your allocation workgroup software directory. This will improve portability.

It is always important to be aware of your current allocation workgroup name. The standard prompt includes the allocation workgroup name, added to your username and host. You must have an allocation workgroup name in your prompt to use that allocation group's compute node resources to submit jobs using sbatch or salloc. An example prompt after the "workgroup" command,

[(it_css:bjones)@login01.darwin ~]$

Changing Directory

When you first connect to the login node, all your commands are executed from your home directory (~). Most of your work will be done in your allocation workgroup directory. The "workgroup" command has an option to start you in the allocation workgroup work directory. For example,

will spawn a new shell in the workgroup directory for it_css.

You can always use the cd bash command.

For example,

The first is using a path name relative to the current working directory (implied ./). The second, third and fourth use the full path ($WORKDIR and $WORKDIR_USERS always begins with a /). In all cases the directory is changed, and the $PWD environment variable is set:

It is always important to be aware of your current working directory. The standard prompt ends with the basename of PWD. In these two examples the basename is the same, 1201, but the standard bash PROMPT_COMMAND, which is executed every time you change directories, will put the full path of your current working directory in your window title. For example,

bjones@login00.darwin:/lustre/it_css/users/1201

Connecting to a Compute Node

To run a job on the compute nodes, you must submit your job script using sbatch or start an interactive session using salloc. In both cases, you will be connected to one of your allocation compute nodes based on the partition (queue) specified with a clean environment. Do not rely on the environment you set on the login node. The variables USER, HOME, WORKGROUP, WORKDIR, WORKDIR_USERS and PWD are all set on the compute node to match the ones you had on the login node, but two variables are set to node-specific values:

An empty directory is created by the SLURM job scheduler that is associated with your job and defined as TMPDIR. This is a safe place to store temporary files that will not interfere with other jobs and tasks you or other members of your group may be executing. This directory is automatically emptied on normal termination of your job. This way the usage on the node scratch file system will not grow over time.

Before submitting jobs you must first use the "workgroup" command. Type workgroup -h for additional information. Both "sbatch" and "salloc" will start in the same project directory you set on the login node and will require a single partition to be specified to be able to submit a batch or interactive session.

Using VALET

The UD-developed VALET system facilitates your use of compilers, libraries, programming tools and application software. It provides a uniform mechanism for setting up a package's required UNIX environment. VALET is a recursive acronym for VALET Automates Linux Environment Tasks. It provides functionality similar to the Modules package used at other HPC sites.

VALET commands set the basic environment for software. This may include the PATH, MANPATH, INFOPATH, LDPATH, LIBPATH and LD_LIBRARY_PATH environment variables, compiler flags, software directory locations, and license paths. This reduces the need for you to set them or update them yourself when changes are made to system and application software. For example, you might find several versions for a single package name, such as Mathematica/8 and Mathematica/8.0.4. You can even apply VALET commands to packages that you install or alter its actions by customizing VALET's configuration files. Type man valet for instructions or see the VALET software documentation for complete details.

The table below shows the basic informational commands for VALET. In subsequent sections, VALET commands are illustrated in the contexts of application development (e.g., compiling, using libraries) and running IT-installed applications.

Programming Environment

Programming Models

There are two memory models for computing: distributed-memory and shared-memory. In the former, the message passing interface (MPI) is employed in programs to communicate between processors that use their own memory address space. In the latter, open multiprocessing (OMP) programming techniques are employed for multiple threads (light weight processes) to access memory in a common address space. When your job spans several compute nodes, you must use an MPI model.

Distributed memory systems use single-program multiple-data (SPMD) and multiple-program multiple-data (MPMD) programming paradigms. In the SPMD paradigm, each processor loads the same program image and executes and operates on data in its own address space (different data). It is the usual mechanism for MPI code: a single executable is available on each node (through a globally accessible file system such as $WORKDIR), and launched on each node (through the MPI wrapper command, mpirun).

The shared-memory programming model is used on Symmetric Multi-Processor (SMP) nodes such as a single typical compute node (20 or 24 cores, 64 GB memory). The programming paradigm for this memory model is called Parallel Vector Processing (PVP) or Shared-Memory Parallel Programming (SMPP). The former name is derived from the fact that vectorizable loops are often employed as the primary structure for parallelization. The main point of SMPP computing is that all of the processors in the same node share data in a single memory subsystem. There is no need for explicit messaging between processors as with MPI coding.

The SMPP paradigm employs compiler directives (as pragmas in C/C++ and special comments in Fortran) or explicit threading calls (e.g. with Pthreads). The majority of science codes now use OpenMP directives that are understood by most vendor compilers, as well as the GNU compilers.

In cluster systems that have SMP nodes and a high-speed interconnect between them, programmers often treat all CPUs within the cluster as having their own local memory. On a node, an MPI executable is launched on each processor core and runs within a separate address space. In this way, all processor cores appear as a set of distributed memory machines, even though each node has processor cores that share a single memory subsystem.

Clusters with SMPs sometimes employ hybrid programming to take advantage of higher performance at the node-level for certain algorithms that use SMPP (OMP) parallel coding techniques. In hybrid programming, OMP code is executed on the node as a single process with multiple threads (or an OMP library routine is called), while MPI programming is used at the cluster-level for exchanging data between the distributed memories of the nodes.

Compiling Code

Fortran, C, C++, Java and Matlab programs should be compiled on the login node, however if lengthy compiles or extensive resources needed, you may need to schedule a job for compilation using salloc or sbatch which will be billed to your allocation. All resulting executables should only be run on the compute nodes.

The Compiler Suites

There are three 64-bit compiler suites that IT generally installs and supports: PGI CDK (Portland Group Inc.'s Cluster Development Kit), Intel Composer XE 2011, and GNU. In addition, IT has installed OpenJDK (Open Java Development Kit), which must only be used on the compute nodes. (Type vpkg_info openjdk for more information on OpenJDK.)

The PGI compilers exploit special features of AMD processors. If you use open-source compilers, we recommend the GNU collection.

You can use a VALET vpkg_require command to set the UNIX environment for the compiler suite you want to use. After you issue the corresponding vpkg_require command, the compiler path and supporting environment variables will be defined.

A general command for basic source code compilation is:

<compiler> <compiler_flags> <source_code_filename> -o <executable_filename>

For each compiler suite, the table below displays the compiler name, a link to documentation describing the compiler flags, and the appropriate filename extension for the source code file. The executable will be named a.out unless you use the -o <executable_filename> option.

To view the compiler option flags, their syntax, and a terse explanation, execute a compiler command with the -help option. Alternatively, read the compiler's man pages.

Compiling Serial Programs

This section uses the PGI compiler suite to illustrate simple Fortran and C compiler commands that create an executable. For each compiler suite, you must first set the UNIX environment so the compilers and libraries are available to you. VALET commands provide a simple way to do this.

The examples below show the compile and link steps in a single command. These illustrations use source code files named fdriver.f90 (Fortran 90) or cdriver.c (C). They all use the -o option to produce an executable named 'driver.' The optional -fpic PGI compiler flag generates position-independent code and creates smaller executables. You might also use code optimization option flags such as -fast after debugging your program.

You can use the -c option instead to create a .o object file that you would later link to other object files to create the executable.

Some people use the UNIX make command to compile source code. There are many good online tutorials on the basics of using make. Also available is a cross-platform makefile generator, cmake. You can set the UNIX environment for cmake by typing the vpkg_require cmake command.

Using the PGI suite to illustrate:

First use a VALET command to set the environment:

Then use that compiler suite's commands:

Fortran 90 example:

C example:

Compiling Parallel Programs with OpenMP

If your program only uses OpenMP directives, has no message passing, and your target is a single SMP node, you should add the OpenMP compiler flag to the serial compiler flags.

Instead of using OpenMP directives in your program, you can add an OpenMP-based library. You will still need the OpenMP compiler flag when you use the library.

Compiling Parallel Programs with MPI

MPI Implementations

In the distributed-memory model, the message passing interface (MPI) allows programs to communicate between processors that use their own node's memory address space. It is the most commonly used library and runtime environment for building and executing distributed-memory applications on clusters of computers.

OpenMPI is the most desirable MPI implementation to use. It is the only one that works for job suspension, checkpointing, and task migration to other processors. These capabilities are needed to enable opportunistic use of idle nodes as well as to configure short-term and long-term queues.

Some software comes packaged with other MPI implementations that IT cannot change. In those cases, their VALET configuration files use the bundled MPI implementation. However, we recommend that you use OpenMPI whenever you need an MPI implementation.

MPI Compiler Wrappers

The OpenMPI implementation provides OpenMPI library compilers for C, C++, Fortran 77, 90, and 95. These compiler wrappers add MPI support to the actual compiler suites by passing additional information to the compiler. You simply use the MPI compiler wrapper in place of the compiler name you would normally use.

The compiler suite that's used depends on your UNIX environment settings. Use VALET commands to simultaneously set your environment to use the OpenMPI implementation and to select a particular compiler suite. The commands for the four compiler suites are:

(Type vpkg_versions openmpi to see if newer versions are available.)

The vpkg_require command selects the MPI and compiler suite combination, and then you may use the compiler wrapper commands repeatedly. The wrapper name depends only on the language used, not the compiler suite you choose: mpicc (C), mpicxx or mpic++ (C++), mpi77 (Fortran 77), and mpif90 (Fortran 90 and 95).

Fortran example:

C example:

You may use other compiler flags listed in each compiler suite's documentation.

To modify the options used by the MPI wrapper commands, consult the FAQ section of the OpenMPI web site.

Programming Libraries

IT installs high-quality math and utility libraries that are used by many applications. These libraries provide highly optimized math packages and functions. To determine which compilers IT used to prepare a library version, use the vpkg_versions VALET command.

Here is a representative sample of installed libraries. Use the vpkg_list command to see the most current list of libraries.

Open-source libraries

• ATLAS: Automatically Tuned Linear Algebra Software (portable)

• FFTW: Discrete Fast Fourier Transform library

• BLAS and LAPACK at TACC: Enhanced BLAS routines from the Texas Advanced Computing Center (TACC)

• HDF4 and HDF5: Hierarchical Data Format suite (file formats and libraries for storing and organizing large, numerical data collections)

• HYPRE: High-performance preconditioners for linear system solvers (from LLNL)

• LAPACK: Linear algebra routines

• Matplotlib: Python-based 2D publication-quality plotting library

• netCDF: network Common Data Form for creation, access and sharing of array-oriented scientific data

• ScaLAPACK - Scalable LAPACK: Subset of LAPACK routines redesigned for distributed memory MIMD parallel computers using MPI

• VTK – Visualization ToolKit: A platform for 3D computer graphics and visualization

Commercial Libraries

• IMSL: RogueWave's mathematical and statistical libraries

• MKL: Intel's Math Kernel Library

• NAG: Numerical Algorithms Group's numerical libraries

The libraries will be optimized a given cluster architecture. Note that the calling sequences of some of the commercial library routines differ from their open-source counterparts.

Using Libraries

This section shows you how to link your program with libraries you or your colleagues have created or with centrally installed libraries such as ACML or FFTW. The examples introduce special environment variables (FFLAGS, CFLAGS, CPPFLAGS and LDFLAGS) whose use simplifies a command's complexity. The VALET commands vpkg_require and vpkg_devrequire can easily define the working environment for your compiler suite choice.

Joint use of VALET and these environment variables will also prepare your UNIX environment to support your use of make for program development. VALET will accommodate using one or several libraries, and you can extend its functionality for software you develop or install.

Intel Compiler Suite

You should use Intel MKL — it's a highly-optimized BLAS/LAPACK library.

If you use the Intel compilers, you can add -mkl to your link command, e.g.

The former uses the serial library, the latter uses the threaded library that respects the OpenMP runtime environment of the job for multithreaded BLAS/LAPACK execution.

If you're not using the Intel compilers, you'll need to generate the appropriate compiler directives using Intel's online tool.

Please use "dynamic linking" since that allows MKL to adjust the underlying kernel functions at runtime according to the hardware on which you're running. If you use static linking, you're tied to the lowest common hardware model available and you will usually not see as good performance.

You'll need to load a version of Intel into the environment before compiling/building and also at runtime using VALET such as

Among other things, this will set MKLROOT in the environment to the appropriate path, which the link advisor references. The MKL version (year) matches that of the compiler version (year).

To determine the available versions of Intel installed use

PGI Compiler Suite

Fortran Examples illustrated with the PGI compiler suite

Reviewing the basic compilation command

The general command for compiling source code:

«compiler» «compiler_flags» «source_code_filename» -o «executable_filename»

For example:

Using user-supplied libraries

To compile fdriver.f90 and link it to a shared F90 library named libfstat.so stored in $HOME/lib, add the library location and the library name (fstat) to the command:

The -L option flag is for the shared library directory's name; the -l flag is for the specific library name.

You can simplify this compiler command by creating and exporting two special environment variables. FFLAGS represents a set of Fortran compiler option flags; LDFLAGS represents the location and choice of your library.

Extending this further, you might have several libraries in one or more locations. In that case, list all of the '-l' flags in the LDLIBS statement, for example,

and all of the '-L' flags in the LDFLAGS statement. (The order in which the '-L' directories appear in LDFLAGS determines the search order.)

Using centrally supplied libraries (ACML, MKL, FFTW, etc.)

This extends the previous section's example by illustrating how to use VALET's vpkg_devrequire command to locate and link a centrally supplied library such as AMD's Core Math Library, ACML. Several releases (versions) of a library may be installed, and some may have been compiled with several compiler suites.

To view your choices, use VALET's vpkg_versions command:

The example below uses the acml/5.0.0-pgi-fma4 version, the single-threaded, ACML 5.0.0 FMA4 library compiled with the PGI 11 compilers. Since that version depends on the PGI 11 compiler suite,