Open Science Grid - OSG
System Overview
The OSG's Open Science Pool
The OSG provides technologies and services for supporting pools of distributed high-throughput computing (dHTC) and data capacity that serve a number of research communities, worldwide. Among these services, the OSG operates the Open Science Pool (OSPool), which includes capacity contributed from clusters at campuses and national labs across and beyond the US, organized as a single virtual cluster.
The OSPool delivered more than 280 million compute hours in 2021, with individual users achieving more than 30 million hours without fee or allocation requirement (see OSG's Grid Accounting system).
Computations Accelerated by the OSPool
Given the distributed and opportunistic (excess backfill) nature of this capacity, the OSPool is a tremendous resource for work that can be run with a high-throughput computing (HTC) approach, in the form of numerous independent and self-contained tasks, each with input and output data. On the OSPool, a single user with single-core jobs can achieve up to 1000s of concurrent cores in use within hours of submitting, and more than 100,000 hours of usage per day. The OSG uses the HTCondor compute scheduling software, which includes first-class support for submitting and managing large numbers of jobs, via a single submit file or multi-step workflow. Read more about applicable computations and throughput in the OSG User Documentation.
Features of computational work that scales well across the OSPool:
individual jobs that complete (or checkpoint) within 20 hours on one or few cores; jobs requiring large numbers of cores and/or inter-node communication do not run (or run well) on the OSPool, though manager-worker applications may be supported
software that can be distributed across each job in the form of static binaries, self-contained installations (e.g. directories, tarballs), or containers (OSG staff are happy to help you pursue the ‘right' approach); most commercially-licensed software is not suitable, per license limitations
'open' data and software; the OSPool is not suitable for proprietary or secure data requirements, given that jobs run on a variety of clusters, each with their own administrative access policies
less than 20GB of input and output, per-job; multiple TBs of total data can be supported across an entire batch of jobs, and as long as each job's input and output can be handled separately from other jobs (no shared filesystem required when each job executes)
A wide range of computational research tasks can be executed with an HTC approach, often with modest changes to a user's code and/or by splitting datasets. Examples include:
image analysis (including MRI, GIS, etc.)
text-based analysis, including any/all DNA read mapping, quality control/trimming, and other bioinformatics tasks
high-throughput drug or materials screening, including simulations and energy calculations
parameter sweeps, Monte Carlo methods, model optimization, and ensembles of simulations
machine learning, especially with hyperparameter treatments or multi-model training
these and other GPU-dependent computations that can be run with an HTC approach
Get in touch with an OSG Research Computing Facilitator to discuss what's possible for your own work: support@osg-htc.org
Who Can Use the OSPool
Access to OSPool capabilities is available for free and without an allocation via the OSG Connect Service for individuals and projects associated with a US-based academic, government, or non-profit organization. Additionally, OSG allocations for OSPool computing resources are available, including for those who may not meet the requirements for OSG Connect but who do meet ACCESS requirements. Users and groups with allocations also receive higher fair-share priority, which may be another reason for pursuing an allocation via ACCESS.
Interested users should also consult the OSG Connect and OSPool Policies.
Integrate Workloads and Allocations across Resources
One advantage of using an OSG access point (i.e. login node or submit server, including the OSG Connect access points) is that users can run work not only on the OSPool resources (best suited to work described above), but also on other ACCESS-allocated resources, commercial cloud (credits or paid), and other computing systems. This may be particularly useful for projects with computational requirements outside of those suitable for the OSPool, or that consist of a workflow of tasks that need to run across differently-suited computing systems, all through a single interface and submission point. Contact the OSG via support@osg-htc.org if you'd like to discuss options.
Get Help
Any user of the OSPool and/or OSG Connect services can contact the OSG Research Facilitation team by emailing support@osg-htc.org.
System Access
As of December 2021, users with ACCESS allocations (startup or regular) will be requested to 'Sign Up' for an account via the OSG Connect service, after which an OSG Research Computing Facilitator will meet with them in order to provide personalized guidance and set up appropriate accounts. Given differences between the OSPool and other ACCESS-participating resources, login via the ACCESS single sign-on portal is not functional.
System Features
The Open Science Pool includes capacity contributed by dozens of campus, national labs, and non-profit organizations, often opportunistically via backfill of a cluster local to the site. These resources are organized into a single pool (virtual cluster) that is dynamically sized based upon demand (the number of jobs in the queue across configured access points) and the supply of resources currently available at contributing sites.
System Configuration
On average, there are more than 40,000 total cores available across the pool, with peaks of more than 70,000 cores, and with hundreds of GPUs available. Nodes from each contributing cluster may differ in CPU and/or GPU models, number of cores, RAM, etc. However, each job requests the computing resources it needs (usually one or few cores, memory, disk, GPU features, etc.), such that multiple jobs run on each contributed node, and even sub-node units of computing are contributed. Compute slots are created dynamically (multiple per node) to server job requirements that fit within available resources on each node, and are the unit of computing for the purpose of matching jobs.
There are a number of access points into the OSPool, including access points operated as part of the OSG Connect services, where users can log in, stage data, and submit jobs. These have sufficient CPU, memory, and storage resources to support job submission and data transfer. Users are expected to refrain from running their software on the OSG Connect access points and should consult with OSG Research Computing Facilitators about relevant HTCondor features in lieu of running user scripts to automate job submission/resubmission or ordered workflows, as these are likely to cause performance issues on the access point.
One important difference between the OSPool and most other ACCESS resources is that there is no shared file system across access points and execute servers. This means that each job needs to bring along necessary executables, other software, and input data, which are supported features of the HTCondor scheduling software; users specify data and software to-be-transferred as part of the submit file. See Job Submission, Data Management, and Software Solutions within the below section on Using the OSPool.
Fair-Share and Computational Throughput
While the HTCondor scheduler includes tools for viewing bulk numbers of nodes and cores in the pool, as well as per-node data about memory, disk, and CPU/GPU features, the dynamic scaling of the pool means that resources are always changing and always near-fully occupied, by design. Because most jobs use only a single core, there are slots of computing capacity coming available constantly (as other jobs finish), meaning that queue wait times for a first job to start are generally within seconds or minutes (not hours).
Jobs are matched and executed based upon the user's fair-share priority and per-job resource requirements. A user's priority will be highest when they haven't executed any work recently, and will decrease as they accumulate usage relative to other users, and with a decay factor such that usage from more than a week ago counts near-zero toward the users at-the-moment priority calculation. In general, though, jobs requiring fewer and/or less specialized (less scarce) resources are able to match to more resources, to achieve a larger number of concurrently-running cores, and to reach that peak sooner, even with a low fair-share priority. This also means that a user with batches of jobs differing in resource requirements will see different throughput between the batches, such that batches with different resource requirements will run concurrently, rather than following a first-in-first-out scheduling.
Users running work as part of an ACCESS allocation will experience a higher priority factor than other users, until their allocation runs out. Jobs submitted without identifying an ACCESS allocation will be scheduled with a standard fair-share priority factor, but may be scheduled ahead of or alongside jobs associated with an allocation, depending on resource requirements. While rough estimates of computational throughput (as number of concurrent cores in use) are provided in the OSG User Documentation, throughput metrics will be best informed by running scale tests, as is the case for performance-based testing on other ACCESS resources.
Performance and Scaling
For the above reasons - and with HTC computing in general - the performance of individual jobs will vary significantly across jobs running in the OSPool and matters less for scaling, while the number of concurrently-running jobs matters much more for overall completion, i.e. computational throughput. Especially because multi-core and otherwise large jobs will match to far fewer total resources, optimal throughput in the OSPool is achieved for single-core jobs, or with jobs running with as few cores as possible (which also generally corresponds to the best per-core performance). Therefore, traditional code performance and scaling calculations are less relevant to allocation resource justifications than the overall computational throughput, which is demonstrable via the concurrent cores achievable across jobs, jobs completed per day, and total core hours required based upon the number cores requested per-job, based upon the scale achieve with test batches.
Using the OSPool
Job Submission
Details about logging in and submitting jobs are available via the OSG User Documentation, including a Roadmap for building and scaling an HTC workload.
Users submit jobs to the OSPool using the HTCondor jobs scheduler, which has first-class features for submitting highly numerous jobs via a single submit file, and with customizable variables to specify filenames, arguments, data transfer, and/or other features that differ across a set of jobs. Guides in the OSG User Documentation include detailed information about the most helpful HTCondor features and integration of HTCondor learning resources, which OSG's Research Computing Facilitators contribute to, directly.
Data Management
Data for jobs can be staged via the user's OSPool-associated access point (likely an OSG Connect login node) for transfer to/from jobs performed by the HTCondor scheduler, or can be transferred to/from each job from via web-accessible storage services (e.g. HTTP, S3) using HTCondor's built-in file transfer features. The OSPool's distributed resources take advantage of OSG's Open Science Data Federation, which automatedly and regionally caches data, where possible, for enhanced transfer performance.
While the data locations accessible via OSG Connect access points are not backed up and should be treated as temporary (scratch) space only used for actively-running jobs, individual users may use up to TBs of storage space, with justified quota.
Software Solutions
Given the distributed and heterogeneous nature of the resources in the OSPool, software can be distributed to jobs with one of just a few options described in the OSG user documentation.
These include distribution of software in the form of static binaries or self-contained installation directories (delivered to jobs in the form of tarballs), according to a software's documented installation procedures. System-level support includes the execution of jobs within automatedly-deployed containers, which are used both to provide the user with a consistent OS environment for non-containerized software. For software with complex dependencies, the same container deployment can be triggered to run each job within a user-specified container available via a community repository (e.g. DockerHub) or staged via the user's access point (e.g. Singularity .sif files). OSG's Research Computing Facilitators are happy to help you identify and implement the most appropriate and scalable solution for your software.