The way to Run A number of AI Workloads on a Single GPU

Introduction: What’s GPU Fractioning?

GPUs are in extraordinarily excessive demand proper now, particularly with the speedy development of AI workloads throughout industries. Environment friendly useful resource utilization is extra essential than ever, and GPU fractioning is among the handiest methods to realize it.

GPU fractioning is the method of dividing a single bodily GPU into a number of logical items, permitting a number of workloads to run concurrently on the identical {hardware}. This maximizes {hardware} utilization, lowers operational prices, and permits groups to run numerous AI duties on a single GPU.

On this weblog submit, we’ll cowl what GPU fractioning is, discover technical approaches like TimeSlicing and Nvidia MIG, focus on why you want GPU fractioning, and clarify how Clarifai Compute Orchestration handles all of the backend complexity for you. This makes it straightforward to deploy and scale a number of workloads throughout any infrastructure.

Now that now we have a high-level understanding of what GPU fractioning is and why it issues, let’s dive into why it’s important in real-world situations.

Why GPU Fractioning Is Important

In lots of real-world situations, AI workloads are light-weight in nature, typically requiring solely 2-3 GB of VRAM whereas nonetheless benefiting from GPU acceleration. GPU fractioning permits:

• Value Effectivity: Run a number of duties on a single GPU, considerably lowering {hardware} prices.

• Higher Utilization: Prevents under-utilization of high-priced GPU assets by filling idle cycles with extra workloads.

• Scalability: Simply scale the variety of concurrent jobs, with some setups permitting 2 to eight jobs on a single GPU.

• Flexibility: Helps diversified workloads, from inference and mannequin coaching to information evaluation, on one piece of {hardware}.

These advantages make fractional GPUs significantly engaging for startups and analysis labs, the place maximizing each greenback and each compute cycle is vital. Within the subsequent part, we’ll take a more in-depth have a look at the commonest strategies used to implement GPU fractioning in observe.

Deep Dive: Widespread Strategies for Fractioning GPUs

These are probably the most extensively used, low-level approaches to fractional GPU allocation. Whereas they provide efficient management, they typically require handbook setup, hardware-specific configurations, and cautious useful resource administration to stop conflicts or efficiency degradation.

1. TimeSlicing

TimeSlicing is a software-level method that permits a number of workloads to share a single GPU by allocating time-based slices. The GPU is just about divided into a set variety of slices, and every workload is assigned a portion primarily based on what number of slices it receives.

For instance, if a GPU is split into 20 slices:

• Workload A: Allotted 4 slices → 0.2 GPU

• Workload B: Allotted 10 slices → 0.5 GPU

• Workload C: Allotted 6 slices → 0.3 GPU

This provides every workload a proportional share of compute and reminiscence, however the system doesn’t implement these limits on the {hardware} stage. The GPU scheduler merely time-shares entry amongst processes primarily based on these allocations.

Vital traits:

• No precise isolation: All workloads run on the identical GPU with no assured separation. On a 24GB GPU, for example, Workload A ought to keep beneath 4.8GB of VRAM, Workload B beneath 12GB, and Workload C beneath 7.2GB. If any workload exceeds its anticipated utilization, it will possibly crash others.

• Shared compute with context switching: If one workload is idle, others can quickly make the most of extra compute, however that is opportunistic and never enforced.

• Excessive danger of interference: Since enforcement is handbook, incorrect reminiscence assumptions can result in instability.

2. MIG (Multi-Occasion GPU)

MIG is a {hardware} characteristic accessible on NVIDIA A100 and H100 GPUs that permits a single GPU to be cut up into remoted cases. Every MIG occasion has devoted compute cores, reminiscence, and scheduling assets, offering predictable efficiency and strict isolation.

MIG cases are primarily based on predefined profiles, which decide the quantity of reminiscence and compute allotted to every slice. For instance, a 40GB A100 GPU could be divided into:

• 4 cases utilizing the 2g.10gb profile, every with round 10GB VRAM

• 7 smaller cases utilizing the 1g.5gb profile, every with about 5GB VRAM

Every profile represents a set unit of GPU assets, and workloads can solely use one occasion at a time. You can not mix two profiles to present a workload extra compute or reminiscence. Whereas MIG presents strict isolation and dependable efficiency, it lacks the flexibleness to share or dynamically shift assets between workloads.

Key traits of MIG:

• Sturdy isolation: Every workload runs in its personal devoted area, with no danger of crashing or affecting others.

• Mounted configuration: You will need to select from a set of predefined occasion sizes.

• No dynamic sharing: In contrast to TimeSlicing, unused compute or reminiscence in a single occasion can’t be borrowed by one other.

• Restricted {hardware} help: MIG is just accessible on sure information center-grade GPUs and requires specialised setup.

How Compute Orchestration Simplifies GPU Fractioning

One of many greatest challenges in GPU fractioning is managing the complexity of organising compute clusters, allocating slices of GPU assets, and dynamically scaling workloads as demand adjustments. Clarifai’s Compute Orchestration handles all of this for you within the background. You don’t must handle infrastructure or tune useful resource settings manually. The platform takes care of every thing, so you possibly can deal with constructing and transport fashions.

Reasonably than counting on static slicing or hardware-level isolation, Clarifai makes use of clever time slicing and customized scheduling on the orchestration layer. Mannequin runner pods are positioned throughout GPU nodes primarily based on their GPU reminiscence requests, making certain that the full reminiscence utilization on a node by no means exceeds its bodily GPU capability.

Let’s say you will have two fashions deployed on a single NVIDIA L40S GPU. One is a big language mannequin for chat, and the opposite is a imaginative and prescient mannequin for picture tagging. As a substitute of spinning up separate machines or configuring complicated useful resource boundaries, Clarifai routinely manages GPU reminiscence and compute. If the imaginative and prescient mannequin is idle, extra assets are allotted to the language mannequin. When each are energetic, the system dynamically balances utilization to make sure each run easily with out interference.

This method brings a number of benefits:

• Sensible scheduling that adapts to workload wants and GPU availability

• Automated useful resource administration that adjusts in actual time primarily based on load

• No handbook configuration of GPU slices, MIG cases, or clusters

• Environment friendly GPU utilization with out overprovisioning or useful resource waste

• A constant and remoted runtime surroundings for all fashions

• Builders can deal with functions whereas Clarifai handles infrastructure

Compute Orchestration abstracts away the infrastructure work required to share GPUs successfully. You get higher utilization, smoother scaling, and nil friction transferring from prototype to manufacturing. If you wish to discover additional, take a look at the getting began information.

Conclusion

On this weblog, we went over what GPU fractioning is and the way it works utilizing strategies like TimeSlicing and MIG. These strategies allow you to run a number of fashions on the identical GPU by dividing up compute and reminiscence.

We additionally discovered how Clarifai Compute Orchestration handles GPU fractioning on the orchestration layer. You may spin up devoted compute tailor-made to your workloads, and Clarifai takes care of scheduling and scaling primarily based on demand.

Able to get began? Join Compute Orchestration immediately and be a part of our Discord channel to attach with consultants and optimize your AI infrastructure!