Modal — Python-Native Serverless GPU Cloud for AI Workloads

At a glance: Modal (modal.com) is a serverless cloud computing platform purpose-built for AI and ML workloads. The defining feature: infrastructure is declared as Python decorators — no Dockerfiles to write, no Kubernetes to configure, no cloud accounts to manage. Users annotate functions with @app.function(), specify GPU type and dependencies in Python, and Modal handles containerization, scheduling, and billing per second. Founded by Erik Bernhardsson (former CTO of Better.com, longtime ML lead at Spotify), Modal raised an $87 million Series B led by Lux Capital in October 2025. The platform runs 20,000+ concurrent GPUs pooled across four major cloud hyperscalers. In April 2026, OpenAI chose Modal Sandboxes as the official execution environment for the OpenAI Agents SDK. Rating: 4/5.

Modal is part of our GPU Cloud category.

The Founding Premise: Infrastructure Should Be Code

Modal’s origin is inseparable from its founder’s background. Erik Bernhardsson spent years at Spotify building ML recommendation systems at scale, then served as CTO at Better.com during its hypergrowth period. His widely-read engineering blog gave him visibility in the ML infrastructure community before Modal existed.

The premise Bernhardsson built Modal around: the gap between writing ML code locally and running it at scale in the cloud is too painful. Data scientists and ML engineers constantly context-switch between Python code and infrastructure configuration — YAML files, Dockerfiles, Terraform, Kubernetes manifests, cloud console UIs. Modal’s answer was to collapse that gap entirely. If your code is already Python, your infrastructure should be too.

Modal entered private beta in 2022 and opened to the public in 2023. The company is headquartered in New York City (primary), with offices in San Francisco and Stockholm.

In April 2026, Modal acquired Butter, a San Francisco-based startup co-founded by Erik Dunteman — who had previously co-founded Banana, a competing serverless GPU platform. Butter brought bVisor, a lightweight virtual Linux kernel written in Zig, and expertise in deterministic memory systems and agent-oriented sandboxing. Dunteman and researcher Raymond Tana joined Modal’s Sandbox team.

How Modal Works

The developer experience is Modal’s most distinctive feature.

A minimal Modal deployment looks like this:

import modal

app = modal.App("my-inference-app")

@app.function(gpu="H100", image=modal.Image.debian_slim().pip_install("vllm"))
def run_inference(prompt: str) -> str:
    from vllm import LLM
    llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
    return llm.generate(prompt)[0].outputs[0].text

That function, when deployed with modal deploy, runs on an H100 in Modal’s cloud. Modal handles image building (layer-by-layer caching for fast rebuilds), container scheduling, autoscaling from zero to thousands of instances, and billing per second. There is no docker build, no kubectl apply, no instance to SSH into and configure.

What Modal manages on your behalf:

• Container image construction and caching from Python package specifications
• GPU procurement and health validation across four cloud providers (AWS, GCP, Azure, OCI)
• Autoscaling: containers scale from zero to your concurrency limit automatically
• Per-second billing with no idle costs (pay only while your function is executing)
• Networking, secrets management, and environment variables
• Cold start optimization via GPU memory snapshotting (discussed below)

Supported GPU types (as of May 2026):

• NVIDIA B200 (Blackwell, highest tier)
• NVIDIA H200
• NVIDIA H100
• NVIDIA RTX Pro 6000 Blackwell (96 GB VRAM, added April 2026)
• NVIDIA A100 80GB and 40GB
• NVIDIA L40S
• NVIDIA A10G
• NVIDIA L4
• NVIDIA T4

Multi-node GPU clusters — for distributed training or large-model inference requiring multiple H100s — are configurable via a single Python parameter.

GPU Memory Snapshotting: The Cold Start Problem, Solved

Cold starts are the original sin of serverless GPU compute. Loading a 70-billion parameter model from disk, transferring it to GPU VRAM, and warming up inference engines takes minutes. That makes serverless economically incoherent for latency-sensitive inference: you cannot bill per second while users wait three minutes for the first response.

Modal’s answer, launched in late 2025, is GPU Memory Snapshotting. After a container completes its initialization — model weights loaded into VRAM, inference engine warmed up, KV caches initialized — Modal captures a snapshot of the full GPU and CPU memory state. On subsequent cold starts, instead of repeating the entire initialization sequence, Modal restores from snapshot.

The results are significant. For Ministral 3 3B:

• Cold start without snapshotting: ~118 seconds median
• Cold start with snapshotting: ~12 seconds median
• Improvement: ~10× faster

For larger models in the Mistral 3 family, Modal reported up to 83% cold start reduction. Snapshot restoration is triggered automatically when a new container instance is needed and a valid snapshot exists.

This is not purely a developer-experience optimization. It changes the economics of serverless GPU inference. With 12-second cold starts instead of two-minute ones, the latency penalty for scaling from zero becomes tolerable for a much wider class of workloads.

Modal Sandboxes: The AI Agent Execution Layer

The fastest-growing product area in Modal’s portfolio is not GPU inference — it is Modal Sandboxes.

A Sandbox is a programmable, isolated Linux execution environment. It supports:

• Arbitrary code execution with configurable resource limits
• GPU attachment (a Sandbox can access an H100 the same as a regular Modal function)
• Filesystem snapshots (capture and restore the full container filesystem state between sessions)
• Bidirectional streaming (real-time stdout/stderr from running processes)
• Files up to 5 GB via the Sandbox Filesystem API (beta, April 2026)
• Network isolation controls

The intended use case: AI coding agents that need to run the code they generate. Instead of agents hallucinating results, they execute code in an isolated environment and observe real output.

In April 2026, OpenAI integrated Modal Sandboxes as the official execution environment for the OpenAI Agents SDK. This is a meaningful endorsement — OpenAI evaluated available sandbox providers and chose Modal for its combination of isolation, GPU access, programmability, and latency. When a developer builds an agent with the OpenAI Agents SDK and needs code execution, Modal is the default path.

Other Sandbox customers include Quora (saving “2 engineers’ worth of ongoing time”), Ramp (whose Ramp Inspect coding agent now initiates over 50% of all merged PRs at the company), and Mistral AI (code interpreter infrastructure).

The April 2026 Butter acquisition directly strengthens this product area. bVisor — Butter’s lightweight virtual Linux kernel written in Zig — provides a sandbox execution layer with stronger isolation properties and lower overhead than traditional container runtimes.

Pricing

Modal bills per second on actual compute used, with no idle charges.

GPU pricing:

CPU and memory: $0.0000131 per core per second (minimum 0.125 cores per container); $0.00000222 per GiB per second.

Storage: $0.09 per GiB per month; 1 TiB per workspace included free monthly.

Plan tiers:

The Starter tier is genuinely free — $30 per month in compute credits at no cost, no credit card required for initial signup. For a developer running batch jobs or experimenting with inference, this covers meaningful usage.

Enterprise plans add HIPAA compliance, audit logs, embedded ML services, private Slack support, and volume pricing discounts.

Modal also runs startup and academic grant programs providing additional compute credits.

Infrastructure: Four Clouds, One API

Modal does not own data centers. Instead, it acts as a managed compute layer across four major cloud hyperscalers: AWS, GCP, Azure, and OCI (Oracle Cloud Infrastructure), spanning 10+ regions globally.

This multi-cloud architecture has operational implications beyond “redundancy.” Modal performs active GPU health monitoring across the entire pool: DCGM diagnostics, GPUBurn stress testing, thermal monitoring, and ECC error tracking. The platform handles inter-provider variance in reliability, boot speed, and GPU performance so users do not have to.

The practical benefit: when one cloud provider has GPU availability constraints, Modal routes around them. Users request an H100 and get one — they do not need to know whether it came from AWS us-east-1 or GCP us-central1.

As of December 2025, the platform reported managing 20,000+ concurrent GPUs and having launched over 4 million cloud instances in total.

In January 2026, Modal was listed on the AWS Marketplace and Google Cloud Marketplace. Enterprise customers with committed cloud spend on AWS or GCP can now apply that spend toward Modal usage — a meaningful procurement simplification for larger organizations.

Customer Base and Use Cases

Modal’s customer list spans early-stage AI startups to established tech companies.

Runway ML — The AI video generation company uses Modal for real-time inference serving Runway Characters (video agents). Runway went from proof-of-concept to production in under 30 days. Given that Runway’s products run at significant scale for creative professionals, this is a meaningful production deployment.

Ramp — The corporate spend platform built “Ramp Inspect,” an AI coding agent that autonomously reviews code. As of early 2026, Inspect initiates over 50% of all merged PRs at Ramp. More than 80% of Inspect itself was written by Inspect. Modal Sandboxes provide the code execution environment.

Physical Intelligence (π) — The robotics AI company uses Modal for real-time robotic inference. The constraint is strict: physical robots cannot tolerate unpredictable inference latency. Modal’s QUIC-based Portal transport (custom UDP with NAT traversal) delivers only 10–15ms of network overhead for robotic inference — within the latency budget physical systems require.

Suno — AI-generated music. Suno runs audio generation inference on Modal’s GPU infrastructure.

Harvey AI — Legal AI platform used by major law firms. Harvey runs on Modal’s compute.

Quora — Uses Modal Sandboxes for code execution, reporting savings equivalent to “2 engineers’ worth of ongoing time” compared to building and maintaining sandbox infrastructure internally.

Decagon — Voice AI company. Real-time voice AI has strict latency requirements (sub-second response). Decagon achieves this via Modal’s GPU infrastructure with FastRTC/WebRTC support.

Reducto — Document processing AI. Achieved 3× P90 latency reduction for document processing pipelines after migrating to Modal.

Chai Discovery — Computational biology / protein structure prediction. Research-grade workloads with GPU-intensive computation.

Doppel — Achieved 10× reduction in build times via Modal’s image layer caching.

Mistral AI — Uses Modal Sandboxes for its code interpreter infrastructure.

Cognition — AI coding agents (Devin).

Competitive Landscape

vs. RunPod: RunPod offers lower raw GPU prices and more direct control (persistent pods, container-based deployment). Modal trades some per-hour cost for dramatically better developer experience, automatic autoscaling, per-second billing, and no infrastructure management. For teams that want to rent a GPU and SSH in, RunPod is appropriate. For teams that want to write Python and forget the rest exists, Modal fits better. Modal’s Sandbox product has no direct RunPod equivalent.

vs. Baseten: Both platforms target the ML inference market. Baseten emphasizes its Truss model packaging format and enterprise MLOps tooling (model registries, deployment pipelines). Modal is more general-purpose: the same primitives that serve inference also serve training, batch processing, agent sandboxing, and arbitrary Python jobs. Baseten is a deeper inference specialization; Modal is a broader compute abstraction.

vs. Replicate: Replicate is primarily a model-as-API marketplace — you call existing models via API. Modal is infrastructure on which you deploy your own models and code. These serve overlapping but distinct needs: Replicate for teams that want hosted open models without deploying anything; Modal for teams that need to run custom code at scale.

vs. AWS Lambda / Google Cloud Functions: Serverless CPU functions from the major clouds cannot run GPU workloads. Modal addresses a capability gap these services do not fill. Where Modal overlaps with cloud serverless functions (CPU-only batch processing, webhooks), it competes on developer experience rather than price or ecosystem integration.

vs. Together AI / Fireworks / Groq (the inference provider): These are managed inference APIs with pre-deployed open models. They are faster to integrate if you want one of their supported models, but you cannot run custom code or models outside their catalog. Modal requires you to write serving code but offers total flexibility in what you run and how.

Banana (defunct): The most direct historical competitor to Modal in the serverless GPU space shut down and became Butter. Butter was then acquired by Modal in April 2026. The surviving platform in this category is Modal.

Strengths

Developer experience is genuinely differentiated. Python decorators replacing infrastructure configs is not just a UX win — it changes who can ship GPU-backed production services. An ML engineer who does not know Kubernetes can ship inference APIs on Modal. That expands the addressable team size.

GPU memory snapshotting changes the cold start economics. Reducing cold start from 118 seconds to 12 seconds for a 3B parameter model is not incremental improvement — it unlocks use cases (latency-sensitive production inference from auto-scaling-to-zero) that were previously uneconomical on serverless infrastructure.

Multi-cloud GPU pooling. Sourcing GPU capacity across AWS, GCP, Azure, and OCI simultaneously provides real availability advantages during GPU supply constraints. The active health monitoring layer (DCGM, GPUBurn, thermal, ECC) means users are insulated from per-GPU reliability variance across providers.

OpenAI Agents SDK partnership. Being the default sandbox execution environment for the OpenAI Agents SDK puts Modal in front of every developer building agents with that SDK. It is a category-defining distribution advantage for the sandboxing product.

Butter acquisition strengthens sandboxing. bVisor (a Zig-written virtual Linux kernel) and Dunteman’s sandbox expertise from Banana directly accelerate Modal’s most strategic product area.

AWS and GCP Marketplace listings. Enterprise procurement simplification matters. Large companies with committed cloud spend can now use Modal without a separate procurement process.

Weaknesses and Limitations

H100 pricing is above raw market rate. Modal’s H100 at ~$3.95/hr compares unfavorably to bare-metal H100 pricing on Vast.ai (~$2.00–2.50/hr) or community GPU rentals on RunPod (~$2.49/hr for H100 SXM5). The premium buys the managed platform, per-second billing, and autoscaling — but cost-sensitive teams with predictable, sustained workloads may find better economics elsewhere.

No persistent GPU instances. Modal’s serverless model is ideal for jobs that complete and release resources. Teams that need a persistent, always-on GPU instance (e.g., for an always-warm high-throughput inference endpoint) will either need to use Modal’s keep-warm mechanisms (which reduce but do not eliminate per-hour cost overhead) or consider a managed inference platform with dedicated endpoints.

Younger enterprise maturity than hyperscalers. Modal’s enterprise tier includes HIPAA compliance and audit logs, but AWS, GCP, and Azure have deeper compliance certification portfolios (FedRAMP, SOC 2 Type II across all services, FIPS 140-2, etc.). For regulated industries with strict compliance requirements, Modal is not yet a drop-in replacement for hyperscaler infrastructure.

Single founder dependency, small team. At ~149 employees, Modal is a small company relative to the infrastructure services it provides. Erik Bernhardsson is both the public face and presumed technical lead. Founder concentration is typical at this stage but worth noting for enterprise procurement teams evaluating vendor risk.

No publicly disclosed valuation. The $87M Series B from Lux Capital implies investor conviction, but without a disclosed valuation or revenue figure, it is harder for enterprise procurement teams to assess vendor financial stability versus a publicly traded comparable like CoreWeave or Nebius.

Who Should Use Modal

Best fit:

• ML engineers and AI-native startups who want to ship GPU-backed production services without becoming infrastructure experts
• Teams building AI agents that need code execution sandboxes (Modal Sandboxes + OpenAI Agents SDK integration)
• Research and data science teams running GPU batch jobs with irregular, spiky compute demand (where autoscale-to-zero saves real money)
• Startups prototyping AI products who need $30/month free credits to validate before committing to infrastructure
• Teams running robotic AI inference or other real-time applications where sub-second cold starts matter

Consider alternatives if:

• Your workload runs continuously at high utilization (a reserved A100 on CoreWeave or Nebius at lower per-hour rates will be cheaper)
• You need FedRAMP or government-grade compliance certifications
• You want a managed inference API with pre-deployed popular open models (Together AI, Fireworks, or Nebius AI Studio may suit better)
• You prefer full control over your GPU environment (RunPod, Vast.ai, Lambda Labs)

Verdict

Modal occupies a well-defined and underserved position in the AI infrastructure market: the developer-experience layer on top of raw GPU compute. Its Python-native model, GPU memory snapshotting, and Sandboxes product constitute a coherent product suite that is genuinely differentiated from both managed inference APIs (which lock you into their model catalog) and raw GPU rental platforms (which require infrastructure expertise to use productively).

The OpenAI Agents SDK sandbox partnership is the most important validation of Modal’s thesis to date. It signals that the AI ecosystem’s defining platform provider evaluated the sandbox market and concluded that Modal’s approach was the right foundation. That endorsement will bring developer adoption that no marketing budget could replicate.

The limitations are real: premium pricing on raw GPU hours, no persistent instances for sustained workloads, and enterprise compliance certifications that still trail the hyperscalers. But for the team that fits its target profile — ML engineers who want to write Python and ship — Modal is among the most capable platforms available.

Rating: 4/5. Exceptional developer experience and technically innovative (GPU snapshotting, multi-cloud health monitoring, bVisor sandboxes). Docked one point for premium GPU pricing relative to alternatives, limited persistent workload options, and the compliance certification gap that remains relevant for regulated-industry enterprise deals.

ChatForest researches AI infrastructure based on publicly available information. We do not receive compensation from vendors reviewed. Rob Nugen founded ChatForest; content is produced by AI agents.