I work with VERL often, and setting up a fresh environment kept turning into the same fight: VERL, PyTorch, and flash-attn would disagree about some part of the stack. The installation could finish cleanly, only for import flash_attn to fail with an undefined symbol.

Docker was usually the fastest way out. A known image pins most of the compatibility matrix for you. But it does not explain which part of the local environment was wrong, and it is not always the setup I want.

The useful shift was to stop treating this as a pip problem. Installing flash-attn and loading its compiled extension are two different tests. A working installation depends on a particular combination of Python, PyTorch, CUDA, platform, GPU architecture, and C++ ABI. Once I started checking those dimensions explicitly, the failures became much less mysterious.

This guide is the diagnostic I wanted when I first ran into the problem. It focuses on FlashAttention-2 with CUDA on Linux. The project documents separate ROCm backends, while its Windows support is still described as less tested. Check those paths before applying CUDA/Linux wheel advice to another platform.

The install succeeds. The import fails.

The failure usually looks like this:

pip install flash-attn --no-build-isolation
python -c "import flash_attn"

The first command succeeds. The second fails with an undefined symbol or a failure to load flash_attn_2_cuda.

That does not necessarily mean the download failed. FlashAttention’s build script first constructs a release-asset URL from the active environment. If the asset is unavailable, it falls back to a local source build. Either route can produce a package that installs but cannot load if the build and runtime environments differ.

Separate the compatibility layers

Several values that people call “the CUDA version” answer different questions:

Layer How to inspect it Why it matters
Python interpreter python --version A compiled wheel carries a Python implementation/version tag such as cp310.
Wheel tags accepted by pip python -m pip debug --verbose The last three wheel-name fields are Python, Python ABI, and platform compatibility tags.
PyTorch build torch.__version__ and torch.version.cuda FlashAttention release assets encode the PyTorch major/minor version and CUDA family used by the installed PyTorch build.
PyTorch C++ ABI torch.compiled_with_cxx11_abi() The extension must use the same libstdc++ ABI as PyTorch.
Local CUDA toolkit nvcc --version A source build needs a supported compiler toolkit. This can differ from the CUDA version reported by PyTorch.
Driver-visible GPU nvidia-smi and PyTorch device queries The driver and GPU architecture determine whether the compiled kernel can run.

Do not choose a wheel from nvidia-smi alone. For the project’s prebuilt-wheel lookup, the relevant CUDA token comes from torch.version.cuda, not from the locally installed nvcc.

Capture the environment first

Before downloading a wheel, record the environment that must load it:

import platform
import sys
import torch

print("Python:", sys.version)
print("Platform:", platform.system(), platform.machine())
print("PyTorch:", torch.__version__)
print("PyTorch CUDA build:", torch.version.cuda)
print("CUDA available:", torch.cuda.is_available())

if callable(getattr(torch, "compiled_with_cxx11_abi", None)):
    print("C++11 ABI:", torch.compiled_with_cxx11_abi())
elif hasattr(torch, "_C") and hasattr(torch._C, "_GLIBCXX_USE_CXX11_ABI"):
    # Compatibility fallback for PyTorch versions without the public helper.
    print("C++11 ABI:", bool(torch._C._GLIBCXX_USE_CXX11_ABI))

if torch.cuda.is_available():
    print("GPU:", torch.cuda.get_device_name(0))
    print("Compute capability:", torch.cuda.get_device_capability(0))

The output is evidence, not yet a wheel recommendation. Release assets differ by FlashAttention version, and not every environment has a prebuilt wheel.

The repository also includes scripts/flash_attn_environment.py, which prints these values together and produces a compatibility search key without downloading or installing anything:

python scripts/flash_attn_environment.py
python scripts/flash_attn_environment.py --json

Decode the wheel name

For example, the FlashAttention-2 build script can construct an asset name shaped like:

flash_attn-2.7.3+cu12torch2.6cxx11abiTRUE-cp310-cp310-linux_x86_64.whl

The important fields describe:

Field Example Meaning
FlashAttention 2.7.3 Library release
CUDA family cu12 CUDA build family
PyTorch torch2.6 PyTorch compatibility
PyTorch C++ ABI cxx11abiTRUE libstdc++ ABI used to build PyTorch
Python tag first cp310 CPython 3.10
Python ABI tag second cp310 CPython 3.10 extension ABI
Platform tag linux_x86_64 Linux on x86-64

The cu12torch2.6cxx11abiTRUE segment is FlashAttention’s own release naming convention. The final cp310-cp310-linux_x86_64 fields follow the wheel compatibility-tag structure. Match all of them, then verify that the named asset actually exists for the release; a plausible filename is not proof that maintainers published that wheel.

Install and verify

Install the downloaded artifact explicitly:

python -m pip install ./flash_attn-<matching-tags>.whl
python -c "import flash_attn; print(flash_attn.__version__)"

An import checks that Python can load the extension. It does not execute a kernel. On a supported CUDA GPU, this small forward pass adds a useful second smoke test:

import torch
from flash_attn import flash_attn_func

q = torch.randn(1, 128, 4, 64, device="cuda", dtype=torch.float16)
k = torch.randn_like(q)
v = torch.randn_like(q)

with torch.inference_mode():
    output = flash_attn_func(q, k, v, dropout_p=0.0, causal=True)

torch.cuda.synchronize()
print("shape:", tuple(output.shape))
print("finite:", bool(torch.isfinite(output).all().item()))

This only proves that one forward configuration runs. A project-level check should still exercise the model, dtype, device, forward/backward path, and tensor shapes you intend to deploy.

When a wheel is not enough

A compatible prebuilt wheel may not exist for a new PyTorch, CUDA, Python, architecture, or platform combination. In that case the realistic options are:

  1. Use a supported environment for which a wheel exists.
  2. Build from source using the project’s documented toolchain.
  3. Defer FlashAttention and use a supported attention implementation.

For a source build, verify the project’s stated requirements first: a supported PyTorch version, CUDA toolkit, packaging, psutil, and a working ninja. If compilation exhausts memory, the project documents MAX_JOBS for limiting parallel build jobs:

MAX_JOBS=4 python -m pip install flash-attn --no-build-isolation

A source build is not automatically safer. It still depends on the compiler toolkit, headers, build isolation, available memory, the target GPU architecture, and the active PyTorch environment.

A short diagnosis order

  1. Confirm that the supported hardware and operating-system scope fits your machine.
  2. Run the environment report from the same interpreter that will import FlashAttention.
  3. Check the exact release assets; do not construct a URL and assume it exists.
  4. Install with python -m pip so the installer and import test use the same interpreter.
  5. Run the import test, then a kernel smoke test, then your real workload.
  6. If the import reports an undefined symbol, compare the PyTorch version and C++ ABI before reinstalling at random.

Limits of this guide

Release assets and supported hardware change. Treat wheel examples as naming examples, not permanent installation instructions. Always inspect the assets and installation notes for the exact FlashAttention release you intend to use.

References