SDSC Expanse GPU CUDA Unable to Declare RectilinearGrid

[acgoering]

I am trying to run Oceananigans on an SDSC Expanse GPU node, and I am running into an error when I try to declare a RectilinearGrid. I start with an interactive GPU node:

srun --partition=gpu-shared --pty --account=uri107 --ntasks-per-node=20 --nodes=1 --mem=96G --gpus=1 -t 01:00:00 --wait=0 --export=ALL /bin/bash

And then I load CUDA 12.2:

module purge
module restore
module load cuda12.2/toolkit

And then I start Julia:

julia

And run the following commands in REPL:

julia> using CUDA
┌ Warning: CUDA runtime library `libcublasLt.so.12` was loaded from a system path, `/cm/shared/apps/cuda12.2/toolkit/12.2.2/targets/x86_64-linux/lib/libcublasLt.so.12`.
│ This may cause errors.
│ 
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│ 
│ In any other case, please file an issue.
└ @ CUDA ~/.julia/packages/CUDA/Wfi8S/src/initialization.jl:218
┌ Warning: CUDA runtime library `libnvJitLink.so.12` was loaded from a system path, `/cm/shared/apps/cuda12.2/toolkit/12.2.2/targets/x86_64-linux/lib/libnvJitLink.so.12`.
│ This may cause errors.
│ 
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│ 
│ In any other case, please file an issue.
└ @ CUDA ~/.julia/packages/CUDA/Wfi8S/src/initialization.jl:218
┌ Warning: CUDA runtime library `libcusparse.so.12` was loaded from a system path, `/cm/shared/apps/cuda12.2/toolkit/12.2.2/targets/x86_64-linux/lib/libcusparse.so.12`.
│ This may cause errors.
│ 
│ If you're running under a profiler, this situation is expected. Otherwise,
│ ensure that your library path environment variable (e.g., `PATH` on Windows
│ or `LD_LIBRARY_PATH` on Linux) does not include CUDA library paths.
│ 
│ In any other case, please file an issue.
└ @ CUDA ~/.julia/packages/CUDA/Wfi8S/src/initialization.jl:218

julia> grid = RectilinearGrid(GPU(),
                              size = (1024, 1024),
                              x = (-π, π),
                              y = (-π, π),
                              topology = (Periodic, Periodic, Flat))
ERROR: UndefVarError: `GPU` not defined in `Main`
Suggestion: check for spelling errors or missing imports.
Hint: a global variable of this name may be made accessible by importing KernelAbstractions in the current active module Main
Stacktrace:
 [1] top-level scope
   @ REPL[2]:1

It throws an error when I try to declare a RectilinearGrid. I can run

julia> CUDA.versioninfo()
CUDA toolchain: 
- runtime 12.9, artifact installation
- driver 525.85.12 for 12.0
- compiler 12.9

CUDA libraries: 
- CUBLAS: 12.2.5
- CURAND: 10.3.10
- CUFFT: 11.4.1
- CUSOLVER: 11.7.5
- CUSPARSE: 12.5.10
- CUPTI: 2025.2.1 (API 28.0.0)
- NVML: 12.0.0+525.85.12

Julia packages: 
- CUDA: 5.8.3
- CUDA_Driver_jll: 13.0.1+0
- CUDA_Compiler_jll: 0.2.1+0
- CUDA_Runtime_jll: 0.19.1+0

Toolchain:
- Julia: 1.11.7
- LLVM: 16.0.6

1 device:
  0: Tesla V100-SXM2-32GB (sm_70, 31.736 GiB / 32.000 GiB available)

which looks fine. Does anyone know what would be causing this error? Thanks!

[glwagner]

I think you are missing using Oceananigans, right?

I'm also worried about the warnings... you could try unset LD_LIBRARY_PATH, but not sure if anything will work if you do that.

[acgoering]

Yes, that was the issue. I needed to execute using Oceananigans. Turns out that all those warnings that pop up when I run using CUDA aren't actually a problem. I should have realized that RectilinearGrid and GPU() are part of Oceananigans, not Julia or CUDA. I can now run the full GPU example script given here. Next I'm going to try using the numerical wave tank script. Thanks so much for the help!

[glwagner]

nice work!