Create VGPU changes with VFIO Framework #139

JunAr7112 · 2025-11-08T00:34:17Z

No description provided.

copy-pr-bot · 2025-11-08T00:34:20Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Arjun <[email protected]>

tariq1890 · 2025-11-10T17:42:44Z

pkg/vgpu/config.go

+	return 0, fmt.Errorf("vGPU type %s not found in file %s", vgpuTypeName, filePath)
+}
+
+func (m *nvlibVGPUConfigManager) IsVFIOEnabled() (bool, error) {


In this method, you're checking if there are occurrences of ubuntu 24.04 or rhel 10 in the /etc/os-release file. How does that tell us if VFIO is enabled? I also don't see the receiver m *nvlibVGPUConfigManager being used anywhere in this method

Ok, I see it would probably be better to just check the directories directly rather than look for distro. I updated the method to check for devices in /sys/class/mdev_bus.

tariq1890 · 2025-11-10T17:44:48Z

pkg/vgpu/config.go

+		return nil, fmt.Errorf("unable to get GPU by index %d: %v", gpu, err)
+	}
+	vgpuConfig := types.VGPUConfig{}
+	VFnum := 0


let's not use capitalisation when naming local variables

Updated to remove capitalization

tariq1890 · 2025-11-10T17:45:59Z

pkg/vgpu/config.go

+		return fmt.Errorf("GPU at index %d not found in available NVIDIA devices", gpu)
+	}
+
+	cmd := exec.Command("chroot", "/host", "/run/nvidia/driver/usr/lib/nvidia/sriov-manage", "-e", nvdevice.Address)


we shouldn't be hardcoding the /run/nvidia/driver/ path here. They should be retrieved from a parameter instead. The param is called driverRoot

Do you know where driverRoot is defined? Should I manually define it as a constant in the file?

I was having some difficulties running sriov-manage using driverRoot from within the container as the container is built as a distroless image and cannot run sriov-manage which is a bash script

Signed-off-by: Arjun <[email protected]>

cdesiniotis

Thanks @JunAr7112, this is a good start. As we make iterations on this and get more familiar with the internals here, it may be valuable to create a new internal/vgpu package that hides away the vfio vs mdev framework complexity. We need to think through what the right interface would be, but I imagine we will need methods for 1) getting all vGPU devices, 2) getting all parent devices (of which you can create a vGPU device on top of), 3) creating a vGPU device. The pkg/vgpu/config.go file, which is concerned with getting / setting a particular vGPU config, can invoke these methods without having to know what vfio / mdev is.

cdesiniotis · 2025-11-18T16:59:43Z

pkg/vgpu/config.go

+    // Check if mdev_bus exists and has entries
+    mdevBusPath := "/sys/class/mdev_bus"


Is this the right check? Is it possible for mdev_bus to exist in cases where we want to use the VFIO framework?

cdesiniotis · 2025-11-18T17:17:12Z

pkg/vgpu/config.go

+	}
+	vgpuConfig := types.VGPUConfig{}
+	vfnum := 0
+	if nvdevice.SriovInfo.PhysicalFunction == nil {


nit

Suggested change

if nvdevice.SriovInfo.PhysicalFunction == nil {

if nvdevice.SriovInfo.IsVF() {

Switched to this.

cdesiniotis · 2025-11-18T17:26:18Z

pkg/vgpu/config.go

+		if _, err := os.Stat(vfAddr); err != nil {
+			vfnum++
+			continue
+		}


Why are we incrementing vfnum here when the directory does not exist? As I said above, we can just use the number of VFs already calculated when constructing the NVIDIA PCI device using go-nvlib.

I'll remove this check.

cdesiniotis · 2025-11-18T17:27:31Z

pkg/vgpu/config.go

+		return vgpuConfig, nil
+	}
+	totalVF := int(nvdevice.SriovInfo.PhysicalFunction.TotalVFs)
+	for vfnum < totalVF {


Question -- don't we already know the number of VFs from nvdevice.SriovInfo.PhysicalFunction.NumVFs?

I think we should be able to iterate over 0....nvdevice.SriovInfo.PhysicalFunction.NumVFs. The other virtual functions outside that range should be empty. I'll switch to this.

Would we ever run into a scenario where say devices were manually deployed on Virtual Functions in a non-sequential order?

cdesiniotis · 2025-11-18T17:27:54Z

pkg/vgpu/config.go

+	}
+	totalVF := int(nvdevice.SriovInfo.PhysicalFunction.TotalVFs)
+	for vfnum < totalVF {
+		vfAddr := HostPCIDevicesRoot + "/" + nvdevice.Address + "/virtfn" + strconv.Itoa(vfnum) + "/nvidia"


nit: when constructing file paths let's use filepath.Join() throughout.

Switched to filepath.Join

cdesiniotis · 2025-11-18T17:48:25Z

pkg/vgpu/config.go

+		remainingToCreate := val
+		for remainingToCreate > 0 {
+			vfAddr := HostPCIDevicesRoot + "/" + nvdevice.Address + "/virtfn" + strconv.Itoa(vfnum) + "/nvidia"
+			number, err := m.getVGPUTypeNumberforVFIO(vfAddr + "/creatable_vgpu_types", key)


The name of this method could be improved. Maybe getIdForVGPUTypeName.

Switched to this.

cdesiniotis · 2025-11-18T17:49:24Z

pkg/vgpu/config.go

+			vfnum++
+			continue
+		}
+		vgpuTypeName, err := m.getVGPUTypeNameforVFIO(vfAddr + "/creatable_vgpu_types", vgpuTypeNumber)


The name of this method could be improved. Maybe getVGPUTypeNameForId

Switched to this.

cdesiniotis · 2025-11-18T17:52:49Z

pkg/vgpu/config.go

+	IsVFIOEnabled() (bool, error)
+	GetVGPUConfigforVFIO(gpu int) (types.VGPUConfig, error)
+	SetVGPUConfigforVFIO(gpu int, config types.VGPUConfig) error


I would argue this Interface should not change. The caller should just call GetVGPUConfig / SetVGPUConfig and the details about vfio / mdev should be hidden from them.

Ok I switched back to the old interface names. We will use the VFIO enabled check in this file instead.

cdesiniotis · 2025-11-18T17:53:25Z

cmd/nvidia-vgpu-dm/assert/config.go

 	matched := make([]bool, len(gpus))
 	err = WalkSelectedVGPUConfigForEachGPU(c.VGPUConfig, func(vc *v1.VGPUConfigSpec, i int, d types.DeviceID) error {
 		configManager := vgpu.NewNvlibVGPUConfigManager()
-		current, err := configManager.GetVGPUConfig(i)


Let's keep the original interface and just call configManager.GetVGPUConfig(i) here. The vfio/mdev details can be captured in that method.

Switched to this.

cdesiniotis · 2025-11-18T17:54:58Z

cmd/nvidia-vgpu-dm/apply/config.go

 func VGPUConfig(c *Context) error {
 	return assert.WalkSelectedVGPUConfigForEachGPU(c.VGPUConfig, func(vc *v1.VGPUConfigSpec, i int, d types.DeviceID) error {
 		configManager := vgpu.NewNvlibVGPUConfigManager()
-		current, err := configManager.GetVGPUConfig(i)


Let's keep the original interface and just call configManager.GetVGPUConfig(i) / configManager.SetVGPUConfig(i) here. The vfio/mdev details can be captured in those methods.

Switched to this.

Signed-off-by: Arjun <[email protected]>

JunAr7112 added 2 commits November 6, 2025 18:57

created vfio framework

1c55d02

updated to one file

b17d50a

JunAr7112 force-pushed the vfio_changes branch from fffa26f to cbe892c Compare November 10, 2025 16:06

updated setVGPUConfig

3f31828

Signed-off-by: Arjun <[email protected]>

JunAr7112 force-pushed the vfio_changes branch from cbe892c to 3f31828 Compare November 10, 2025 17:09

tariq1890 reviewed Nov 10, 2025

View reviewed changes

JunAr7112 added 2 commits November 10, 2025 18:22

updated variables

29b3788

updated isVFIOEnabled

4a52cbe

Signed-off-by: Arjun <[email protected]>

JunAr7112 force-pushed the vfio_changes branch from 1682d72 to 4a52cbe Compare November 10, 2025 18:58

JunAr7112 requested a review from cdesiniotis November 12, 2025 21:28

cdesiniotis reviewed Nov 18, 2025

View reviewed changes

JunAr7112 force-pushed the vfio_changes branch 5 times, most recently from 8846795 to 5daf473 Compare November 24, 2025 16:54

updated with feedback

89a1e62

JunAr7112 force-pushed the vfio_changes branch from 5daf473 to 89a1e62 Compare November 25, 2025 21:03

JunAr7112 force-pushed the vfio_changes branch 6 times, most recently from 15a9586 to b1fd32d Compare December 4, 2025 17:17

added vfio framework

82b9164

Signed-off-by: Arjun <[email protected]>

JunAr7112 force-pushed the vfio_changes branch from b1fd32d to 82b9164 Compare December 4, 2025 17:34

added vgpu-combined

50e2185

Signed-off-by: Arjun <[email protected]>

JunAr7112 force-pushed the vfio_changes branch from ae21d52 to 50e2185 Compare December 5, 2025 22:43

		// Check if mdev_bus exists and has entries
		mdevBusPath := "/sys/class/mdev_bus"

	if nvdevice.SriovInfo.PhysicalFunction == nil {
	if nvdevice.SriovInfo.IsVF() {

Create VGPU changes with VFIO Framework #139

Are you sure you want to change the base?

Create VGPU changes with VFIO Framework #139

Uh oh!

Conversation

JunAr7112 commented Nov 8, 2025

Uh oh!

copy-pr-bot bot commented Nov 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdesiniotis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JunAr7112 Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

JunAr7112 Nov 19, 2025 •

edited

Loading