Add support for S3 virtual-hosted-style URLs

[Copilot]

Pull request description

Testkube's MinIO client always used path-style URLs (https://endpoint.com/bucket/object), breaking S3-compatible storage that only supports virtual-hosted-style URLs (https://bucket.endpoint.com/object). This caused duplicate bucket names in paths and 403 errors with services like Tencent Cloud COS CDC.

Checklist (choose whats happened)

• breaking change! (describe)
• tested locally
• tested on cluster
• added new dependencies
• updated the docs
• added a test

Breaking changes

None

Changes

• Config: Added StorageUseVirtualHostedStyle boolean (env: STORAGE_USE_VIRTUAL_HOSTED_STYLE, default: false)
• MinIO client: Sets BucketLookup to BucketLookupDNS when enabled, BucketLookupPath when disabled
• Helm chart: Added storage.useVirtualHostedStyle option with environment variable mapping

Usage:

testkube-api:
  storage:
    endpoint: "bucket-name.cluster-id.cos-cdc.ap-hongkong.myqcloud.com"
    bucket: "bucket-name"
    useVirtualHostedStyle: true

Fixes

• Enables compatibility with S3 services requiring virtual-hosted-style URLs (Tencent Cloud COS CDC, AWS S3 post-deprecation)
• Prevents duplicate bucket names in URL paths that caused 403 AccessDenied errors

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• atomicgo.dev
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• cloud.google.com
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• dario.cat
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• filippo.io
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-df35dde677ee -j DROP /go/pkg/mod/cache/vcs/151ee5770fba399f4d420b82471cb3084bd8f4d1773e6b4b4fb14378ac141bd7/shallow.l/usr/lib/systemd/systemd-sysctl fetch /bin/git --depth=1 origin 2b087070 /usr/lib/git-core/git --sh�� /home/dependabot/go/pkg/mod/cach-n1 om/mailru/easyjson /bin/git --pack_header=2,/usr/sbin/iptables e/git test git (dns block)
• go.googlesource.com
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)
• go.mongodb.org
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• go.opentelemetry.io
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• go.uber.org
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• go.yaml.in
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• gomodules.xyz
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
• google.golang.org
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)
• gopkg.in
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)
• k8s.io
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)
• sigs.k8s.io
  • Triggering command: /update-job-proxy /update-job-proxy s/tests/v1 mpile rg/[email protected] -o ecrets/mapper.go mpile rg/[email protected]/pkg/tool/linux_amd64/compile (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy DROP ps e/git ls-r�� -q REDACTED /usr/lib/git-core/git =fetch log ps (dns block)
  • Triggering command: /update-job-proxy /update-job-proxy -o br-5bb0e5e50d5f -j DROP 672026ac74fb69c8--detach dabf77401b04b57597914595d1708830--quiet /bin/git add %H %ct %D /bin/git git init�� tformat id e/git =2 log /bin/git e/git (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add Support for S3 Virtual-Hosted-Style URLs</issue_title>
<issue_description>Problem Description
Testkube currently cannot work with S3-compatible storage services that only support virtual-hosted-style URLs (e.g., Tencent Cloud COS CDC). This limitation prevents users from using Testkube with certain cloud providers' object storage services.

Current Behavior
When configuring Testkube with external S3-compatible storage, the MinIO client always uses path-style URLs, even when AWS_S3_FORCE_PATH_STYLE=false is set.

Example configuration:

global:
  storage:
    endpoint: "bucket-name.cluster-id.cos-cdc.ap-hongkong.myqcloud.com"
    outputsBucket: "bucket-name"

testkube-api:
  storage:
    endpoint: "bucket-name.cluster-id.cos-cdc.ap-hongkong.myqcloud.com"
    bucket: "bucket-name"
  extraEnvVars:
    - name: AWS_S3_FORCE_PATH_STYLE
      value: "false"

Current URL generated:

https://bucket-name.cluster-id.cos-cdc.ap-hongkong.myqcloud.com/bucket-name/testworkflows/...

This creates a duplicate bucket name in the path, resulting in errors like:

failed to save file in the object storage: ContentLength=3067 with Body length 0

AccessDenied (403) errors

Root Cause
According to minio-go PR kubeshop/testkube#1949:

"Tencent Cloud COS does not support Path-style, so we need to use virtual-hosted-style."

When the endpoint is configured with the bucket name (virtual-hosted-style), Testkube's MinIO client still appends the bucket name to the URL path, creating invalid requests.

Expected Behavior
Correct virtual-hosted-style URL should be:

https://bucket-name.cluster-id.cos-cdc.ap-hongkong.myqcloud.com/testworkflows/...

The bucket name should not be added to the path when using virtual-hosted-style endpoints.

Use Cases
This affects users deploying Testkube with:

Tencent Cloud COS CDC (Cloud Dedicated Cluster) - only supports virtual-hosted-style

Other S3-compatible services that enforce virtual-hosted-style URLs

AWS S3 (after path-style deprecation)

Proposed Solution
Add a new Helm configuration option to support virtual-hosted-style:

Option 1: Explicit flag

testkube-api:
  storage:
    endpoint: "bucket-name.endpoint.com"
    bucket: "bucket-name"
    useVirtualHostedStyle: true  # NEW OPTION

Option 2: Auto-detection
Automatically detect if the endpoint already contains the bucket name and skip adding it to the path.

Option 3: Environment variable

testkube-api:
  extraEnvVars:
    - name: STORAGE_USE_VIRTUAL_HOSTED_STYLE
      value: "true"

Implementation Details
When useVirtualHostedStyle: true:

The MinIO client should use the endpoint as-is

The bucket name should not be prepended to the object key path

URL construction: https://{endpoint}/{key} instead of https://{endpoint}/{bucket}/{key}

Workarounds Attempted
❌ Setting AWS_S3_FORCE_PATH_STYLE=false - still generates path-style URLs

❌ Removing bucket from endpoint - generates path-style URLs

❌ Including bucket in endpoint - creates duplicate bucket in path

✅ Using local MinIO pod (current workaround, but loses cloud storage benefits)

Additional Context
Testkube version: 2.1.236

Storage backend: Tencent Cloud COS CDC

Endpoint format: bucket-name.cluster-id.cos-cdc.region.myqcloud.com

References
Minio-go PR kubeshop/testkube#1949: Add support Tencent Cloud COS virtual hosted style

Velero Issue kubeshop/testkube#7624: Support Tencent COS virtual-styled domain

AWS S3 Path-Style Deprecation
</issue_description>

Comments on the Issue (you are @copilot in this section)

@vsukhin hey @narsdk thank you for great suggeston! cc @olensmar @jmorante-ks for planning @olensmar thanks @narsdk for this 🙏 @vsukhin any objection to letting copilot have a go at the suggested option 1? @vsukhin no, let's try it!

• Fixes Add Support for S3 Virtual-Hosted-Style URLs #6890

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[vsukhin]

looks good, but need to be tested