KubeSphere DevOps Pipeline Management

Overview

Pipelines in KubeSphere DevOps are Kubernetes custom resources that integrate with Jenkins. KubeSphere uses a cloud-native, object-reconcile approach where Kubernetes resources are the source of truth.

When to Use

• Creating or updating CI/CD pipelines
• Triggering pipeline runs
• Monitoring pipeline execution
• Retrieving pipeline logs and artifacts
• Troubleshooting failed pipeline runs

Architecture Mapping

KubeSphere DevOps maps Kubernetes resources to Jenkins objects:

KubeSphere Resource	K8s Resource	Jenkins Resource
DevOpsProject	DevOpsProject CR + Namespace	Folder
Pipeline	Pipeline CR	WorkflowJob
PipelineRun	PipelineRun CR	Build Run
Workspace	Workspace CR	(authorization context)

KubeSphere          Kubernetes                Jenkins
─────────────────────────────────────────────────────────────
Workspace demo
└── DevOpsProject   → demo-project NS          → Folder demo-project
    └── Pipeline    → Pipeline CR              → WorkflowJob
        └── Run     → PipelineRun CR           → Build #1

Triggering Pipeline Runs (Recommended: Object-Reconcile)

CRITICAL: ALWAYS Check for Parameters First!

Before triggering ANY pipeline (regular or multi-branch), you MUST check if the pipeline has parameters defined. Triggering a pipeline without required parameters will cause the build to fail or use incorrect defaults.

For Multi-Branch Pipelines: Query /branches/{branch} endpoint to get .parameters array For Regular Pipelines: Query the Pipeline CR and check .spec.pipeline.jenkinsfile for parameters {} directive

Preferred Approach: Create a PipelineRun custom resource. KubeSphere watches for these resources and triggers the corresponding Jenkins build.

Create a PipelineRun

apiVersion: devops.kubesphere.io/v1alpha3
kind: PipelineRun
metadata:
  name: my-pipeline-run-001
  namespace: demo-project
spec:
  pipelineRef:
    name: my-pipeline
  parameters:
    - name: BRANCH
      value: "main"

Apply with kubectl:

kubectl apply -f pipelinerun.yaml

Check PipelineRun Status

# List all runs
kubectl get pipelineruns -n demo-project

# Get specific run status
kubectl get pipelinerun my-pipeline-run-001 -n demo-project -o yaml

# Watch run progress
kubectl get pipelineruns -n demo-project -w

PipelineRun Status Fields

Field	Description
status.phase	Current state (Pending, Running, Succeeded, Failed, Unknown)
status.conditions	Detailed conditions (Succeeded, Ready)
status.completionTime	When run finished
status.startTime	When run started

Delete a PipelineRun

kubectl delete pipelinerun my-pipeline-run-001 -n demo-project

Working Pipeline Example

Here's a complete, working pipeline that builds a Go application:

apiVersion: devops.kubesphere.io/v1alpha3
kind: Pipeline
metadata:
  name: go-demo-pipeline
  namespace: demo-project
spec:
  type: pipeline
  pipeline:
    name: go-demo-pipeline
    description: "Build and test Go application"
    jenkinsfile: |
      pipeline {
        agent any

        stages {
          stage('Build, Test and Archive') {
            agent {
              kubernetes {
                yaml '''
                  apiVersion: v1
                  kind: Pod
                  spec:
                    containers:
                    - name: golang
                      image: golang:1.21
                      command: ["sleep"]
                      args: ["99d"]
                '''
              }
            }
            steps {
              container('golang') {
                sh '''
                  export GO111MODULE=on
                  git clone https://github.com/kubesphere-sigs/demo-go-http.git .
                  go mod download
                  go test ./... -v
                  go build -o service main.go
                '''
              }
              archiveArtifacts artifacts: 'service', followSymlinks: false
            }
          }
        }
      }

Key Points:

• Uses agent { kubernetes { yaml ... } } to define a custom pod with Go container
• The archiveArtifacts step must be in the same stage as the build (same workspace)
• Container name in container('golang') must match the container name in the YAML

Pipeline Resource

apiVersion: devops.kubesphere.io/v1alpha3
kind: Pipeline
metadata:
  name: my-pipeline
  namespace: demo-project
spec:
  type: pipeline
  pipeline:
    name: my-pipeline
    description: "Build and deploy app"
    jenkinsfile: |-
      pipeline {
        agent any
        stages {
          stage('Build') {
            steps {
              sh 'make build'
            }
          }
        }
      }

Tenant-Created Resources: Creator Annotation

When creating pipelines as a tenant (not cluster-admin), you MUST include the kubesphere.io/creator annotation to properly track ownership:

apiVersion: devops.kubesphere.io/v1alpha3
kind: Pipeline
metadata:
  name: my-pipeline
  namespace: demo-project
  annotations:
    kubesphere.io/creator: "stone-ns-admin"  # Required for tenant-created resources
spec:
  type: pipeline
  pipeline:
    name: my-pipeline
    description: "Pipeline created by tenant"
    jenkinsfile: |-
      pipeline {
        agent any
        stages {
          stage('Build') {
            steps {
              sh 'echo Building...'
            }
          }
        }
      }

Why this matters:

• KubeSphere uses this annotation for ownership tracking
• UI displays creator information
• Required for proper RBAC enforcement
• CRITICAL: Always set this annotation when creating ANY pipeline via API (both regular and multi-branch)

Example API call with creator annotation for regular pipeline:

curl -s -X POST "${KUBESPHERE_API}/kapis/devops.kubesphere.io/v1alpha3/namespaces/demo-project/pipelines" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "devops.kubesphere.io/v1alpha3",
    "kind": "Pipeline",
    "metadata": {
      "name": "my-pipeline",
      "namespace": "demo-project",
      "annotations": {
        "kubesphere.io/creator": "'${USERNAME}'"
      }
    },
    "spec": {
      "type": "pipeline",
      "pipeline": {
        "name": "my-pipeline",
        "description": "Tenant-created pipeline",
        "jenkinsfile": "pipeline { agent any; stages { stage(\"Build\") { steps { sh \"echo hello\" } } } }"
      }
    }
  }'

Multi-Branch Pipeline

Multi-branch pipelines automatically discover branches from SCM and create jobs for each branch. The Jenkinsfile is loaded from the repository.

⚠️ CRITICAL: Always Check Repository Type First

Before creating a multi-branch pipeline, you MUST ask the user:

"Is this a private repository?"

If YES (Private Repo):

1. Ask if they want to use an existing credential or create a new one
2. Create a DevOps credential (basic-auth type with GitHub PAT) if needed
3. Reference the credential in git_source.credential_id
4. (Optional) Create a GitRepository CR for additional metadata

If NO (Public Repo):

• Set credential_id: "" (empty string)

Never assume repository type - always confirm with the user first.

Create Multi-Branch Pipeline

Step 1: Check Repository Type

• Ask user: "Is the repository private?"
• If yes, proceed with credential creation

Step 2: Create Credential (For Private Repos Only)

# Create GitHub credential
export GITHUB_PAT="ghp_xxxxxxxxxxxxxxxxxxxx"

curl -s -X POST "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/credentials" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "v1",
    "kind": "Secret",
    "metadata": {
      "name": "github-token",
      "namespace": "'${DEVOPS_PROJECT}'",
      "annotations": {
        "kubesphere.io/creator": "'${USERNAME}'",
        "credential.devops.kubesphere.io/type": "basic-auth"
      }
    },
    "stringData": {
      "username": "git",
      "password": "'${GITHUB_PAT}'"
    },
    "type": "credential.devops.kubesphere.io/basic-auth"
  }'

Step 3a: For Public Repository:

apiVersion: devops.kubesphere.io/v1alpha3
kind: Pipeline
metadata:
  name: demo-jenkinsfiles-go
  namespace: demo-project
  annotations:
    kubesphere.io/creator: "stone-ns-admin"  # Required for tenant-created resources
spec:
  type: multi-branch-pipeline
  multi_branch_pipeline:
    name: demo-jenkinsfiles-go
    description: "Multi-branch Go pipeline"
    source_type: git
    git_source:
      url: https://github.com/kubesphere/demo-jenkinsfiles
      credential_id: ""  # Empty for public repos
      discover_branches: true
      discover_tags: false
    script_path: go/Jenkinsfile  # Path to Jenkinsfile in repo

Step 3b: For Private Repository (with credential):

apiVersion: devops.kubesphere.io/v1alpha3
kind: Pipeline
metadata:
  name: private-repo-pipeline
  namespace: demo-project
  annotations:
    kubesphere.io/creator: "stone-ns-admin"
spec:
  type: multi-branch-pipeline
  multi_branch_pipeline:
    name: private-repo-pipeline
    description: "Pipeline for private repository"
    source_type: git
    git_source:
      url: https://github.com/org/private-repo.git
      credential_id: "github-token"  # Reference to DevOps credential
      discover_branches: true
      discover_tags: false
    script_path: Jenkinsfile

Complete Flow for Private Repo:

1. Ask user if repository is private (ALWAYS do this first)
2. Create credential (basic-auth type with GitHub PAT)
3. (Optional) Create GitRepository CR (with provider and secret fields)
4. Create multi-branch pipeline referencing the credential in git_source.credential_id

Important: Never use GITHUB_ env vars directly in the pipeline spec. Always create proper DevOps credentials.

SCM Source Types:

Type	Field	Use Case
Git	git_source	Generic Git repositories
GitHub	github_source	GitHub.com or GitHub Enterprise
GitLab	gitlab_source	GitLab.com or self-hosted GitLab
SVN	svn_source	Subversion repositories

Trigger Multi-Branch Pipeline Run

apiVersion: devops.kubesphere.io/v1alpha3
kind: PipelineRun
metadata:
  name: demo-jenkinsfiles-go-main-run
  namespace: demo-project
spec:
  pipelineRef:
    name: demo-jenkinsfiles-go
  scm:
    refName: main    # Branch name
    refType: branch  # or 'tag'

Check Discovered Branches

Via v1alpha3 API (preferred):

curl -s "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}/branches?filter=origin&page=1&limit=10" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" | jq -r '.items[] | "- Branch: \(.name) | Latest Run: \(.latestRun.id // "N/A") | Status: \(.latestRun.result // "N/A")"'

Via Jenkins (admin only):

kubectl run curl-jenkins --rm -i --restart=Never --image=curlimages/curl \
  -- "http://admin:${TOKEN}@devops-jenkins.kubesphere-devops-system:80/job/demo-project/job/demo-jenkinsfiles-go/api/json"

Trigger Repository Scanning

Note: Repository scanning uses v1alpha2 API (not v1alpha3). This is an exception to the general rule of preferring v1alpha3.

Step 1: Trigger Scan

curl -X POST "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha2/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}/scan" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{}'

Step 2: Fetch Scanning Log

curl -s "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha2/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}/consolelog" \
  -H "Authorization: Bearer ${API_TOKEN}"

When to use:

• Force immediate repository re-scan
• Discover new branches that aren't showing up
• Troubleshoot branch detection issues

Example scanning log output:

Started by user admin
Starting branch indexing...
 > git ls-remote --symref -- https://github.com/org/repo.git
Checking branches:
  Checking branch main ✓
      'Jenkinsfile' found
    Met criteria
  Checking branch feature-x ✓
      'Jenkinsfile' found
    Met criteria
  Checking branch old-branch
      'Jenkinsfile' not found
    Does not meet criteria
Processed 3 branches
Finished branch indexing. Indexing took 3 sec
Finished: SUCCESS

Complete Private Repo Setup Example

Scenario: Create a multi-branch pipeline for a private GitHub repository with proper authentication.

Step 1: Ask User About Repository Type

Question: "Is https://github.com/stoneshi-yunify/jenkinsfiles a private repository?"
User Answer: "Yes"

Step 2: Create GitHub Credential

export KUBESPHERE_API="http://kubesphere-apiserver:80"
export API_TOKEN="<tenant-oauth-token>"
export DEVOPS_PROJECT="devopstestc2nj7"
export USERNAME="stone-ns-admin"
export GITHUB_PAT="ghp_xxxxxxxxxxxxxxxxxxxx"

curl -s -X POST "${KUBESPHERE_API}/clusters/member-1/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/credentials" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "v1",
    "kind": "Secret",
    "metadata": {
      "name": "github-token",
      "namespace": "'${DEVOPS_PROJECT}'",
      "annotations": {
        "kubesphere.io/creator": "'${USERNAME}'",
        "credential.devops.kubesphere.io/type": "basic-auth"
      }
    },
    "stringData": {
      "username": "git",
      "password": "'${GITHUB_PAT}'"
    },
    "type": "credential.devops.kubesphere.io/basic-auth"
  }'

Step 3: Create Multi-Branch Pipeline with Credential

curl -s -X POST "${KUBESPHERE_API}/clusters/member-1/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "devops.kubesphere.io/v1alpha3",
    "kind": "Pipeline",
    "metadata": {
      "name": "echo-jenkinsfile-pipeline",
      "namespace": "'${DEVOPS_PROJECT}'",
      "annotations": {
        "kubesphere.io/creator": "'${USERNAME}'"
      }
    },
    "spec": {
      "type": "multi-branch-pipeline",
      "multi_branch_pipeline": {
        "name": "echo-jenkinsfile-pipeline",
        "description": "Multi-branch pipeline with GitHub auth",
        "source_type": "git",
        "git_source": {
          "url": "https://github.com/stoneshi-yunify/jenkinsfiles.git",
          "credential_id": "github-token",
          "discover_branches": true,
          "discover_tags": false
        },
        "script_path": "echo/Jenkinsfile"
      }
    }
  }'

Note: The kubesphere.io/creator annotation is required for all pipelines (both regular and multi-branch). Without it, the pipeline may not appear correctly in the KubeSphere UI.

Step 4: Verify Creation

curl -s "${KUBESPHERE_API}/clusters/member-1/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/echo-jenkinsfile-pipeline" \
  -H "Authorization: Bearer ${API_TOKEN}" | jq -r '{
    name: .metadata.name,
    syncStatus: .metadata.annotations."pipeline.devops.kubesphere.io/syncstatus",
    credential: .spec.multi_branch_pipeline.git_source.credential_id
  }'

Expected output:

{
  "name": "echo-jenkinsfile-pipeline",
  "syncStatus": "successful",
  "credential": "github-token"
}

---

Complete Multi-Branch Workflow Example

Scenario: Trigger build, monitor, retrieve logs, and download artifacts.

Step 1: Check Pipeline Exists

kubectl get pipelines -n demo-project
# NAME                   TYPE                    AGE
# demo-jenkinsfiles-go   multi-branch-pipeline   12d

Step 2: Trigger Build via Jenkins API

# Get token
TOKEN=$(kubectl -n kubesphere-devops-system get secret devops-jenkins -o jsonpath='{.data.jenkins-admin-token}' | base64 -d)

# Trigger main branch build
kubectl run curl-trigger --rm -i --restart=Never --image=curlimages/curl \
  -- "http://admin:${TOKEN}@devops-jenkins.kubesphere-devops-system:80/job/demo-project/job/demo-jenkinsfiles-go/job/main/build" \
  -X POST -w "\nHTTP Status: %{http_code}\n"

# Expected: HTTP Status: 201

Step 3: Monitor Build Status

# Check PipelineRun created
kubectl get pipelineruns -n demo-project --sort-by=.metadata.creationTimestamp | tail -3

# Example output:
# demo-jenkinsfiles-go-vf8p5       3     Succeeded   2m

# Get detailed status
# Or check via Jenkins API
kubectl run curl-status --rm -i --restart=Never --image=curlimages/curl \
  -- "http://admin:${TOKEN}@devops-jenkins.kubesphere-devops-system:80/job/demo-project/job/demo-jenkinsfiles-go/job/main/3/api/json" \
  | grep -E '"result"|"building"|"duration"'

Step 4: Retrieve Console Log

# Get full console log from build #3
kubectl run curl-log --rm -i --restart=Never --image=curlimages/curl \
  -- "http://admin:${TOKEN}@devops-jenkins.kubesphere-devops-system:80/job/demo-project/job/demo-jenkinsfiles-go/job/main/3/consoleText"

# Look for:
# [Pipeline] Start of Pipeline
# [Pipeline] { (Clone Repository)
# [Pipeline] { (Run Tests)
# === RUN   TestHello
# --- PASS: TestHello (0.00s)
# [Pipeline] End of Pipeline
# Finished: SUCCESS

Step 5: Download Build Artifacts

For binary artifacts, use pod-based download:

# Create download pod
kubectl run artifact-downloader --image=curlimages/curl -- sleep 300
kubectl wait --for=condition=Ready pod/artifact-downloader --timeout=60s

# Download artifact inside pod
TOKEN=$(kubectl -n kubesphere-devops-system get secret devops-jenkins -o jsonpath='{.data.jenkins-admin-token}' | base64 -d)
kubectl exec artifact-downloader -- sh -c \
  "curl -s -o /tmp/service 'http://admin:${TOKEN}@devops-jenkins.kubesphere-devops-system:80/job/demo-project/job/demo-jenkinsfiles-go/job/main/3/artifact/service'"

# Copy to local
kubectl cp artifact-downloader:/tmp/service /tmp/service

# Clean up
kubectl delete pod artifact-downloader --force

Step 6: Verify Downloaded Artifact

# Check file details
ls -lh /tmp/service
file /tmp/service

# Expected output:
# /tmp/service: ELF 64-bit LSB executable, x86-64...

# Make executable and test
chmod +x /tmp/service
/tmp/service --help

Build Summary Example:

Build #3 Summary:
- Status: SUCCESS
- Duration: ~54 seconds
- Test Results: 1/1 passed
- Artifact: service (8.0 MB Go binary)
- Stages: Checkout → Clone → Dependencies → Test → Build → Archive

Multi-Branch vs Regular Pipeline

Feature	Regular Pipeline	Multi-Branch Pipeline
Type	pipeline	multi-branch-pipeline
Jenkinsfile	Inline in CRD	From SCM (script_path)
Branches	Single	Auto-discovered
SCM Config	Manual checkout	Automatic
Trigger	PipelineRun/API	Branch indexing + webhooks
Get Parameters	From .spec.pipeline.jenkinsfile	From /branches/{branch} endpoint

Triggering Regular Pipeline Runs with Parameters (v1alpha3 API)

For regular pipelines, use this two-step procedure to handle parameters:

Step 1: Get Pipeline Definition and Extract Parameters

Unlike multi-branch pipelines, regular pipelines don't have a dedicated /parameters endpoint. Instead, retrieve the Pipeline CR and extract parameters from the Jenkinsfile:

# Get the pipeline definition
curl -s "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" | jq -r '.spec.pipeline.jenkinsfile'

Look for the parameters {} directive in the Jenkinsfile:

pipeline {
  agent any
  stages {
    stage('Example') {
      steps {
        echo "Hello ${params.PERSON}"
      }
    }
  }

  parameters {
    string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')
    booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value')
    choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something')
  }
}

Parameter Types in Jenkinsfile:

Directive	Type	Example
string()	Single-line text	string(name: 'VAR', defaultValue: 'default', description: '...')
text()	Multi-line text	text(name: 'VAR', defaultValue: '', description: '...')
booleanParam()	True/false	booleanParam(name: 'FLAG', defaultValue: true, description: '...')
choice()	Dropdown	choice(name: 'ENV', choices: ['dev', 'prod'], description: '...')
password()	Hidden value	password(name: 'SECRET', defaultValue: '', description: '...')

Step 2: Trigger Pipeline with Parameters

Use the /pipelineruns endpoint to create a new run with parameters:

# Trigger with parameters
curl -s -X POST "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}/pipelineruns" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": [
      {"name": "PERSON", "value": "John Doe"},
      {"name": "TOGGLE", "value": "true"},
      {"name": "CHOICE", "value": "Two"}
    ]
  }' | jq -r '.metadata.name'

Key Points:

• No branch query parameter needed (regular pipelines have no branches)
• Parameters are passed in the request body as a JSON array
• Boolean values must be strings: "true" or "false"
• The response includes the created PipelineRun resource with your parameters

Complete Example for Regular Pipelines

#!/bin/bash
export KUBESPHERE_API="http://kubesphere-apiserver:80"
export API_TOKEN="<tenant-oauth-token>"
export DEVOPS_PROJECT="devopstestc2nj7"
export PIPELINE_NAME="my-regular-pipeline"

# Step 1: Get pipeline definition and check for parameters
echo "Checking for parameters in pipeline..."
JENKINSFILE=$(curl -s "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}" \
  -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.spec.pipeline.jenkinsfile')

# Check if parameters exist
if echo "$JENKINSFILE" | grep -q "parameters {"; then
  echo "Pipeline has parameters defined. Extracting..."
  # In practice, parse the Jenkinsfile to extract parameter names and defaults
  echo "$JENKINSFILE" | grep -E "(string|booleanParam|choice|password|text)\(name:"
else
  echo "No parameters found. Triggering without parameters."
fi

# Step 2: Trigger with parameters
echo "Triggering pipeline with parameters..."
curl -s -X POST "${KUBESPHERE_API}/clusters/${CLUSTER}/kapis/devops.kubesphere.io/v1alpha3/namespaces/${DEVOPS_PROJECT}/pipelines/${PIPELINE_NAME}/pipelineruns" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": [
      {"name": "PERSON", "value": "Test User"},
      {"name": "TOGGLE", "value": "true"},
      {"name": "CHOICE", "value": "Two"},
      {"name": "BIOGRAPHY", "value": "Testing API"},
      {"name": "PASSWORD", "value": "secret123"}
    ]
  }' | jq -r '.metadata.name'

Triggering Multi-Branch Pipeline Runs (v1alpha3 API)

⚠️ API Version Notice: The /kapis/devops.kubesphere.io/v1alpha2/ APIs are deprecated. Always prefer v1alpha3 APIs when available.

For multi-branch pipelines, use this three-step procedure with v1alpha3 APIs: