WIP: enabling udn on ovn-kubernetes spike

[eslutsky]

enabling user defined network for multi-cluster communication

starting point : Microshift 4.22 build from PR sources, and Multus RPM installed

OVNK patching:

• custom build ovnk image src

• Deployment

  1. applying CRD manifests

      oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_clusteruserdefinednetworks.yaml.j2
      oc apply -f https://raw.githubusercontent.com/ovn-kubernetes/ovn-kubernetes/refs/heads/master/dist/templates/k8s.ovn.org_userdefinednetworks.yaml.j2
  

  2. update RBAC rules

     	  oc patch clusterrole openshift-ovn-kubernetes-node --type='json' -p='[
        {
          "op": "add", 
          "path": "/rules/-", 
          "value": {
            "apiGroups":["k8s.ovn.org"],
            "resources":["userdefinednetworks","clusteruserdefinednetworks"],
            "verbs":["get","list","watch","patch","update"]
          }
        }
      ]'

  3. create NS

  cat <<EOF | oc apply -f -
  apiVersion: v1
  kind: Namespace
  metadata:
    name: udnspike
    labels:
      k8s.ovn.org/primary-user-defined-network: ""
  EOF
  

  4. create network

  cat <<EOF | oc apply -f -
  apiVersion: k8s.ovn.org/v1
  kind: ClusterUserDefinedNetwork
  metadata:
    name: netext1
  spec:
    namespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values: ["udnspike"]
    network:
      topology: Layer3
      layer3:
        role: Primary
        subnets:
          - cidr: 10.100.0.0/16
            hostSubnet: 24
  EOF

  5. Create pod in the NS with Primary network

  cat <<EOF | oc apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: network-debug-pod
    namespace: udnspike
  
  spec:
    securityContext:
      runAsNonRoot: true
      seccompProfile:
        type: RuntimeDefault
    containers:
    - name: debug-container
      # Using netshoot instead of the restricted redhat image
      image: docker.io/nicolaka/netshoot:latest
      command: ["/bin/sh", "-c", "sleep infinity"]
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        runAsUser: 10001    
  EOF
  

[root@i-08482c17698046dd0 ~]# oc get network-attachment-definitions -oyaml -A
apiVersion: v1
items:
- apiVersion: k8s.cni.cncf.io/v1
  kind: NetworkAttachmentDefinition
  metadata:
    annotations:
      k8s.ovn.org/network-id: "1"
      k8s.ovn.org/network-name: cluster_udn_netext1
      k8s.ovn.org/tunnel-keys: '[]'
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"k8s.ovn.org/v1","kind":"ClusterUserDefinedNetwork","metadata":{"annotations":{},"name":"netext1"},"spec":{"namespaceSelector":{"matchExpressions":[{"key":"kubernetes.io/metadata.name","operator":"In","values":["udnspike"]}]},"network":{"layer3":{"role":"Primary","subnets":[{"cidr":"10.100.0.0/16","hostSubnet":24}]},"topology":"Layer3"}}}
    creationTimestamp: "2026-01-20T13:10:38Z"
    finalizers:
    - k8s.ovn.org/user-defined-network-protection
    generation: 1
    labels:
      k8s.ovn.org/user-defined-network: ""
    name: netext1
    namespace: udnspike
    ownerReferences:
    - apiVersion: k8s.ovn.org/v1
      blockOwnerDeletion: true
      controller: true
      kind: ClusterUserDefinedNetwork
      name: netext1
      uid: a87bbaab-5743-4b63-bb33-8c99ff88ae20
    resourceVersion: "909"
    uid: 0dd56527-daab-4f2a-9f3c-d9102f44cfa9
  spec:
    config: '{"cniVersion":"1.0.0","joinSubnet":"100.65.0.0/16,fd99::/64","name":"cluster_udn_netext1","netAttachDefName":"udnspike/netext1","role":"primary","subnets":"10.100.0.0/16/24","topology":"layer3","type":"ovn-k8s-cni-overlay"}'
kind: List
metadata:
  resourceVersion: ""

inside the test pod:

2 IP Addresses while d.g is the UDN (10.100.0.1):
network-debug-pod:/root$ ip -br addr show
lo               UNKNOWN        127.0.0.1/8 ::1/128 
eth0@if13        UP             10.42.0.7/24 fe80::858:aff:fe2a:7/64 
ovn-udn1@if14    UP             10.100.0.3/24 fe80::858:aff:fe64:3/64 


network-debug-pod:/root$ ip route
default via 10.100.0.1 dev ovn-udn1 
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.7 
10.42.0.0/16 via 10.42.0.1 dev eth0 
10.43.0.0/16 via 10.100.0.1 dev ovn-udn1 
10.100.0.0/24 dev ovn-udn1 proto kernel scope link src 10.100.0.3 
10.100.0.0/16 via 10.100.0.1 dev ovn-udn1 
100.64.0.0/16 via 10.42.0.1 dev eth0 
100.65.0.0/16 via 10.100.0.1 dev ovn-udn1 
network-debug-pod:/root$ 

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eslutsky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [eslutsky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@eslutsky: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/ocp-full-conformance-rhel-eus	f75480f	link	true	/test ocp-full-conformance-rhel-eus
ci/prow/ocp-full-conformance-serial-rhel-eus	f75480f	link	true	/test ocp-full-conformance-serial-rhel-eus

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.