Добавление oVirt CSI в платформе установленной методом UPI

В руководстве описан процесс добавления oVirt CSI в платформе, но только на Worker узле, который располагается в системе виртуализации zVirt.

1. Предварительные условия

Nova Container Platform установлена методом UPI в минимальной конфигурации с двумя Worker узлами. Один из узлов является виртуальной машиной в системе виртуализации zVirt, а второй - физическим сервером.

2. Описание процесса добавления oVirt CSI

Настройка zVirt.
Настройка StarVault.
Настройка Nova Container Platform.
Проверка.

3. Настройка zVirt

Настройте пользователя и получите данные для настройки интеграции с zVirt согласно статье Интеграция с zVirt.

4. Настройка StarVault

Зайдите в веб-консоль StarVault с использованием root токена.
На вкладке Secrets перейдите в nova-secrets credentials и создайте секрет с именем ovirt-csi.
Добавьте следующие ключи-значения в новый секрет:
- ovirt_ca_bundle — цепочка корневых TLS-сертификатов для подключения к интерфейсу zVirt API.
  
  Пример: "LS0k1JSURxekNDQXBPZ0F3SUJBZ0lWUS0tLQo=…"
- ovirt_password — пароль от учётной записи пользователя zVirt
- ovirt_url — URL-адрес API среды виртуализации zVirt.
  
  Пример: "https://zvirt.nova.internal/ovirt-engine/api"
- ovirt_username — имя пользователя zVirt
Перейдите на вкладку Policies и создайте новую политику с именем nova-system-ovirt-csi и следующей настройкой:
```
path "nova-secrets/data/credentials/ovirt-csi" { capabilities = ["read"] }
```
Перейдите на вкладку Access в раздел Auth Methods nova-kubernetes и создайте новую роль со следующими параметрами:
- Name: nova-system-ovirt-csi
- Alias name source: serviceaccount_uid
- Bound service account names:
  - ovirt-csi-driver-controller-sa
  - ovirt-csi-driver-node-sa
- Bound service account namespaces: nova-csi-drivers
- Раскройте блок Tokens
  - Включите опцию Generated Token’s Maximum TTL и поставьте значение в 1 час
  - Включите опцию Do Not Attach 'default' Policy To Generated Tokens
  - В поле Generated Token’s Policies добавьте имя политики созданной ранее
  - Включите опцию Generated Token’s Initial TTL и поставьте значение в 1 час
  - В поле Generated Token’s Type поставьте значение default

5. Настройка Nova Container Platform

В веб-консоли Nova Container Platform перейдите на вкладку Узлы кластера Nodes.
Повторите следующие действия для всех Worker узлов на zVirt:
1. Нажмите на выбранный узел
2. Перейдите на вкладку YAML и добавьте метку:
  labels: ovirt: 'true'

Создайте сервисные аккаунты и роли для них.

Манифесты

kind: ServiceAccount
apiVersion: v1
metadata:
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: serviceaccount
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: ovirt-csi-driver-node-sa
  namespace: nova-csi-drivers
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: serviceaccount
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ovirt-csi-driver-controller-sa
subjects:
  - kind: ServiceAccount
    name: ovirt-csi-driver-controller-sa
    namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: 'system:auth-delegator'
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ovirt-csi-driver-node-sa
subjects:
  - kind: ServiceAccount
    name: ovirt-csi-driver-node-sa
    namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: 'system:auth-delegator'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: csi-ovirtplugin:secrets-webhook-mutation-helper
  namespace: nova-csi-drivers
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - delete
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: csi-ovirtplugin:secrets-webhook-mutation-helper
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: csi-ovirtplugin:secrets-webhook-mutation-helper
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: csi-ovirtplugin-node:secrets-webhook-mutation-helper
  namespace: nova-csi-drivers
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - delete
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: csi-ovirtplugin-node:secrets-webhook-mutation-helper
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: csi-ovirtplugin-node:secrets-webhook-mutation-helper
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-node-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: attacher-cluster-role
  name: csi-ovirtplugin-attacher-cluster-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-attacher-runner
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: health-monitor-controller-cluster-role
  name: csi-ovirtplugin-health-monitor-controller-cluster-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-health-monitor-controller-runner
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: provisioner-cluster-role
  name: csi-ovirtplugin-provisioner-cluster-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-provisioner-runner
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: resizer-cluster-role
  name: csi-ovirtplugin-resizer-cluster-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-resizer-runner
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: attacher-role
  name: csi-ovirtplugin-attacher-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: external-attacher-cfg
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: health-monitor-controller-role
  name: csi-ovirtplugin-health-monitor-controller-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: external-health-monitor-controller-cfg
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: provisioner-role
  name: csi-ovirtplugin-provisioner-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: external-provisioner-cfg
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: resizer-role
  name: csi-ovirtplugin-resizer-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: external-resizer-cfg
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-controller-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: health-monitor-controller-cluster-role
  name: csi-ovirtplugin-node-health-monitor-controller-cluster-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-health-monitor-controller-runner
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-node-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: health-monitor-controller-role
  name: csi-ovirtplugin-node-health-monitor-controller-role
  namespace: nova-csi-drivers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: external-health-monitor-controller-cfg
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-node-sa
  namespace: nova-csi-drivers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ovirt-node-privileged-binding
roleRef:
  kind: ClusterRole
  name: ovirt-privileged-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: ovirt-csi-driver-node-sa
  namespace: nova-csi-drivers
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-attacher-cfg
  namespace: nova-csi-drivers
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-health-monitor-controller-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "patch"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-health-monitor-controller-cfg
  namespace: nova-csi-drivers
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-provisioner-cfg
  namespace: nova-csi-drivers
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csistoragecapacities"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: ["apps"]
  resources: ["replicasets"]
  verbs: ["get"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-resizer-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims/status"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: external-resizer-cfg
  namespace: nova-csi-drivers
rules:
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "watch", "list", "delete", "update", "create"]
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ovirt-privileged-role
rules:
  - apiGroups: ["security.openshift.io"]
    resourceNames: ["privileged"]
    resources: ["securitycontextconstraints"]
    verbs: ["use"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]

Установите контроллер и агенты.

Манифесты

kind: Deployment
apiVersion: apps/v1
metadata:
  name: csi-ovirtplugin
  namespace: nova-csi-drivers
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: plugin
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: ovirt.csi.k8s.io
      app.kubernetes.io/part-of: csi-driver-ovirt
      app.kubernetes.io/name: ovirt-csi-driver
      app.kubernetes.io/component: plugin
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: ovirt.csi.k8s.io
        app.kubernetes.io/part-of: csi-driver-ovirt
        app.kubernetes.io/name: ovirt-csi-driver
        app.kubernetes.io/component: plugin
      annotations:
        vault.security.banzaicloud.io/vault-role: "nova-system-ovirt-csi"
        vault.security.banzaicloud.io/vault-path: "nova-kubernetes"
        vault.security.banzaicloud.io/vault-trust-manager-tls-bundle: "trusted-ca-bundle"
        vault.security.banzaicloud.io/run-as-non-root: "true"
    spec:
      priorityClassName: system-cluster-critical
      nodeSelector:
        node-role.kubernetes.io/infra: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
      serviceAccountName: ovirt-csi-driver-controller-sa
      initContainers:
        - name: secrets-webhook-mutation-helper
          image: "hub.nova-platform.io/registry/nova/secrets-webhook-mutation-helper:v1.0.0"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
          env:
          - name: CA_CERT_PATH
            value: /etc/ssl/certs/ca-certificates.crt
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: trusted-ca-bundle
              readOnly: true
        - name: prepare-ovirt-config
          env:
            - name: OVIRT_URL
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_url
            - name: OVIRT_USERNAME
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_username
            - name: OVIRT_PASSWORD
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_password
            - name: OVIRT_CAFILE
              value: /tmp/config/ovirt-engine-ca.pem
            - name: OVIRT_INSECURE
              value: ""
            - name: OVIRT_CA_BUNDLE
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_ca_bundle
          image: hub.nova-platform.io/registry/nova/ovirt-csi-driver:4.16-nova
          imagePullPolicy: IfNotPresent
          command:
            - /bin/sh
            - -c
            - |
              #!/bin/sh
              cat << EOF > /tmp/config/ovirt-config.yaml
              ovirt_url: $OVIRT_URL
              ovirt_username: "$OVIRT_USERNAME"
              ovirt_password: "$OVIRT_PASSWORD"
              # set a valid path only if ca bundle has content
              ovirt_cafile: ${OVIRT_CA_BUNDLE:+$OVIRT_CAFILE}
              ovirt_insecure: $OVIRT_INSECURE
              EOF
              if [[ -n "$OVIRT_CA_BUNDLE" ]]; then echo "$OVIRT_CA_BUNDLE" | base64 -d > $OVIRT_CAFILE ; fi
          volumeMounts:
            - name: config
              mountPath: /tmp/config
          resources:
            requests:
              memory: 128Mi
              cpu: 160m
      containers:
        - name: ovirt
          image: hub.nova-platform.io/registry/nova/ovirt-csi-driver:4.16-nova
          imagePullPolicy: IfNotPresent
          args:
            - --endpoint=$(CSI_ENDPOINT)
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/ovirt-plugin-csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: OVIRT_CONFIG
              value: /tmp/config/ovirt-config.yaml
          securityContext:
            privileged: true
          ports:
          - containerPort: 9898
            name: healthz
            protocol: TCP
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            timeoutSeconds: 3
            periodSeconds: 2
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
            - name: socket-dir
              mountPath: /csi/
            - name: config
              mountPath: /tmp/config
        - name: liveness-probe
          image: hub.nova-platform.io/registry/sig-storage/livenessprobe:v2.7.0
          args:
            - --csi-address=/csi/ovirt-plugin-csi.sock
            - --health-port=9898
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
          - name: socket-dir
            mountPath: /csi/
        - name: csi-attacher
          image: hub.nova-platform.io/registry/sig-storage/csi-attacher:v3.5.0
          args:
            - --csi-address=/csi/ovirt-plugin-csi.sock
            - --v=5
          securityContext:
            privileged: true
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
          - name: socket-dir
            mountPath: /csi/
        - name: csi-provisioner
          image: hub.nova-platform.io/registry/sig-storage/csi-provisioner:v3.2.0
          args:
            - --csi-address=/csi/ovirt-plugin-csi.sock
            - --feature-gates=Topology=true
            - -v=5
          securityContext:
            privileged: true
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
            - name: socket-dir
              mountPath: /csi/
        - name: csi-resizer
          image: hub.nova-platform.io/registry/sig-storage/csi-resizer:v1.5.0
          args:
            - -csi-address=/csi/ovirt-plugin-csi.sock
            - -v=5
          securityContext:
            privileged: true
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
            - name: socket-dir
              mountPath: /csi/
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi.ovirt.org
            type: DirectoryOrCreate
        - name: config
          emptyDir:
            medium: Memory
            sizeLimit: 10Mi
        - name: trusted-ca-bundle
          configMap:
            name: trusted-ca-bundle
            items:
            - key: ca-certificates.pem
              path: ca-certificates.crt
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-ovirtplugin-node
  namespace: nova-csi-drivers
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: plugin
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: ovirt.csi.k8s.io
      app.kubernetes.io/part-of: csi-driver-ovirt
      app.kubernetes.io/name: ovirt-csi-driver-node
      app.kubernetes.io/component: plugin
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: ovirt.csi.k8s.io
        app.kubernetes.io/part-of: csi-driver-ovirt
        app.kubernetes.io/name: ovirt-csi-driver-node
        app.kubernetes.io/component: plugin
      annotations:
        vault.security.banzaicloud.io/vault-role: "nova-system-ovirt-csi"
        vault.security.banzaicloud.io/vault-path: "nova-kubernetes"
        vault.security.banzaicloud.io/vault-trust-manager-tls-bundle: "trusted-ca-bundle"
        vault.security.banzaicloud.io/run-as-non-root: "true"
    spec:
      nodeSelector:
        ovirt: "true"
      priorityClassName: system-node-critical
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
      serviceAccountName: ovirt-csi-driver-node-sa
      initContainers:
        - name: secrets-webhook-mutation-helper
          image: "hub.nova-platform.io/registry/nova/secrets-webhook-mutation-helper:v1.0.0"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            runAsNonRoot: true
          env:
          - name: CA_CERT_PATH
            value: /etc/ssl/certs/ca-certificates.crt
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: trusted-ca-bundle
              readOnly: true
        - name: prepare-ovirt-config
          env:
            - name: OVIRT_URL
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_url
            - name: OVIRT_USERNAME
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_username
            - name: OVIRT_PASSWORD
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_password
            - name: OVIRT_CAFILE
              value: /tmp/config/ovirt-engine-ca.pem
            - name: OVIRT_INSECURE
              value: ""
            - name: OVIRT_CA_BUNDLE
              value: vault:nova-secrets/data/credentials/ovirt-csi#ovirt_ca_bundle
          image: hub.nova-platform.io/registry/nova/ovirt-csi-driver:4.16-nova
          imagePullPolicy: IfNotPresent
          command:
            - /bin/sh
            - -c
            - |
              #!/bin/sh
              cat << EOF > /tmp/config/ovirt-config.yaml
              ovirt_url: $OVIRT_URL
              ovirt_username: "$OVIRT_USERNAME"
              ovirt_password: "$OVIRT_PASSWORD"
              # set a valid path only if ca bundle has content
              ovirt_cafile: ${OVIRT_CA_BUNDLE:+$OVIRT_CAFILE}
              ovirt_insecure: $OVIRT_INSECURE
              EOF
              if [[ -n "$OVIRT_CA_BUNDLE" ]]; then echo "$OVIRT_CA_BUNDLE" | base64 -d > $OVIRT_CAFILE ; fi
          volumeMounts:
            - name: config
              mountPath: /tmp/config
          resources:
            requests:
              memory: 50Mi
              cpu: 10m
      containers:
        - name: ovirt
          image: hub.nova-platform.io/registry/nova/ovirt-csi-driver:4.16-nova
          imagePullPolicy: IfNotPresent
          args:
            - --endpoint=$(CSI_ENDPOINT)
            - --node-name=$(KUBE_NODE_NAME)
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: spec.nodeName
            - name: OVIRT_CONFIG
              value: /tmp/config/ovirt-config.yaml
          securityContext:
            privileged: true
          ports:
          - containerPort: 9898
            name: healthz
            protocol: TCP
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            timeoutSeconds: 3
            periodSeconds: 2
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              mountPropagation: Bidirectional
            - name: dev-dir
              mountPath: /dev
            - name: config
              mountPath: /tmp/config
        - name: liveness-probe
          image: hub.nova-platform.io/registry/sig-storage/livenessprobe:v2.7.0
          args:
            - --csi-address=/csi/csi.sock
            - --health-port=9898
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
          - name: socket-dir
            mountPath: /csi
        - name: csi-driver-registrar
          image: hub.nova-platform.io/registry/sig-storage/csi-node-driver-registrar:v2.5.0
          args:
            - "--v=5"
            - "--csi-address=/csi/csi.sock"
            - "--kubelet-registration-path=/var/lib/kubelet/plugins/csi.ovirt.org/csi.sock"
          livenessProbe:
            exec:
              command:
              - /csi-node-driver-registrar
              - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.ovirt.org/csi.sock
              - --mode=kubelet-registration-probe
            initialDelaySeconds: 30
            timeoutSeconds: 15
          resources:
            requests:
              memory: 64Mi
              cpu: 80m
          volumeMounts:
          - name: socket-dir
            mountPath: /csi
          - name: registration-dir
            mountPath: /registration
      volumes:
        - name: trusted-ca-bundle
          configMap:
            name: trusted-ca-bundle
            items:
            - key: ca-certificates.pem
              path: ca-certificates.crt
        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi.ovirt.org
            type: DirectoryOrCreate
        - name: kubelet-dir
          hostPath:
            path: /var/lib/kubelet
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry
            type: Directory
        - name: dev-dir
          hostPath:
            path: /dev
            type: Directory
        - name: config
          emptyDir:
            sizeLimit: 10Mi
            medium: Memory
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: csi.ovirt.org
  labels:
    app.kubernetes.io/instance: ovirt.csi.k8s.io
    app.kubernetes.io/part-of: csi-driver-ovirt
    app.kubernetes.io/name: ovirt-csi-driver
    app.kubernetes.io/component: csi-driver
spec:
  attachRequired: true
  podInfoOnMount: false
  fsGroupPolicy: File
  requiresRepublish: false
  storageCapacity: false
  volumeLifecycleModes:
    - Persistent

Если платформа была установлена используя Universe, то замените в манифестах hub.nova-platform.io/registry на hub.universe.nova.internal/nova-universe.
Не забудьте заменить universe.nova.internal на FQDN вашего Universe сервера. Также вы можете взять это значение из поля hubRegistryURL в файле nova-deployment-conf.yaml.

Установите манифест для StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ovirt-csi-sc
  namespace: nova-csi-drivers
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.ovirt.org
reclaimPolicy: "Delete"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  storageDomainName: "zvirtCsiStorageDomainName" (1)
  thinProvisioning: "true"
  fsType: "xfs"

Измените на имя вашего домена хранения в zVirt.

6. Проверка

Создайте новое пространство имён, PVC и под.

Пример

kind: Namespace
apiVersion: v1
metadata:
  name: test
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: httpd-pvc
  namespace: test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: ovirt-csi-sc
---
apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
  namespace: test
spec:
  nodeSelector:
    ovirt: "true"
  containers:
  - name: httpd
    image: httpd:latest
    volumeMounts:
    - mountPath: /usr/local/apache2/htdocs
      name: httpd-storage
  volumes:
  - name: httpd-storage
    persistentVolumeClaim:
      claimName: httpd-pvc

Зайдите в консоль пода и создайте файл в директории /usr/local/apache2/htdocs
Удалите под и создайте его заново.
Убедитесь, что файл существует в директории /usr/local/apache2/htdocs
Перейдите на вкладку Узлы кластера Nodes.
Повторите следующие действия для любого Worker узла, который является физическим сервером:
1. Нажмите на выбранный узел.
2. Перейдите на вкладку YAML и добавьте метку:
  labels: ovirt: 'false'

Удалите под и создайте новый со следующим манифестом:

apiVersion: v1
kind: Pod
metadata:
  name: httpd-pod
  namespace: test
spec:
  nodeSelector:
    ovirt: "false"
  containers:
  - name: httpd
    image: httpd:latest
    volumeMounts:
    - mountPath: /usr/local/apache2/htdocs
      name: httpd-storage
  volumes:
  - name: httpd-storage
    persistentVolumeClaim:
      claimName: httpd-pvc

Зайдите в под и перейдите на вкладку События. Убедитесь, что под не может быть создан из-за того, что на узле не установлен CSI.
Зайдите в веб-консоль zVirt и найдите созданный диск. Его имя будет начинаться c pvc.
В веб-консоли платформы удалите под и PVC.
Убедитесь, что диск удалился из zVirt.
Проверка завершена.