背景
翻了一下笔记,2023年底接触了kubevirt 这个k8s下的虚拟机项目,当时也只是简单的部署以及使用现有的几个镜像,2024年也断断续续使用过几次,直到今年2025年8月再次接触,制作了自己的镜像才算基本搞通了这个kubevirt 的使用,时间跨度有点大,这里记录一下。
准备工作
kvm 虚拟化
物理机ubuntu 系统
1 |
apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils -y |
虚拟机 ubuntu 系统
1 |
apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virt-manager -y |
虚拟机 rockylinux 系统
1 |
yum install -y qemu-kvm libvirt virt-install bridge-utils |
嵌套虚拟化
如果测试机器本身是虚拟机,则还有一个工作,开启嵌套虚拟化。
修改 grub 设置
安装了前面的几个kvm 相关包之后,还需要修改 grub 设置。
1 2 |
vi /etc/default/grub 加上最后1部分 GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rl/root intel_iommu=on" |
安装后,使用如下 virt-host-validate qemu 命令检查虚拟化是否正常
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
virt-host-validate qemu QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for device assignment IOMMU support : PASS QEMU: Checking if IOMMU is enabled by kernel : PASS QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support) |
正式部署 kubevirt
参考 https://kubevirt.io/quickstart_cloud/,部署过程基本参考前面的官网链接,如下
yaml部署kubevirt
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
KubeVirt can be installed using the KubeVirt operator, which manages the lifecycle of all the KubeVirt core components. 查看kubevirt 当前最新版本 export VERSION=$(curl -s https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt) echo $VERSION v1.6.0 # 部署 kubevirt-operator kubectl create -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml" # 输出如下 namespace/kubevirt created customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io created priorityclass.scheduling.k8s.io/kubevirt-cluster-critical created clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator created serviceaccount/kubevirt-operator created role.rbac.authorization.k8s.io/kubevirt-operator created rolebinding.rbac.authorization.k8s.io/kubevirt-operator-rolebinding created clusterrole.rbac.authorization.k8s.io/kubevirt-operator created clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator created deployment.apps/virt-operator created # 部署自定义资源 # Again use kubectl to deploy the KubeVirt custom resource definitions: kubectl create -f "https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml" #检查组件 Verify components #By default KubeVirt will deploy 7 pods, 3 services, 1 daemonset, 3 deployment apps, 3 replica sets. kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}" Deploying( 再等等) Check the components: kubectl get all -n kubevirt Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2. NAME READY STATUS RESTARTS AGE pod/virt-api-67778d48b6-7kjhm 0/1 ContainerCreating 0 19s pod/virt-api-67778d48b6-z8lrq 0/1 ContainerCreating 0 20s pod/virt-operator-b87fbb945-n7287 1/1 Running 0 3m35s pod/virt-operator-b87fbb945-xl7pg 1/1 Running 0 3m34s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubevirt-operator-webhook ClusterIP 10.233.48.98 <none> 443/TCP 26s service/kubevirt-prometheus-metrics ClusterIP None <none> 443/TCP 26s service/virt-api ClusterIP 10.233.49.2 <none> 443/TCP 26s service/virt-exportproxy ClusterIP 10.233.29.96 <none> 443/TCP 26s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/virt-api 0/2 2 0 21s deployment.apps/virt-operator 2/2 2 2 3m36s NAME DESIRED CURRENT READY AGE replicaset.apps/virt-api-67778d48b6 2 2 0 21s replicaset.apps/virt-operator-b87fbb945 2 2 2 3m36s NAME AGE PHASE kubevirt.kubevirt.io/kubevirt 92s Deploying 9分钟后,部署完成,主要是拉镜像通过了 kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}" Deployed 这样检查也过了 kubectl get all -n kubevirt Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2. NAME READY STATUS RESTARTS AGE pod/virt-api-67778d48b6-7kjhm 1/1 Running 0 5m59s pod/virt-api-67778d48b6-z8lrq 1/1 Running 0 6m pod/virt-controller-8c6b9f8f4-4xhgd 1/1 Running 0 4m56s pod/virt-controller-8c6b9f8f4-jcqzd 1/1 Running 0 4m56s pod/virt-handler-642df 1/1 Running 0 4m55s pod/virt-handler-crjvl 1/1 Running 0 4m55s pod/virt-handler-tbmv7 1/1 Running 0 4m55s pod/virt-operator-b87fbb945-n7287 1/1 Running 0 9m15s pod/virt-operator-b87fbb945-xl7pg 1/1 Running 0 9m14s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubevirt-operator-webhook ClusterIP 10.233.48.98 <none> 443/TCP 6m5s service/kubevirt-prometheus-metrics ClusterIP None <none> 443/TCP 6m5s service/virt-api ClusterIP 10.233.49.2 <none> 443/TCP 6m5s service/virt-exportproxy ClusterIP 10.233.29.96 <none> 443/TCP 6m5s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/virt-handler 3 3 3 3 3 kubernetes.io/os=linux 4m56s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/virt-api 2/2 2 2 6m deployment.apps/virt-controller 2/2 2 2 4m56s deployment.apps/virt-operator 2/2 2 2 9m15s NAME DESIRED CURRENT READY AGE replicaset.apps/virt-api-67778d48b6 2 2 2 6m replicaset.apps/virt-controller-8c6b9f8f4 2 2 2 4m56s replicaset.apps/virt-operator-b87fbb945 2 2 2 9m15s NAME AGE PHASE kubevirt.kubevirt.io/kubevirt 7m11s Deployed |
部署 Virtctl 工具
1 2 3 4 5 6 |
VERSION=$(kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.observedKubeVirtVersion}") ARCH=$(uname -s | tr A-Z a-z)-$(uname -m | sed 's/x86_64/amd64/') || windows-amd64.exe echo ${ARCH} curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH} chmod +x virtctl sudo install virtctl /usr/local/bin |
部署 kubevirt cdi
1 2 3 |
export VERSION=$(basename $(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)) kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml |
检查cdi 输出
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
kubectl -n cdi get all Warning: kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2. NAME READY STATUS RESTARTS AGE pod/cdi-operator-ccb895984-w4b6n 1/1 Running 0 3m8s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cdi-operator 1/1 1 1 3m8s NAME DESIRED CURRENT READY AGE replicaset.apps/cdi-operator-ccb895984 1 1 1 3m8s [root@gm-10-29-221-9 ~]# kubectl get cdi cdi -n cdi NAME AGE PHASE cdi 7m15s Deployed [root@gm-10-29-221-9 ~]# kubectl get pods -n cdi NAME READY STATUS RESTARTS AGE cdi-apiserver-6c76687b66-6l7d2 1/1 Running 1 (5m48s ago) 7m11s cdi-deployment-5f6ff949d7-mlrth 1/1 Running 0 7m9s cdi-operator-ccb895984-w4b6n 1/1 Running 0 10m cdi-uploadproxy-b499c7956-lsh5r 1/1 Running 0 7m7s |
检查 集群 确认有 sc
1 2 3 4 |
kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE hwameistor-storage-lvm-hdd (default) lvm.hwameistor.io Retain WaitForFirstConsumer true 100d local-path rancher.io/local-path Delete WaitForFirstConsumer false 100d |
测试fedora 镜像, 补上 sc 等字段
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
cat <<EOF > dv_fedora.yml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: "fedora" spec: storage: resources: requests: storage: 5Gi storageClassName: hwameistor-storage-lvm-hdd accessModes: - ReadWriteOnce volumeMode: Filesystem source: http: url: "https://download.fedoraproject.org/pub/fedora/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-AmazonEC2.x86_64-40-1.14.raw.xz" EOF kubectl create -f dv_fedora.yml |
启动一个vm1 虚拟机
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
wget https://kubevirt.io/labs/manifests/vm1_pvc.yml cat vm1_pvc.yml 修改前 apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: creationTimestamp: 2018-07-04T15:03:08Z generation: 1 labels: kubevirt.io/os: linux name: vm1 spec: runStrategy: Always template: metadata: creationTimestamp: null labels: kubevirt.io/domain: vm1 spec: domain: cpu: cores: 2 devices: disks: - disk: bus: virtio name: disk0 - cdrom: bus: sata readonly: true name: cloudinitdisk machine: type: q35 resources: requests: memory: 1024M volumes: - name: disk0 persistentVolumeClaim: claimName: fedora - cloudInitNoCloud: userData: | #cloud-config hostname: vm1 ssh_pwauth: True disable_root: false ssh_authorized_keys: - ssh-rsa YOUR_SSH_PUB_KEY_HERE name: cloudinitdisk # Generate a password-less SSH key using the default location. ssh-keygen PUBKEY=`cat ~/.ssh/id_rsa.pub` sed -i "s%ssh-rsa.*%$PUBKEY%" vm1_pvc.yml kubectl create -f vm1_pvc.yml |
virtctl console vm1 进入 这个虚拟机实例
1 |
virtctl expose vmi vm1 --name=vm1-ssh --port=20222 --target-port=22 --type=NodePort |
顺利完成第一轮测试,后面自制rocky linux 8.10镜像 以及win10-ltsc-2021-6216版本
定制镜像
这次自己打包了2个镜像,rockylinux 和 windows ltsc 2021.
rockylinux 镜像比较简单,rockylinux.org 官方就有 cloud 镜像,下载,重新打包即可。
1 2 3 4 5 6 7 8 9 |
wget -c https://dl.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud-Base.latest.x86_64.qcow2 vi Dockerfile FROM scratch ADD --chown=107:107 Rocky-8-GenericCloud-Base.latest.x86_64.qcow2 /disk/ docker build -t 10.29.221.9/public/rockylinux:v810 . docker push 10.29.221.9/public/rockylinux:v810 |
rocky810的镜像构建
首先rockylinux官网下载 Rocky-8-GenericCloud-Base.latest.x86_64.qcow2 云镜像,然后本地构建一下
1 2 3 4 5 6 7 |
wget -c https://dl.rockylinux.org/pub/rocky/8/images/aarch64/Rocky-8-GenericCloud-Base.latest.aarch64.qcow2 vi Dockerfile FROM scratch ADD --chown=107:107 Rocky-8-GenericCloud-Base.latest.x86_64.qcow2 /disk/ nerdctl build --platform=amd64 -t 10.29.221.9/public/rockylinux:v810 . |
处理 rockylinux8 的 datavolume
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
kubectl get datavolumes.cdi.kubevirt.io rocky810-rootdisk -oyaml apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: labels: kubevirt.io/created-by: 2782666c-6914-4395-8362-0067661a02b0 name: rocky810-rootdisk namespace: default spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: hwameistor-storage-lvm-hdd source: registry: url: docker://10.29.221.9/public/rockylinux:v810 |
rocky810 vm 配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
kubectl get vm rocky810 -oyaml apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: annotations: kubevirt.io/latest-observed-api-version: v1 kubevirt.io/storage-observed-api-version: v1 virtnest.io/alias-name: "" virtnest.io/image-secret: "" virtnest.io/image-source: docker virtnest.io/os-image: 10.29.221.9/public/rockylinux:v810 labels: virtnest.io/os-family: rocky virtnest.io/os-version: "810" name: rocky810 namespace: default spec: dataVolumeTemplates: - metadata: creationTimestamp: null name: rocky810-rootdisk namespace: default spec: pvc: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi storageClassName: hwameistor-storage-lvm-hdd source: registry: url: docker://10.29.221.9/public/rockylinux:v810 runStrategy: Always template: metadata: creationTimestamp: null spec: architecture: amd64 domain: cpu: cores: 1 sockets: 1 threads: 1 devices: disks: - bootOrder: 1 disk: bus: virtio name: rootdisk - disk: bus: virtio name: cloudinitdisk interfaces: - masquerade: {} name: default machine: type: q35 memory: guest: 2Gi resources: {} networks: - name: default pod: {} volumes: - dataVolume: name: rocky810-rootdisk name: rootdisk - cloudInitNoCloud: userDataBase64: I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogZmFsc2UKY2hwYXNzd2Q6IHsibGlzdCI6ICJyb290OkRhb2Nsb3VkLjIwMjMiLCBleHBpcmU6IEZhbHNlfQoKd3JpdGVfZmlsZXM6CiAgLSBwYXRoOiAvZXRjL3N5c3RlbWQvc3lzdGVtL2RoY2xpZW50LW9uLWJvb3Quc2VydmljZQogICAgcGVybWlzc2lvbnM6ICIwNjQ0IgogICAgY29udGVudDogfAogICAgICBbVW5pdF0KICAgICAgRGVzY3JpcHRpb249UnVuIGRoY2xpZW50IG9uIGJvb3QKICAgICAgQWZ0ZXI9bmV0d29yay50YXJnZXQKCiAgICAgIFtTZXJ2aWNlXQogICAgICBUeXBlPW9uZXNob3QKICAgICAgRXhlY1N0YXJ0PS9zYmluL2RoY2xpZW50CiAgICAgIFJlbWFpbkFmdGVyRXhpdD10cnVlCgogICAgICBbSW5zdGFsbF0KICAgICAgV2FudGVkQnk9bXVsdGktdXNlci50YXJnZXQKcnVuY21kOgogIC0gc3lzdGVtY3RsIGRhZW1vbi1yZWxvYWQKICAtIHN5c3RlbWN0bCBlbmFibGUgZGhjbGllbnQtb24tYm9vdC5zZXJ2aWNlCiAgLSBzeXN0ZW1jdGwgc3RhcnQgZGhjbGllbnQtb24tYm9vdC5zZXJ2aWNlCiAgLSBzZWQgLWkgIi8jXD9QZXJtaXRSb290TG9naW4vcy9eLiokL1Blcm1pdFJvb3RMb2dpbiB5ZXMvZyIgL2V0Yy9zc2gvc3NoZF9jb25maWc= name: cloudinitdisk |
1 2 3 4 |
kubectl get vm -A NAMESPACE NAME AGE STATUS READY default rocky810 50d Running True default win10 46d Running True |
windows 10 ltsc 2021版镜像
然后是这次折腾的重点windows 10 ltsc 2021版镜像
当前大家默认使用的是cloudbase 的win10 专业版镜像,使用上感觉有点卡,于是尝试以前使用的windows 10 ltsc 2021。
- 先检查 之前安装的 kubevirt cdi 服务
1 2 3 |
kubectl -n cdi get svc -l cdi.kubevirt.io=cdi-uploadproxy NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE cdi-uploadproxy ClusterIP 10.233.7.11 <none> 443/TCP 3h24m |
- 然后用virtctl 上传原始ISO镜像 ,实际名称为
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
virtctl image-upload --image-path=win10_ltsc.iso --storage-class hwameistor-storage-lvm-hdd pvc iso-win10 --size=7Gi --insecure --uploadproxy-url=https://10.233.7.11 --force-binds 输出日志为 PVC default/iso-win10 not found PersistentVolumeClaim default/iso-win10 created Waiting for PVC iso-win10 upload pod to be ready... Pod now ready Uploading data to https://10.233.7.11 4.64 GiB / 4.64 GiB [-------------------------------------------------------------------------------------] 100.00% 42.76 MiB p/s 1m51s Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress Processing completed successfully Uploading win10_ltsc.iso completed successfully |
- 然后用这个 yaml 部署 vm
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
cat win10.vm.yaml apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: win10 spec: runStrategy: Always template: metadata: labels: kubevirt.io/domain: win10 spec: domain: cpu: cores: 4 devices: disks: - bootOrder: 1 cdrom: bus: sata name: cdromiso - disk: bus: virtio name: harddrive - cdrom: bus: sata name: virtiocontainerdisk interfaces: - masquerade: {} model: e1000 name: default machine: type: q35 resources: requests: memory: 16G networks: - name: default pod: {} volumes: - name: cdromiso persistentVolumeClaim: claimName: iso-win10 - name: harddrive hostDisk: capacity: 50Gi path: /data/disk.img type: DiskOrCreate - containerDisk: image: dce-boot.io/docker.io/kubevirt/virtio-container-disk name: virtiocontainerdisk |
- kubectl apply -f win10.vm.yaml






1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
kubectl get pvc disk-windows -oyaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"disk-windows","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"25Gi"}},"storageClassName":"local-path"}} pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path volume.kubernetes.io/selected-node: gm-10-29-221-9 volume.kubernetes.io/storage-provisioner: rancher.io/local-path creationTimestamp: "2025-08-16T04:04:44Z" finalizers: - kubernetes.io/pvc-protection name: disk-windows namespace: default resourceVersion: "32202616" uid: d06116f4-606e-4567-884f-cb11bb44e8c0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 25Gi storageClassName: local-path volumeMode: Filesystem volumeName: pvc-d06116f4-606e-4567-884f-cb11bb44e8c0 status: accessModes: - ReadWriteOnce capacity: storage: 25Gi phase: Bound |
然后是正常安装
遇到了一次登录交互报错









1 2 3 4 5 6 7 8 9 10 11 |
cd /opt/local-path-provisioner/pvc-d06116f4-606e-4567-884f-cb11bb44e8c0_default_disk-windows/ qemu-img convert -O qcow2 disk.img win10.ltsc.qcow2 得到1个12G的qcow2 格式镜像 # dockerfile构建 FROM scratch COPY --chown=107:107 --from=build win10.ltsc.qcow2 /disk/win10.ltsc.qcow2 构建并推送 nerdctl build --platform=amd64 -t dce-boot.io/public/win10.ltsc:v1 --insecure-registry . |
遗留问题
dce-boot.io/public/win10.ltsc:v1 这个镜像 没有经过类似 cloudbase 的优化,印象中不能直接给其他环境使用,直接启动,下次有空再补文档吧。
文章评论