[kubernetes] 다중 스케줄러

[kubernetes] 쿠버네티스 목차

ㅁ 들어가며

  쿠버네티스는 기본 스케줄러 하나를 사용한다. 하지만 모든 Application이 하나의 스케줄에 만족하지 못할 경우, 다중스케줄러를 사용할 수 있다. 이 번 글은 쿠버네티스 문서, 다중 스케줄러 설정를 보고 실습 정리한 내용이다.

 

 

ㅁ 스케줄러에서 사용할 Deployment 생성

 아래의 Deployment 생성문은 클러스터 내에 직접 파드를 생성하는 Static Pod([kubernetes] Static Pod란?)가 아닌 Deployment의 형태로 만들어져 레플리카 셋에 의해 관리된다. 스케줄러 또한 파드로 관리하기 때문에 유동적인 스케줄러를 만들 수 있다.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-volume-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:volume-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: my-scheduler
    leaderElection:
      leaderElect: false    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: my-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
        image: gcr.io/my-gcp-project/my-kube-scheduler:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
          - name: config-volume
            mountPath: /etc/kubernetes/my-scheduler
      hostNetwork: false
      hostPID: false
      volumes:
        - name: config-volume
          configMap:
            name: my-scheduler-config

 

 

ㅁ my-scheduler 클러스터에서 실행하기

# 테스트 환경구성
$ minikube start

# my-scheduler.yaml 생성 및 검수
$ kubectl apply -f my-scheduler.yaml --dry-run=server
serviceaccount/my-scheduler created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler created (server dry run)
configmap/my-scheduler-config created (server dry run)
deployment.apps/my-scheduler created (server dry run)

# 적용
$ k apply -f my-scheduler.yaml
serviceaccount/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler created
configmap/my-scheduler-config created
deployment.apps/my-scheduler created

 

ㅁ Pod 생성 실패 원인 분석

# pod 상태 확인
$ k get po -n kube-system my-scheduler-5bff45998-9xtqg
NAME                           READY   STATUS             RESTARTS   AGE
my-scheduler-5bff45998-9xtqg   0/1     ImagePullBackOff   0          4m9s

# pod 상세 조회
$ k describe po -n kube-system my-scheduler-5bff45998-9xtqg
~~~~~~~~~~~~~~~~~~~~
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m22s                  default-scheduler  Successfully assigned kube-system/my-scheduler-5bff45998-9xtqg to minikube
  Normal   Pulling    2m48s (x4 over 4m21s)  kubelet            Pulling image "gcr.io/my-gcp-project/my-kube-scheduler:1.0"
  Warning  Failed     2m48s (x4 over 4m20s)  kubelet            Failed to pull image "gcr.io/my-gcp-project/my-kube-scheduler:1.0": Error response from daemon: Head "https://gcr.io/v2/my-gcp-project/my-kube-scheduler/manifests/1.0": unknown: Service 'containerregistry.googleapis.com' is not enabled for consumer 'project:my-gcp-project'.
  Warning  Failed     2m48s (x4 over 4m20s)  kubelet            Error: ErrImagePull
  Warning  Failed     2m33s (x6 over 4m20s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m19s (x7 over 4m20s)  kubelet            Back-off pulling image "gcr.io/my-gcp-project/my-kube-scheduler:1.0"

 ㅇ 스케줄 Pod의 이미지를 받을 수 없는 상태였다.

 ㅇ gcr.io/my-gcp-project/my-kube-scheduler:1.0은 비공개 Container Registory이다.

 ㅇ 사용가능한 이미지 k8s.gcr.io/kube-scheduler:v1.24.0로 변경 적용하였다.

 

# 정상 실행 확인
$ k get po -n kube-system my-scheduler-54bcd87b49-78qtv
NAME                            READY   STATUS    RESTARTS   AGE
my-scheduler-54bcd87b49-78qtv   1/1     Running   0          2m57s

 

 

ㅁ 리더 선출 활성화

리더 선출이 활성화된 상태로 다중 스케줄러를 실행하기 위한 작업

 

# 리더 선출 옵션 변경
$  vi my-scheduler.yaml
.....
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: my-scheduler
    leaderElection:
      leaderElect: true  <=== 수정
.....


# 적용
$ k apply -f my-scheduler.yaml
serviceaccount/my-scheduler unchanged
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler unchanged
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler unchanged
configmap/my-scheduler-config configured  <========적용됨
deployment.apps/my-scheduler configured   <========적용됨

 

 

# clusterRole 수정
$ kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: "2024-01-18T12:30:34Z"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-scheduler
  resourceVersion: "21183"
  uid: afd94de9-f539-4580-b42e-be6d29087607
rules:
- apiGroups:
  - ""
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
- apiGroups:
  - coordination.k8s.io
  resourceNames:
  - kube-scheduler
  - my-scheduler  <============추가됨
  resources:
  - leases
  verbs:
  - get
  - update
- apiGroups:
..............

 

 ㅇ 실질적으로 클러스터에 스케줄러를 인식시켜주는 작업이다.

 ㅇ 이제 두 번째 스케줄러가 실행되었다.

 

ㅁ 파드 스케줄러 지정하기

 ㅇ 미지정: 스케줄러를 명시 하지 않으면 기본 스케줄러에 의해 스케줄링된다.

apiVersion: v1
  kind: Pod
  metadata:
    name: no-annotation
    labels:
      name: multischeduler-example
  spec:
    containers:
    - name: pod-with-no-annotation-container
      image: registry.k8s.io/pause:2.0

 

 

ㅇ 기본 지정

apiVersion: v1
kind: Pod
metadata:
  name: annotation-default-scheduler
  labels:
    name: multischeduler-example
spec:
  schedulerName: default-scheduler  <==== 기본지정
  containers:
  - name: pod-with-default-annotation-container
    image: registry.k8s.io/pause:2.0

 

 

ㅇ my-scheduler 지정

apiVersion: v1
kind: Pod
metadata:
  name: annotation-second-scheduler
  labels:
    name: multischeduler-example
spec:
  schedulerName: my-scheduler
  containers:
  - name: pod-with-second-annotation-container
    image: registry.k8s.io/pause:2.0

 

 

ㅁ 이벤트 확인

 

 ㅇ 기본 명령어는 순서가 정리 되지 않고, 스케줄러의 정보가 없다.

 

# 시간소팅 및 스케줄러 정보 표시
$ kubectl get events --sort-by='.metadata.creationTimestamp' -A -o wide
kube-system   13m         Normal    ScalingReplicaSet         deployment/my-scheduler                                                                         deployment-controller                                                    Scaled up replica set my-scheduler-54bcd87b49 to 1                                                                                                                                              13m          1       my-scheduler.17aba51a26a7938a
kube-system   13m         Normal    Scheduled                 pod/my-scheduler-54bcd87b49-6cbp2                                                               default-scheduler                                                        Successfully assigned kube-system/my-scheduler-54bcd87b49-6cbp2 to minikube                                                                                                                     13m          1       my-scheduler-54bcd87b49-6cbp2.17aba51a2877d76a
kube-system   13m         Normal    SuccessfulCreate          replicaset/my-scheduler-54bcd87b49                                                              replicaset-controller                                                    Created pod: my-scheduler-54bcd87b49-6cbp2                                                                                                                                                      13m          1       my-scheduler-54bcd87b49.17aba51a2799ee9c
kube-system   13m         Normal    Pulling                   pod/my-scheduler-54bcd87b49-6cbp2      spec.containers{kube-second-scheduler}                   kubelet, minikube                                                        Pulling image "k8s.gcr.io/kube-scheduler:v1.24.0"                                                                                                                                               13m          1       my-scheduler-54bcd87b49-6cbp2.17aba51a5aa0a807
kube-system   13m         Normal    Pulled                    pod/my-scheduler-54bcd87b49-6cbp2      spec.containers{kube-second-scheduler}                   kubelet, minikube                                                        Successfully pulled image "k8s.gcr.io/kube-scheduler:v1.24.0" in 4.294s (4.294s including waiting)                                                                                              13m          1       my-scheduler-54bcd87b49-6cbp2.17aba51b5a93a379
kube-system   13m         Normal    Created                   pod/my-scheduler-54bcd87b49-6cbp2      spec.containers{kube-second-scheduler}                   kubelet, minikube                                                        Created container kube-second-scheduler                                                                                                                                                         13m          1       my-scheduler-54bcd87b49-6cbp2.17aba51b603a37ef
kube-system   13m         Normal    Started                   pod/my-scheduler-54bcd87b49-6cbp2      spec.containers{kube-second-scheduler}                   kubelet, minikube                                                        Started container kube-second-scheduler                                                                                                                                                         13m          1       my-scheduler-54bcd87b49-6cbp2.17aba51b66e5e84a
kube-system   10m         Normal    LeaderElection            lease/my-scheduler                                                                              my-scheduler                                                             my-scheduler-54bcd87b49-6cbp2_a7be60ad-da2b-4511-b4f6-24cf321fff5b became leader                                                                                                                10m          1       my-scheduler.17aba544fa73bcc1
default       8m46s       Normal    Scheduled                 pod/no-annotation                                                                               default-scheduler                                                        Successfully assigned default/no-annotation to minikube                                                                                                                                         8m46s        1       no-annotation.17aba55cc9219670
default       8m45s       Normal    Pulling                   pod/no-annotation                      spec.containers{pod-with-no-annotation-container}        kubelet, minikube                                                        Pulling image "registry.k8s.io/pause:2.0"                                                                                                                                                       8m45s        1       no-annotation.17aba55cfd2d24f7
default       8m43s       Normal    Created                   pod/no-annotation                      spec.containers{pod-with-no-annotation-container}        kubelet, minikube                                                        Created container pod-with-no-annotation-container                                                                                                                                              8m43s        1       no-annotation.17aba55d785d71dc
default       8m43s       Normal    Started                   pod/no-annotation                      spec.containers{pod-with-no-annotation-container}        kubelet, minikube                                                        Started container pod-with-no-annotation-container                                                                                                                                              8m43s        1       no-annotation.17aba55d7f08631c
default       8m43s       Normal    Pulled                    pod/no-annotation                      spec.containers{pod-with-no-annotation-container}        kubelet, minikube                                                        Successfully pulled image "registry.k8s.io/pause:2.0" in 2.034s (2.034s including waiting)                                                                                                      8m43s        1       no-annotation.17aba55d76787d27
default       8m13s       Normal    Scheduled                 pod/annotation-default-scheduler                                                                default-scheduler                                                        Successfully assigned default/annotation-default-scheduler to minikube                                                                                                                          8m13s        1       annotation-default-scheduler.17aba56454769252
default       8m12s       Normal    Pulled                    pod/annotation-default-scheduler       spec.containers{pod-with-default-annotation-container}   kubelet, minikube                                                        Container image "registry.k8s.io/pause:2.0" already present on machine                                                                                                                          8m12s        1       annotation-default-scheduler.17aba564854c59d7
default       8m12s       Normal    Started                   pod/annotation-default-scheduler       spec.containers{pod-with-default-annotation-container}   kubelet, minikube                                                        Started container pod-with-default-annotation-container                                                                                                                                         8m12s        1       annotation-default-scheduler.17aba5648df5d339
default       8m12s       Normal    Created                   pod/annotation-default-scheduler       spec.containers{pod-with-default-annotation-container}   kubelet, minikube                                                        Created container pod-with-default-annotation-container                                                                                                                                         8m12s        1       annotation-default-scheduler.17aba56486cf4337
default       7m49s       Normal    Scheduled                 pod/annotation-second-scheduler                                                                 my-scheduler, my-scheduler-my-scheduler-54bcd87b49-6cbp2                 Successfully assigned default/annotation-second-scheduler to minikube                                                                                                                           7m49s        1       annotation-second-scheduler.17aba569cbfdf07a
default       7m49s       Normal    Created                   pod/annotation-second-scheduler        spec.containers{pod-with-second-annotation-container}    kubelet, minikube                                                        Created container pod-with-second-annotation-container                                                                                                                                          7m49s        1       annotation-second-scheduler.17aba56a01275c29
default       7m49s       Normal    Started                   pod/annotation-second-scheduler        spec.containers{pod-with-second-annotation-container}    kubelet, minikube                                                        Started container pod-with-second-annotation-container                                                                                                                                          7m49s        1       annotation-second-scheduler.17aba56a073a5ae5
default       7m49s       Normal    Pulled                    pod/annotation-second-scheduler        spec.containers{pod-with-second-annotation-container}    kubelet, minikube                                                        Container image "registry.k8s.io/pause:2.0" already present on machine                                                                                                                          7m49s        1       annotation-second-scheduler.17aba569ff401a72

 

ㅁ함께 보면 좋은 사이트

ㅇ 쿠버네티스 공식문서 - 다중 스케줄러 설정