[EKS] 쿠버네티스 etcd 클러스터 백업, #3 Velero를 이용한 AWS EKS 백업 복구 방법

 

1편 
  kubernetes에게 Etcd란?
  kubernetes 백업의 필요성
  Etcd의 백업 및 복구(Backup and Restore) 
  Etcd의 백업의 두가지 방법

2편 
  etcd 내장 스냅샷

3편 
  Velero를 이용한 AWS EKS 백업 복구 방법

 

ㅁ 개요

 ㅇ 지난 시간 Etcd 내장 스냅샷 기능에 대해서 실습하는 과정을 정리하였다.

 ㅇ 이번 글에서는 Velero를 이용한 AWS EKS 백업 복구 방법을 정리하도록 하겠다.

 

 

ㅁ EKS Cluster

 ㅇ 백업 대상이 되는 EKS Cluster이다.

 

 

ㅁ WorkNode Scale Out

$ eksctl scale nodegroup --name=work-nodes --cluster=k8s-peterica --nodes=1 --nodes-min=1 --nodes-max=2
2022-10-08 00:21:26 [ℹ]  scaling nodegroup "work-nodes" in cluster k8s-peterica
2022-10-08 00:21:27 [ℹ]  waiting for scaling of nodegroup "work-nodes" to complete
2022-10-08 00:21:57 [ℹ]  nodegroup successfully scaled

 

 

ㅁ S3 생성

$ aws s3 mb s3://peter-velero-eks-backup-test
make_bucket: peter-velero-eks-backup-test

 ㅇ 백업을 위한 peter-velero-eks-backup-test S3를 만든다.

 

 ㅇ 만들어진 S3를 콘솔에서 확인할 수 있다.

 ㅇ S3 정책 변경은 이 링크를 참고하면 된다.

 

 

ㅁ Velero 클라이언트 설치 

 ㅇ Velero GitHub 저장소에서 클라이언트를 다운로드한다.

 ㅇ Velero 공식 설치 안내 페이지 

# 다운로드
$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.9.2/velero-v1.9.2-linux-amd64.tar.gz

# 압축해제
$ tar zxvf velero-v1.9.2-linux-amd64.tar.gz

# path 이동(root 권한)
sudo mv velero /usr/local/bin/.

 

# 설치확인
$ velero version
Client:
	Version: v1.9.2
	Git commit: 82a100981cc66d119cf9b1d121f45c5c9dcf99e1

 

 

ㅁ 압축해제한 폴더의 예제 및 설명 페이지

 ㅇ 클라이언트 설치 시 예제가 제공된다.

 

 ㅇ Velero에서 제공하는 실습 예제 페이지

 

 

ㅁ Velero 자동완성

# 자동완성 추가
echo 'source <(velero completion bash)' >>~/.bashrc

# 자동완성 테스트
$ velero
backup             client             debug              get                plugin             schedule           version
backup-location    completion         delete             help               restic             snapshot-location
bug                create             describe           install            restore            uninstall

 ㅇ 자동완성 기능을 추가하였다.

 

 

ㅁ AWS configure

$ aws configure
AWS Access Key ID [****************TAT3]:
AWS Secret Access Key [****************tJ8L]:
Default region name [ap-northeast-2]:
Default output format [None]:

 ㅇ 사전에 aws confiqure를 마친 상태이다.

 ㅇ 완료하여 aws 접속을 위한 엑세스 정보가 ~/.aws/credentials에 저장되어있다.

 

 

ㅁVelero 서버 설치

$ velero install \
>  --provider aws \
>  --plugins velero/velero-plugin-for-aws:v1.4.1 \
>  --bucket peter-velero-eks-backup-test \
>  --backup-location-config region=ap-northeast-2 \
>  --snapshot-location-config region=ap-northeast-2 \
>  --secret-file /home/ec2-user/.aws/credentials

CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
CustomResourceDefinition/backupstoragelocations.velero.io: already exists, proceeding
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
CustomResourceDefinition/deletebackuprequests.velero.io: already exists, proceeding
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
CustomResourceDefinition/downloadrequests.velero.io: already exists, proceeding
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumebackups.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
CustomResourceDefinition/podvolumerestores.velero.io: already exists, proceeding
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
CustomResourceDefinition/resticrepositories.velero.io: already exists, proceeding
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
CustomResourceDefinition/restores.velero.io: already exists, proceeding
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
CustomResourceDefinition/schedules.velero.io: already exists, proceeding
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
CustomResourceDefinition/serverstatusrequests.velero.io: already exists, proceeding
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
CustomResourceDefinition/volumesnapshotlocations.velero.io: already exists, proceeding
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: already exists, proceeding
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
ClusterRoleBinding/velero: already exists, proceeding
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: already exists, proceeding
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: already exists, proceeding
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: already exists, proceeding
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: attempting to create resource client
VolumeSnapshotLocation/default: already exists, proceeding
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: already exists, proceeding
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

 ㅇ 플러그인 정보는 velero plugin git hub에서 확인할 수 있다.

 

 

ㅁ EKS 복구 테스트 Application 설치

# eksbackupdemo 네임스페이스 생성
$ kubectl create namespace eksbackupdemo
namespace/eksbackupdemo created

# sample app 설치
$ kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0 -n eksbackupdemo
deployment.apps/web created

# nginx 설치
$ kubectl create deployment nginx --image=nginx -n eksbackupdemo
deployment.apps/nginx created

# 설치확인
$ kubectl get po -n eksbackupdemo
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-lxct8   1/1     Running   0          32m
web-79d88c97d6-mjfw9     1/1     Running   0          32m

 

 

 

ㅁ 백업 생성과  확인

# 백업 생성
$ velero backup create firstbackup --include-namespaces eksbackupdemo
Backup request "firstbackup" submitted successfully.

Run `velero backup describe firstbackup` or `velero backup logs firstbackup` for more details.

# 백업 생성 확인
$ velero backup describe firstbackup
Name:         firstbackup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.22.13-eks-15b7512
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=22+

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  eksbackupdemo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2022-10-08 09:48:24 +0900 KST
Completed:  2022-10-08 09:48:26 +0900 KST

Expiration:  2022-11-07 09:48:24 +0900 KST

Total items to be backed up:  10
Items backed up:              10

Velero-Native Snapshots: <none included>

 ㅇ 실제로 S3에도 firstbackup이라는 폴더가 생성되었고 백업된 데이터를 확인 할 수 있었다.

 

 

 ㅁ 재해 시뮬레이션

# demo 네임스페이스 삭제
$ kubectl delete namespaces eksbackupdemo
namespace "eksbackupdemo" deleted

# 삭제 확인
$ kubectl get ns
NAME              STATUS   AGE
default           Active   18d
kube-node-lease   Active   18d
kube-public       Active   18d
kube-system       Active   18d
velero            Active   34m

 

 

ㅁ 복구

[ec2-user@ip-172-31-43-214 veleroTest]$ velero restore create --from-backup firstbackup
Restore request "firstbackup-20221008100034" submitted successfully.
Run `velero restore describe firstbackup-20221008100034` or `velero restore logs firstbackup-20221008100034` for more details.

# 복원된 네임스페이스 확인
$ kubectl get ns
NAME              STATUS   AGE
default           Active   18d
eksbackupdemo     Active   18s
kube-node-lease   Active   18d
kube-public       Active   18d
kube-system       Active   18d
velero            Active   35m

# 파드 복구 확인
$ kubectl get po -n eksbackupdemo
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-c5kd7   1/1     Running   0          27s
web-79d88c97d6-9vd7f     1/1     Running   0          27s

 

 

ㅁ 함께 보면 좋은 사이트

Velero

 ㅇ 공식 홈페이지

 

Velero Docs - Examples

 ㅇ 예제 페이지

 

Backup and restore your Amazon EKS cluster resources using Velero | Amazon Web Services

 

 ㅇ aws에서 제공하는 velero 이용방법

 

Backup an entire Kubernetes cluster using Velero to AWS S3 | SkildOps

 ㅇ helm을 이용한 velero 설치, eks 백업복구