[Kubernetes] Kubernetes Pod별 Tcpdump 방법

[kubernetes] 모니터링 방법 정리

ㅁ 개요

 ㅇ kubernetes cluster 운영환경에서 pod에서 발생한 IP packet에 대한 tcpdump를 해야하는 경우가 발생한다.

 

ㅁ tcpdump를 확인해야만 했던 이유

 외부 연동된 사이트와의 통신에서 천만분의 1의 경우로 network timeout이 발생하여 확인하는 과정이었다. 연동된 사이트에서 확인한 결과 우리가 발신한 packet의 Accept-Charse의 길이가 길다는 이슈를 확인하였다. 받는 쪽에서 Accept-Charset 항목이 상당히 긴 편이라 짧은 메시지의 경우에도 네트워크 상에서 패킷이 다건 분할 되어 처리되고 있는 것 같다면서, 혹시 발송된 정보와 다르다면 알려달라는 요청이 있었다. 그 과정에서 나는 1차적으로 소스상 분석을 시작하였고 소스상에서 accept-charset를 설정하는 부분은 찾을 수 없었다. 그래서 결국 pod에서 나가는 packet의  tcpdump를 확인하게 되었다.

 

POST /ag/1.1/message HTTP/1.1 X-Forwarded-For: 52.79.65.145 Host:172.*.*.*:5084 Content-Length: 336 Accept: text/plain, application/json, application/*+json, */* Authorization: Bearer ******************* Content-Type: application/json Accept-Charset: big5, big5-hkscs, cesu-8, euc-jp, euc-kr, gb18030, gb2312, gbk, ibm-thai, ibm00858, ibm01140, ibm01141, ibm01142, ibm01143, ibm01144, ibm01145, ibm01146, ibm01147, ibm01148, ibm01149, ibm037, ibm1026, ibm1047, ibm273, ibm277, ibm278, ibm280, ibm284, ibm285, ibm290, ibm297, ibm420, ibm424, ibm437, ibm500, ibm775, ibm850, ibm852, ibm855, ibm857, ibm860, ibm861, ibm862, ibm863, ibm864, ibm865, ibm866, ibm868, ibm869, ibm870, ibm871, ibm918, iso-2022-cn, iso-2022-jp, iso-2022-jp-2, iso-2022-kr, iso-8859-1, iso-8859-13, iso-8859-15, iso-8859-2, iso-8859-3, iso-8859-4, iso-8859-5, iso-8859-6, iso-8859-7, iso-8859-8, iso-8859-9, jis_x0201, jis_x0212-1990, koi8-r, koi8-u, shift_jis, tis-620, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-32be, utf-32le, utf-8, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258, windows-31j, x-big5-hkscs-2001, x-big5-solaris, x-compound_text, x-euc-jp-linux, x-euc-tw, x-eucjp-open, x-ibm1006, x-ibm1025, x-ibm1046, x-ibm1097, x-ibm1098, x-ibm1112, x-ibm1122, x-ibm1123, x-ibm1124, x-ibm1166, x-ibm1364, x-ibm1381, x-ibm1383, x-ibm300, x-ibm33722, x-ibm737, x-ibm833, x-ibm834, x-ibm856, x-ibm874, x-ibm875, x-ibm921, x-ibm922, x-ibm930, x-ibm933, x-ibm935, x-ibm937, x-ibm939, x-ibm942, x-ibm942c, x-ibm943, x-ibm943c, x-ibm948, x-ibm949, x-ibm949c, x-ibm950, x-ibm964, x-ibm970, x-iscii91, x-iso-2022-cn-cns, x-iso-2022-cn-gb, x-iso-8859-11, x-jis0208, x-jisautodetect, x-johab, x-macarabic, x-maccentraleurope, x-maccroatian, x-maccyrillic, x-macdingbat, x-macgreek, x-machebrew, x-maciceland, x-macroman, x-macromania, x-macsymbol, x-macthai, x-macturkish, x-macukraine, x-ms932_0213, x-ms950-hkscs, x-ms950-hkscs-xp, x-mswin-936, x-pck, x-sjis_0213, x-utf-16le-bom, x-utf-32be-bom, x-utf-32le-bom, x-windows-50220, x-windows-50221, x-windows-874, x-windows-949, x-windows-950, x-windows-iso2022jp User-Agent: Apache-HttpAsyncClient/4.1.4 (Java/1.8.0_212) {body_payload *********************}

 ㅇ 샘플로 보내준 전문에서 Accept-Charset이 너무 긴것을 확인 할 수 있었다.

 

 

 

 kubernetes 환경은 workNode 안에 컨테이너가 구동되는 방식이다. 그렇기 때문에 tcpdump는 확인하는 방법은 Container 내부에서 직접 tcpdump를 수행하거나, WorkNode에서 특정 pod의 tcpdump를 수행할 수 있다.

 

---

   1.  kubernetes cluster 운영환경에서 tcpdump를 확인하는 방법

---

 

ㅁ Container 내부에서 tcpdump

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: centos
  name: centos
spec:
  containers:
  - image: centos:7
    name: centos
    command: ["/bin/sleep", "3650d"]
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

 ㅇ tcpdump 테스트를 위한 centos:7 pod를 생성하기 위한 tcpTest.yaml  파일을 작성하였다.

 

 

$ kubectl apply -f tcpTest.yaml
pod/centos created

 ㅇ pod를 생성하였다.

 

 

 ㅇ 생성된 centos 파드확인

 

 

# pod 내부 접속
$ kubectl exec -it centos bash

 ㅇ pod 내부로 접속한다.

 

 

# 네트워크 확인을 위한 툴 설치
$ yum -y install net-tools
.......
Running transaction
  Installing : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                                                          1/1
  Verifying  : net-tools-2.0-0.25.20131004git.el7.x86_64                                                                                                                                                                                          1/1

Installed:
  net-tools.x86_64 0:2.0-0.25.20131004git.el7

Complete!

 ㅇ 기본 Centos 7에는 ifconfig, netstat 등의 네트워크 명령어가 없기 때문에 net-tools 패키지를 설치한다.

 

 

$ nic 확인
$ [root@centos /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.71.6  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 22:b0:54:32:4b:56  txqueuelen 0  (Ethernet)
        RX packets 6804  bytes 81654878 (77.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4763  bytes 334230 (326.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 ㅇ kubernetes pod 내부의 기본 NIC(Network Interface Controller)는 eth0이다.

 

참고
NIC(Network Interface Controller)이란 컴퓨터를 네트워크에 연결하여 통신하기 위해 사용하는 하드웨어 장치

 

 

# NIC eth0에 대한 tcpdump 시작
$ tcpdump -i eth0 -w tcpdump.pcap
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
55 packets captured
55 packets received by filter
0 packets dropped by kernel

 ㅇ tcpdump.pcap가 생성되었다.

 

 

 

---

   2. EKS Work Node에서 Pod별 Tcpdump 방법

---

ㅁ 대상 Pod의 WorkNode 식별

 ㅇ tcpdump가 필요한 Pod가 실제로 작동하는 WorkNode의 정보를 확인해야 한다.

 ㅇ centos Pod는 192.168.69.80 노드에서 작동 중이다.

 

ㅁ WorkNode 확인

  ㅇ 접속한 work node정보이다.

 

 

ㅁ workNode 접속

# pem 키를 이용한 ssh 접속
$ ssh -i ~/kube/aws-login-key.pem ec2-user@3.37.38.20

 

 

ㅁ tcpdump 툴 설치

# tcpdump 명령어 확인
$ which tcpdump

# tcpdump 설치 
$ sudo yum install -y tcpdump

 ㅇ tcpdump 명령어의 설치여부를 확인하여, 설치하였다.

 

 

ㅁ Pod의 ENI 확인

  ㅇ tcpdump를 목표로 하는 pod의 IP정보를 확인한다.

 

 

 

$ arp -a | grep 192.168.71.6
ip-192-168-71-6.ap-northeast-2.compute.internal (192.168.71.6) at 22:b0:54:32:4b:56 [ether] on enie6451f39401

 ㅇ pod의 192.168.71.6를 통해 ENI 고유정보인  enie6451f39401을 얻게 된다.

 

참고
ENI(Elastic Network Interface)는 인스턴스가  다른 네트워크 리소스와 통신할 수 있게 해준다. ENI는 물리적 서버의 네트워크 인터페이스(NIC)와 같은 기능을 제공하며,  모든 인스턴스는 기본 ENI가 있어야 한다.

 

 

ㅁ ENI 지정한 tcpdump 수행

# ENI enie6451f39401 지정한 tcpdump 수행
$ sudo tcpdump -i enie6451f39401 -w tcpdumpNode.pcap
tcpdump: listening on enie6451f39401, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
46 packets captured
46 packets received by filter
0 packets dropped by kernel

# dump파일 확인
$ ls
awscli-bundle  tcpdumpNode.pcap

 

ㅁ tcpdump pcap 파일 확인

$ tcpdump -r tcpdumpNode.pcap | more
reading from file tcpdumpNode.pcap, link-type EN10MB (Ethernet)
10:22:22.231380 IP ip-192-168-71-6.ap-northeast-2.compute.internal.56132 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 51412+ A? peterica.tistory.com.default.svc.cluster.local. (64)
10:22:22.231805 ARP, Request who-has ip-192-168-71-6.ap-northeast-2.compute.internal tell ip-192-168-69-80.ap-northeast-2.compute.internal, length 28
10:22:22.231815 ARP, Reply ip-192-168-71-6.ap-northeast-2.compute.internal is-at 22:b0:54:32:4b:56 (oui Unknown), length 28
10:22:22.231818 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.56132: 51412 NXDomain*- 0/1/0 (157)
10:22:22.231858 IP ip-192-168-71-6.ap-northeast-2.compute.internal.56132 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 15584+ AAAA? peterica.tistory.com.default.svc.cluster.local. (64)
10:22:22.231993 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.56132: 15584 NXDomain*- 0/1/0 (157)
10:22:22.232078 IP ip-192-168-71-6.ap-northeast-2.compute.internal.45115 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 33323+ A? peterica.tistory.com.svc.cluster.local. (56)
10:22:22.232197 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.45115: 33323 NXDomain*- 0/1/0 (149)
10:22:22.232217 IP ip-192-168-71-6.ap-northeast-2.compute.internal.45115 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 20530+ AAAA? peterica.tistory.com.svc.cluster.local. (56)
10:22:22.232295 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.45115: 20530 NXDomain*- 0/1/0 (149)
10:22:22.232344 IP ip-192-168-71-6.ap-northeast-2.compute.internal.34012 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 32854+ A? peterica.tistory.com.cluster.local. (52)
10:22:22.232436 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.34012: 32854 NXDomain*- 0/1/0 (145)
10:22:22.232458 IP ip-192-168-71-6.ap-northeast-2.compute.internal.34012 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 40539+ AAAA? peterica.tistory.com.cluster.local. (52)
10:22:22.232528 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.34012: 40539 NXDomain*- 0/1/0 (145)
10:22:22.232583 IP ip-192-168-71-6.ap-northeast-2.compute.internal.46871 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 20085+ A? peterica.tistory.com.ap-northeast-2.compute.internal. (70)
10:22:22.232664 IP ip-192-168-71-6.ap-northeast-2.compute.internal.46871 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 55932+ AAAA? peterica.tistory.com.ap-northeast-2.compute.internal. (70)
10:22:22.233981 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.46871: 20085 NXDomain 0/1/0 (193)
10:22:22.234158 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.46871: 55932 NXDomain 0/1/0 (193)
10:22:22.234216 IP ip-192-168-71-6.ap-northeast-2.compute.internal.34264 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 30363+ A? peterica.tistory.com. (38)
10:22:22.234243 IP ip-192-168-71-6.ap-northeast-2.compute.internal.34264 > ip-10-100-0-10.ap-northeast-2.compute.internal.domain: 29345+ AAAA? peterica.tistory.com. (38)
10:22:22.234651 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.34264: 29345 1/0/0 CNAME wildcard-tistory-fz0x1pwf.kgslb.com. (107)
10:22:22.307538 IP ip-10-100-0-10.ap-northeast-2.compute.internal.domain > ip-192-168-71-6.ap-northeast-2.compute.internal.34264: 30363 2/0/0 CNAME wildcard-tistory-fz0x1pwf.kgslb.com., A 211.231.99.250 (158)
10:22:22.355613 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [S], seq 1240980653, win 62727, options [mss 8961,sackOK,TS val 741852428 ecr 0,nop,wscale 7], length 0
10:22:22.428376 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [S.], seq 3193383228, ack 1240980654, win 28960, options [mss 1460,sackOK,TS val 2188351204 ecr 741852428,n
op,wscale 10], length 0
10:22:22.428444 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [.], ack 1, win 491, options [nop,nop,TS val 741852500 ecr 2188351204], length 0
10:22:22.569894 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [P.], seq 1:207, ack 1, win 491, options [nop,nop,TS val 741852642 ecr 2188351204], length 206
10:22:22.645318 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [P.], seq 1:4180, ack 207, win 30, options [nop,nop,TS val 2188351420 ecr 741852642], length 4179
10:22:22.645430 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [.], ack 4180, win 459, options [nop,nop,TS val 741852717 ecr 2188351420], length 0
10:22:22.652961 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [P.], seq 207:333, ack 4180, win 459, options [nop,nop,TS val 741852725 ecr 2188351420], length 126
10:22:22.727361 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [P.], seq 4180:4231, ack 333, win 30, options [nop,nop,TS val 2188351503 ecr 741852725], length 51
10:22:22.727643 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [P.], seq 333:449, ack 4231, win 459, options [nop,nop,TS val 741852800 ecr 2188351503], length 116
10:22:22.840017 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], ack 449, win 30, options [nop,nop,TS val 2188351616 ecr 741852800], length 0
10:22:22.849298 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], seq 4231:15815, ack 449, win 30, options [nop,nop,TS val 2188351625 ecr 741852800], length 11584
10:22:22.849416 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], seq 15815:18711, ack 449, win 30, options [nop,nop,TS val 2188351625 ecr 741852800], length 2896
10:22:22.849448 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [.], ack 18711, win 436, options [nop,nop,TS val 741852921 ecr 2188351625], length 0
10:22:22.922271 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], seq 18711:36087, ack 449, win 30, options [nop,nop,TS val 2188351698 ecr 741852921], length 17376
10:22:22.922321 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], seq 36087:41879, ack 449, win 30, options [nop,nop,TS val 2188351698 ecr 741852921], length 5792
10:22:22.922350 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [.], ack 41879, win 391, options [nop,nop,TS val 741852994 ecr 2188351698], length 0
10:22:22.922387 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [P.], seq 41879:42410, ack 449, win 30, options [nop,nop,TS val 2188351698 ecr 741852921], length 531
10:22:22.925658 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [P.], seq 449:480, ack 42410, win 443, options [nop,nop,TS val 741852998 ecr 2188351698], length 31
10:22:22.925692 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [F.], seq 480, ack 42410, win 443, options [nop,nop,TS val 741852998 ecr 2188351698], length 0
10:22:22.998418 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [.], ack 480, win 30, options [nop,nop,TS val 2188351774 ecr 741852998], length 0
10:22:22.998430 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [P.], seq 42410:42441, ack 480, win 30, options [nop,nop,TS val 2188351774 ecr 741852998], length 31
10:22:22.998431 IP 211.231.99.250.https > ip-192-168-71-6.ap-northeast-2.compute.internal.44538: Flags [F.], seq 42441, ack 481, win 30, options [nop,nop,TS val 2188351774 ecr 741852998], length 0
10:22:22.998500 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [R], seq 1240981133, win 0, length 0
10:22:22.998505 IP ip-192-168-71-6.ap-northeast-2.compute.internal.44538 > 211.231.99.250.https: Flags [R], seq 1240981134, win 0, length 0

 ㅇ 수집된 tcpdump의 내용을 확인할 수 있다.

 ㅇ 만들어진 pcap 파일은 Wireshark으로 열어볼 수 있다.

 

 

ㅁ SCP to local

[ec2-user@ip-192-168-69-80 ~]$ scp tcpdumpNode.pcap peterseo@peterica.iptime.org:/Users/peterseo/kube/tcpdumpTest/.
The authenticity of host 'peterica.iptime.org (11*.9*.14*.12*)' can't be established.
ECDSA key fingerprint is SHA256:w9p7ZNqoLuZP0BV29X9aKxOKSSas**.
ECDSA key fingerprint is MD5:ea:16:ca:e6:6c:fe:14:00:*.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'peterica.iptime.org,11*.9*.14*.12*' (ECDSA) to the list of known hosts.
Password:
tcpdumpNode.pcap

 

ㅁ 함께 보면 좋은 사이트

Kubernetes Container에 대한 TCPDUMP 방법

 

 

탄력적 네트워크 인터페이스 - Amazon Elastic Compute Cloud

 ㅇ 탄력적 네트워크 인터페이스(ENI)란 

 

Wireshark · Go Deep.

ㅇ 패킷 분석 툴, 와이어샤크(WireShark) 다운로드