{"id":7416,"date":"2023-01-10T14:49:53","date_gmt":"2023-01-10T05:49:53","guid":{"rendered":"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=7416"},"modified":"2023-01-25T14:34:21","modified_gmt":"2023-01-25T05:34:21","slug":"kubernetes-calico-troubleshooting","status":"publish","type":"post","link":"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=7416","title":{"rendered":"Kubernetes &#8211; Calico Troubleshooting"},"content":{"rendered":"<h1>Kubernetes &#8211; Calico Troubleshooting<\/h1>\n<p>Calico \uc124\uce58\uc911 \uc624\ub958\uac00 \ubc1c\uc0dd\ud558\uc5ec \ud574\ub2f9 \uc624\ub958\ub97c \uc81c\uac70\ud558\ub294 \ubc29\ubc95\uc744 \uc815\ub9ac\ud569\ub2c8\ub2e4.<\/p>\n<h2>\ubc29\ud654\ubcbd \ud574\uc81c<\/h2>\n<p>\uac00\uc7a5 \ub9ce\uc774 \ubc1c\uc0dd\ud558\ub294 \uc624\ub958\ub294 \ubc29\ud654\ubcbd\uc774\ubbc0\ub85c \uc77c\ub2e8 \ubc29\ud654\ubcbd\uc740 \ud574\uc81c\ud574\uc900\ub2e4.<\/p>\n<p>\uc774\ud6c4\uc5d0 \uc624\ub958\uac00 \ud574\uacb0\ub418\uba74 \uadf8\ub54c \ud2b9\uc815 \ud3ec\ud2b8\ub9cc \uc624\ud508\ud558\ub294 \uc791\uc5c5\uc744 \ud574\uc900\ub2e4.<\/p>\n<h2>\uc774\uc804 CNI \uc794\uc7ac \uc81c\uac70<\/h2>\n<pre><code class=\"language-bash\">k describe po calico-node-lqstp -n kube-system\n......\nEvents:\n  Type     Reason     Age                     From     Message\n  ----     ------     ----                    ----     -------\n  Warning  Unhealthy  2m54s (x7432 over 18h)  kubelet  (combined from similar events): Readiness probe failed: 2023-01-10 03:26:10.015 [INFO][340172] confd\/health.go 180: Number of node(s) with BGP peering established = 0\ncalico\/node is not ready: BIRD is not ready: BGP not established with 10.44.0.0,172.16.0.202,172.16.0.203,172.16.0.204<\/code><\/pre>\n<p>weave \ub97c \uc124\uce58 \ud6c4 \uc81c\uac70\ud558\uace0, calico \ub97c \uc124\uce58\ud55c \uc0c1\ud0dc\uc774\ub2e4.<br \/>\nweave \uc758 \uc794\uc7ac\uac00 \ub0a8\uc544\uc788\ub294 \uac83\uc744 \ud655\uc778\ud560 \uc218 \uc788\ub2e4.<br \/>\n(flannel \ub3c4 \uc720\uc0ac\ud55c \ud615\ud0dc\ub85c \uc794\uc7ac\uac00 \ub0a8\ub294\ub2e4.)<\/p>\n<pre><code class=\"language-bash\">ifconfig\n......\nweave: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1376\n        inet 10.44.0.0  netmask 255.240.0.0  broadcast 10.47.255.255\n        inet6 fe80::e03c:19ff:fe0c:834f  prefixlen 64  scopeid 0x20&lt;link&gt;\n        ether e2:3c:19:0c:83:4f  txqueuelen 1000  (Ethernet)\n        RX packets 633  bytes 40806 (39.8 KiB)\n        RX errors 0  dropped 0  overruns 0  frame 0\n        TX packets 967058  bytes 40619287 (38.7 MiB)\n        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0<\/code><\/pre>\n<pre><code class=\"language-bash\">ip link\n......\n23: weave: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1376 qdisc noqueue state UP mode DEFAULT group default qlen 1000\n    link\/ether e2:3c:19:0c:83:4f brd ff:ff:ff:ff:ff:ff<\/code><\/pre>\n<p>\uc81c\uac70\ud574 \uc900\ub2e4.<\/p>\n<pre><code class=\"language-bash\">sudo ifconfig weave down\nsudo ip link delete weave<\/code><\/pre>\n<p>\uc624\ub958 \uc0c1\ud0dc\uc778 Pod \ub97c \uc0ad\uc81c\ud55c\ub2e4.<\/p>\n<pre><code class=\"language-bash\">k delete po calico-node-lqstp -n kube-system\nk get po -n kube-system -o wide\nk describe po calico-node-426rn -n kube-system\n......\nEvents:\n  Type     Reason     Age                From               Message\n  ----     ------     ----               ----               -------\n  Normal   Scheduled  49s                default-scheduler  Successfully assigned kube-system\/calico-node-426rn to es-search05\n  Normal   Pulled     49s                kubelet            Container image &quot;docker.io\/calico\/cni:v3.24.5&quot; already present on machine\n  Normal   Created    49s                kubelet            Created container upgrade-ipam\n  Normal   Started    48s                kubelet            Started container upgrade-ipam\n  Normal   Pulled     48s                kubelet            Container image &quot;docker.io\/calico\/cni:v3.24.5&quot; already present on machine\n  Normal   Created    48s                kubelet            Created container install-cni\n  Normal   Started    47s                kubelet            Started container install-cni\n  Normal   Pulled     47s                kubelet            Container image &quot;docker.io\/calico\/node:v3.24.5&quot; already present on machine\n  Normal   Created    46s                kubelet            Created container mount-bpffs\n  Normal   Started    46s                kubelet            Started container mount-bpffs\n  Normal   Pulled     46s                kubelet            Container image &quot;docker.io\/calico\/node:v3.24.5&quot; already present on machine\n  Normal   Created    46s                kubelet            Created container calico-node\n  Normal   Started    45s                kubelet            Started container calico-node\n  Warning  Unhealthy  43s (x2 over 44s)  kubelet            Readiness probe failed: calico\/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix \/var\/run\/calico\/bird.ctl: connect: connection refused<\/code><\/pre>\n<h2>connect: connection refused<\/h2>\n<pre><code class=\"language-bash\">k get po -n kube-system -o wide\nk describe po calico-node-426rn -n kube-system\n......\nEvents:\n  Type     Reason     Age                From               Message\n  ----     ------     ----               ----               -------\n......\ncalico-node\n  Warning  Unhealthy  43s (x2 over 44s)  kubelet            Readiness probe failed: calico\/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix \/var\/run\/calico\/bird.ctl: connect: connection refused<\/code><\/pre>\n<p>\uc704 \uc624\ub958\ub294 \uc11c\ubc84 \ub0b4\uc5d0 \uc5ec\ub7ec\uac1c\uc758 \ub124\ud2b8\uc6cc\ud06c\uac00 \uc124\uce58\ub41c \uc0c1\ud0dc\ub85c Calico \uac00 \uc798\ubabb\ub41c \ub124\ud2b8\uc6cc\ud06c\ub97c \uc120\ud0dd\ud588\uc744 \ub54c \ubc1c\uc0dd\ud558\ub294 \uc624\ub958\uc774\ub2e4.<\/p>\n<pre><code class=\"language-bash\">kubectl set env daemonset\/calico-node -n kube-system IP_AUTODETECTION_METHOD=interface=bond*<\/code><\/pre>\n<p>\uc704 \uba85\ub839\uc73c\ub85c \uc678\ubd80 \uc811\uc18d\uc6a9 \ub124\ud2b8\uc6cc\ud06c\ub97c \uc9c0\uc815\ud574 \uc8fc\uba74 \uc624\ub958\uac00 \uc5c6\uc5b4\uc9c4\ub2e4.<\/p>\n<p>\uc5b4\ub5a4 \ub124\ud2b8\uc6cc\ud06c\uac00 \uc678\ubd80 \uc811\uc18d\uc6a9 \ub124\ud2b8\uc6cc\ud06c\uc778\uc9c0 \ubaa8\ub978\ub2e4\uba74,<br \/>\n\ud68c\uc0ac \ub0b4\uc5d0 \ub124\ud2b8\uc6cc\ud06c \uad00\ub9ac\uc790\uac00 \ubcc4\ub3c4\ub85c \uc788\ub2e4\ub294 \uc758\ubbf8\uc774\ub2c8,<br \/>\n\uadf8 \uc0ac\ub78c\uc5d0\uac8c \ubb3c\uc5b4\ubcf4\uba74 \ub41c\ub2e4.<\/p>\n<p>Calico \uac00 \uc790\ub3d9\uc73c\ub85c Pod \ub97c \uc7ac\uc0dd\uc131\ud55c\ub2e4.<\/p>\n<pre><code class=\"language-bash\">k logs calico-node-kv8z8 -n kube-system\n......\nbird: Mesh_172_16_0_205: Connected to table master\nbird: Mesh_172_16_0_205: State changed to wait\nbird: Mesh_172_16_0_201: Connected to table master\nbird: Mesh_172_16_0_201: State changed to wait\nbird: Mesh_172_16_0_203: Connected to table master\nbird: Mesh_172_16_0_203: State changed to wait\nbird: Mesh_172_16_0_202: Connected to table master\nbird: Mesh_172_16_0_202: State changed to wait\nbird: Graceful restart done\nbird: Mesh_172_16_0_201: State changed to feed\nbird: Mesh_172_16_0_202: State changed to feed\nbird: Mesh_172_16_0_203: State changed to feed\nbird: Mesh_172_16_0_205: State changed to feed\nbird: Mesh_172_16_0_201: State changed to up\nbird: Mesh_172_16_0_202: State changed to up\nbird: Mesh_172_16_0_203: State changed to up\nbird: Mesh_172_16_0_205: State changed to up\n2023-01-10 05:42:50.004 [INFO][100] felix\/health.go 242: Overall health status changed newStatus=&amp;health.HealthReport{Live:true, Ready:true<\/code><\/pre>\n<pre><code class=\"language-bash\">curl -L https:\/\/github.com\/projectcalico\/calico\/releases\/download\/v3.24.5\/calicoctl-linux-amd64 -o calicoctl\nchmod 700 calicoctl\nsudo mv calicoctl \/usr\/bin\/<\/code><\/pre>\n<pre><code class=\"language-bash\">sudo calicoctl node status\nCalico process is running.\n\nIPv4 BGP status\n+--------------+-------------------+-------+----------+-------------+\n| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |\n+--------------+-------------------+-------+----------+-------------+\n| 172.16.0.202 | node-to-node mesh | up    | 05:42:26 | Established |\n| 172.16.0.203 | node-to-node mesh | up    | 05:42:25 | Established |\n| 172.16.0.204 | node-to-node mesh | up    | 05:42:47 | Established |\n| 172.16.0.205 | node-to-node mesh | up    | 05:42:38 | Established |\n+--------------+-------------------+-------+----------+-------------+\n\nIPv6 BGP status\nNo IPv6 peers found.<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Kubernetes &#8211; Calico Troubleshooting Calico \uc124\uce58\uc911 \uc624\ub958\uac00 \ubc1c\uc0dd\ud558\uc5ec \ud574\ub2f9 \uc624\ub958\ub97c \uc81c\uac70\ud558\ub294 \ubc29\ubc95\uc744 \uc815\ub9ac\ud569\ub2c8\ub2e4. \ubc29\ud654\ubcbd \ud574\uc81c \uac00\uc7a5 \ub9ce\uc774 \ubc1c\uc0dd\ud558\ub294 \uc624\ub958\ub294 \ubc29\ud654\ubcbd\uc774\ubbc0\ub85c \uc77c\ub2e8 \ubc29\ud654\ubcbd\uc740 \ud574\uc81c\ud574\uc900\ub2e4. \uc774\ud6c4\uc5d0 \uc624\ub958\uac00 \ud574\uacb0\ub418\uba74 \uadf8\ub54c \ud2b9\uc815 \ud3ec\ud2b8\ub9cc \uc624\ud508\ud558\ub294 \uc791\uc5c5\uc744 \ud574\uc900\ub2e4. \uc774\uc804 CNI \uc794\uc7ac \uc81c\uac70 k describe po calico-node-lqstp -n kube-system &#8230;&#8230; Events: Type Reason Age From Message &#8212;- &#8212;&#8212; &#8212;- &#8212;- &#8212;&#8212;- Warning\u2026 <span class=\"read-more\"><a href=\"https:\/\/www.skyer9.pe.kr\/wordpress\/?p=7416\">Read More &raquo;<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[],"class_list":["post-7416","post","type-post","status-publish","format-standard","hentry","category-kubernetes"],"_links":{"self":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/7416","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7416"}],"version-history":[{"count":6,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/7416\/revisions"}],"predecessor-version":[{"id":7510,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/7416\/revisions\/7510"}],"wp:attachment":[{"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7416"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7416"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.skyer9.pe.kr\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7416"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}