電子發燒友網>電子資料下載>電子資料>KubeEye Kubernetes集群自動巡檢工具

KubeEye Kubernetes集群自動巡檢工具

1854540 2022-05-10 | zip | 0.42 MB | 次下載 | 2積分

普通下載普通下載

資料介紹

授權協議 Apache

開發語言 Google Go

操作系統跨平臺

軟件類型開源軟件

所屬分類云計算、 PaaS系統/容器

軟件簡介

KubeEye 旨在發現 Kubernetes 上的各種問題，比如應用配置錯誤（使用Polaris）、集群組件不健康和節點問題（使用Node-Problem-Detector）。除了預定義的規則，它還支持自定義規則。

架構圖

KubeEye 通過調用Kubernetes API，通過常規匹配日志中的關鍵錯誤信息和容器語法的規則匹配來獲取集群診斷數據，詳見架構。

怎么使用

機器上安裝 KubeEye
- 從?Releases?中下載預構建的可執行文件。
- 或者你也可以從源代碼構建
```
git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye 
make install
```
[可選] 安裝 Node-problem-Detector
注意：這一行將在你的集群上安裝 npd，只有當你想要詳細的報告時才需要。
ke install npd
KubeEye 執行

root@node1:# ke diag
NODENAME        SEVERITY     HEARTBEATTIME               REASON              MESSAGE
node18          Fatal        2020-11-19T10:32:03+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node19          Fatal        2020-11-19T10:31:37+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node2           Fatal        2020-11-19T10:31:14+08:00   NodeStatusUnknown   Kubelet stopped posting node status.
node3           Fatal        2020-11-27T17:36:53+08:00   KubeletNotReady     Container runtime not ready: RuntimeReady=false reason:DockerDaemonNotReady message:docker: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

NAME            SEVERITY     TIME                        MESSAGE
scheduler       Fatal        2020-11-27T17:09:59+08:00   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0          Fatal        2020-11-27T17:56:37+08:00   Get https://192.168.13.8:2379/health: dial tcp 192.168.13.8:2379: connect: connection refused

NAMESPACE       SEVERITY     PODNAME                                          EVENTTIME                   REASON                MESSAGE
default         Warning      node3.164b53d23ea79fc7                           2020-11-27T17:37:34+08:00   ContainerGCFailed     rpc error: code = Unknown desc = Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
default         Warning      node3.164b553ca5740aae                           2020-11-27T18:03:31+08:00   FreeDiskSpaceFailed   failed to garbage collect required amount of images. Wanted to free 5399374233 bytes, but freed 416077545 bytes
default         Warning      nginx-b8ffcf679-q4n9v.16491643e6b68cd7           2020-11-27T17:09:24+08:00   Failed                Error: ImagePullBackOff
default         Warning      node3.164b5861e041a60e                           2020-11-27T19:01:09+08:00   SystemOOM             System OOM encountered, victim process: stress, pid: 16713
default         Warning      node3.164b58660f8d4590                           2020-11-27T19:01:27+08:00   OOMKilling            Out of memory: Kill process 16711 (stress) score 205 or sacrifice child Killed process 16711 (stress), UID 0, total-vm:826516kB, anon-rss:819296kB, file-rss:0kB, shmem-rss:0kB
insights-agent  Warning      workloads-1606467120.164b519ca8c67416            2020-11-27T16:57:05+08:00   DeadlineExceeded      Job was active longer than specified deadline
kube-system     Warning      calico-node-zvl9t.164b3dc50580845d               2020-11-27T17:09:35+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system     Warning      kube-proxy-4bnn7.164b3dc4f4c4125d                2020-11-27T17:09:09+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29
kube-system     Warning      nodelocaldns-2zbhh.164b3dc4f42d358b              2020-11-27T17:09:14+08:00   DNSConfigForming      Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 100.64.11.3 114.114.114.114 119.29.29.29


NAMESPACE       SEVERITY     NAME                      KIND         TIME                        MESSAGE
kube-system     Warning      node-problem-detector     DaemonSet    2020-11-27T17:09:59+08:00   [livenessProbeMissing runAsPrivileged]
kube-system     Warning      calico-node               DaemonSet    2020-11-27T17:09:59+08:00   [runAsPrivileged cpuLimitsMissing]
kube-system     Warning      nodelocaldns              DaemonSet    2020-11-27T17:09:59+08:00   [cpuLimitsMissing runAsPrivileged]
default         Warning      nginx                     Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing tagNotSpecified]
insights-agent  Warning      workloads                 CronJob      2020-11-27T17:09:59+08:00   [livenessProbeMissing]
insights-agent  Warning      cronjob-executor          Job          2020-11-27T17:09:59+08:00   [livenessProbeMissing]
kube-system     Warning      calico-kube-controllers   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system     Warning      coredns                   Deployment   2020-11-27T17:09:59+08:00   [cpuLimitsMissing]

您可以參考常見FAQ內容來優化您的集群。

KubeEye 能做什么

KubeEye 可以發現你的集群控制平面的問題，包括 kube-apiserver/kube-controller-manager/etcd 等。
KubeEye 可以幫助你檢測各種節點問題，包括內存/CPU/磁盤壓力，意外的內核錯誤日志等。
KubeEye 根據行業最佳實踐驗證你的工作負載 yaml 規范，幫助你使你的集群穩定。

檢查項

是/否	檢查項	描述
	ETCDHealthStatus	如果 etcd 啟動并正常運行
	ControllerManagerHealthStatus	如果 kubernetes kube-controller-manager 正常啟動并運行
	SchedulerHealthStatus	如果 kubernetes kube-schedule 正常啟動并運行
	NodeMemory	如果節點內存使用量超過閾值
	DockerHealthStatus	如果 docker 正常運行
	NodeDisk	如果節點磁盤使用量超過閾值
	KubeletHealthStatus	如果 kubelet 激活狀態且正常運行
	NodeCPU	如果節點 CPU 使用量超過閾值
	NodeCorruptOverlay2	Overlay2 不可用
	NodeKernelNULLPointer	node 顯示 NotReady
	NodeDeadlock	死鎖是指兩個或兩個以上的進程在爭奪資源時互相等待的現象。
	NodeOOM	監控那些消耗過多內存的進程，尤其是那些消耗大量內存非?？斓倪M程，內核會殺掉它們，防止它們耗盡內存
	NodeExt4Error	Ext4 掛載失敗
	NodeTaskHung	檢查D狀態下是否有超過 120s 的進程
	NodeUnregisterNetDevice	檢查對應網絡
	NodeCorruptDockerImage	檢查 docker 鏡像
	NodeAUFSUmountHung	檢查存儲
	NodeDockerHung	Docker hang住, 檢查 docker 的日志
	PodSetLivenessProbe	如果為pod中的每一個容器設置了 livenessProbe
	PodSetTagNotSpecified	鏡像地址沒有聲明標簽或標簽是最新
	PodSetRunAsPrivileged	以特權模式運行 Pod 意味著 Pod 可以訪問主機的資源和內核功能
	PodSetImagePullBackOff	Pod 無法正確拉出鏡像，因此可以在相應節點上手動拉出鏡像
	PodSetImageRegistry	檢查鏡像形式是否在相應倉庫
	PodSetCpuLimitsMissing	未聲明 CPU 資源限制
	PodNoSuchFileOrDirectory	進入容器查看相應文件是否存在
	PodIOError	這通常是由于文件 IO 性能瓶頸
	PodNoSuchDeviceOrAddress	檢查對應網絡
	PodInvalidArgument	檢查對應存儲
	PodDeviceOrResourceBusy	檢查對應的目錄和 PID
	PodFileExists	檢查現有文件
	PodTooManyOpenFiles	程序打開的文件/套接字連接數超過系統設置值
	PodNoSpaceLeftOnDevice	檢查磁盤和索引節點的使用情況
	NodeApiServerExpiredPeriod	將檢查 ApiServer 證書的到期日期少于30天
	PodSetCpuRequestsMissing	未聲明 CPU 資源請求值
	PodSetHostIPCSet	設置主機 IP
	PodSetHostNetworkSet	設置主機網絡
	PodHostPIDSet	設置主機 PID
	PodMemoryRequestsMiss	沒有聲明內存資源請求值
	PodSetHostPort	設置主機端口
	PodSetMemoryLimitsMissing	沒有聲明內存資源限制值
	PodNotReadOnlyRootFiles	文件系統未設置為只讀
	PodSetPullPolicyNotAlways	鏡像拉策略并非總是如此
	PodSetRunAsRootAllowed	以 root 用戶執行
	PodDangerousCapabilities	您在 ALL / SYS_ADMIN / NET_ADMIN 等功能中有危險的選擇
	PodlivenessProbeMissing	未聲明 ReadinessProbe
	privilegeEscalationAllowed	允許特權升級
?	NodeNotReadyAndUseOfClosedNetworkConnection	http 2-max-streams-per-connection
?	NodeNotReady	無法啟動 ContainerManager 無法設置屬性 TasksAccounting 或未知屬性

未標注的項目正在開發中

添加自定義檢查規則

添加 npd 自定義檢查規則

安裝 NPD 指令?ke install npd
由 kubectl 編輯 configmap kube-system/node-problem-detector-config,

kubectl edit cm -n kube-system node-problem-detector-config

在 configMap 的規則下添加異常日志信息，規則遵循正則表達式。

自定義最佳實踐規則

準備一個規則 yaml，例如，下面的規則將驗證你的 pod 規范，以確保鏡像只來自授權的注冊處。

checks:
  imageFromUnauthorizedRegistry: warning

customChecks:
  imageFromUnauthorizedRegistry:
    promptMessage: When the corresponding rule does not match. Show that image from an unauthorized registry.
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          not:
            pattern: ^quay.io

將上述規則保存為yaml，例如，rule.yaml。
用 rule.yaml 運行 KubeEye。

root:# ke diag -f rule.yaml --kubeconfig ~/.kube/config
NAMESPACE     SEVERITY    NAME                      KIND         TIME                        MESSAGE
default       Warning     nginx                     Deployment   2020-11-27T17:18:31+08:00   [imageFromUnauthorizedRegistry]
kube-system   Warning     node-problem-detector     DaemonSet    2020-11-27T17:18:31+08:00   [livenessProbeMissing runAsPrivileged]
kube-system   Warning     calico-node               DaemonSet    2020-11-27T17:18:31+08:00   [cpuLimitsMissing runAsPrivileged]
kube-system   Warning     calico-kube-controllers   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing livenessProbeMissing]
kube-system   Warning     nodelocaldns              DaemonSet    2020-11-27T17:18:31+08:00   [runAsPrivileged cpuLimitsMissing]
default       Warning     nginx                     Deployment   2020-11-27T17:18:31+08:00   [livenessProbeMissing cpuLimitsMissing]
kube-system   Warning     coredns                   Deployment   2020-11-27T17:18:31+08:00   [cpuLimitsMissing]