nginx-ingress-controller日志持久化方案的解决
最近看到一篇公众号讲了nginx-ingress-controller的应用。下面有人评论如何做日志持久化，刚好工作上遇到该问题，整理一个方案，仅供参考。
nginx-ingress-controller的日志
nginx-ingress-controller的日志包括三个部分：
- controller日志： 输出到stdout，通过启动参数中的Clog_dir可已配置输出到文件，重定向到文件后会自动轮转，但不会自动清理
- accesslog：输出到stdout，通过nginx-configuration中的字段可以配置输出到哪个文件。输出到文件后不会自动轮转或清理
- errorlog：输出到stderr，配置方式与accesslog类似。
给controller日志落盘
- 给nginx-ingress-controller挂一个hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
- 给nginx-ingress-controller配置log-dir和logtostderr参数，将日志重定向到/var/log/nginx_ingress_controller/中。
controller的日志需要做定时清理。由于controller的日志是通过klog(k8s.io/klog)输出的，会进行日志滚动，所以我们通过脚本定时清理一定时间之前的日志文件即可。
给nginx日志落盘
修改configmap： nginx-configuration。配置accesslog和errorlog的输出路径，替换默认的stdout和stderr。输出路径我们可以与controller一致，便于查找。
accesslog和errorlog都只有一个日志文件，我们可以使用logrotate进行日志轮转，将输出到宿主机上的日志进行轮转和清理。配置如：
$ cat /etc/logrotate.d/nginx.log /data/log/nginx_ingress_controller/access.log { su root list rotate 7 daily maxsize 50M copytruncate missingok create 0644 www-data root }
官方提供的模板中，nginx-ingress-controller默认都是以33这个用户登录启动容器的，因此挂载hostpath路径时存在权限问题。我们需要手动在机器上执行chown -R 33:33 /data/log/nginx_ingress_controller.
自动化ops
nginx日志落盘中，第2、3两点均需要人工运维，有什么解决办法吗？
问题的关键是：有什么办法可以在nginx-ingress-controller容器启动之前加一个hook，将宿主机的指定目录执行chown呢？
可以用initContainer。initcontainer必须在containers中的容器运行前运行完毕并成功退出。利用这一k8s特性，我们开发一个docker image，里面只执行如下脚本：
#!/bin/bash logdir=$LOG_DIR userID=$USER_ID echo "try to set dir: $logdir 's group as $userID" chown -R $userID:$userID $logdir
脚本读取一些环境变量， 确认需要修改哪个目录，改成怎样的user group。
将脚本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作为initcontainers。 注意要对该initcontainer配置环境变量和volumeMount.
再说第二点，我们注意到nginx-ingress-controller的基础镜像中就自带了logrotate，那么问题就简单了，我们将写好的logrotate配置文件以configmap的形式挂载到容器中就可以了。
一个deploy yaml如下：
--- apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: kube-system spec: type: ClusterIP ports: - name: http port: 80 targetPort: 80 protocol: TCP - name: https port: 443 targetPort: 443 protocol: TCP selector: app: ingress-nginx --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: kube-system labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: default namespace: kube-system spec: backend: serviceName: default-http-backend servicePort: 80 --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: kube-system labels: app: ingress-nginx data: use-forwarded-headers: "true" # 此处配置nginx日志的重定向目标 access-log-path: /var/log/nginx_ingress_controller/access.log error-log-path: /var/log/nginx_ingress_controller/error.log --- # 创建一个configmap，配置nginx日志的轮转策略，对应的是nginx日志在容器内的日志文件 apiVersion: v1 data: nginx.log: | {{ user_nginx_log.host_path }}/access.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root } {{ user_nginx_log.host_path }}/error.log { rotate {{ user_nginx_log.rotate_count }} daily maxsize {{ user_nginx_log.rotate_size }} minsize 10M copytruncate missingok create 0644 root root } kind: ConfigMap metadata: name: nginx-ingress-logrotate namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: kube-system rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: kube-system --- apiVersion: apps/v1 kind: DaemonSet metadata: name: ingress-nginx namespace: kube-system spec: selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" dnsPolicy: ClusterFirstWithHostNet hostNetwork: true # 配置initcontainer，确保在nginx-ingress-controller容器启动前将日志目录的权限配置好 initContainers: - name: adddirperm image: "{{ image_registry.addr }}/{{ image.adddirperm }}" env: - name: LOG_DIR value: /var/log/nginx_ingress_controller - name: USER_ID value: "33" volumeMounts: - name: logdir mountPath: /var/log/nginx_ingress_controller containers: - name: nginx-ingress-controller image: "{{ image_registry.addr }}/{{ image.ingress }}" imagePullPolicy: IfNotPresent args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io # 设置controller日志的输出路径和方式 - --log_dir=/var/log/nginx_ingress_controller - --logtostderr=false securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 resources: requests: cpu: 100m memory: 256Mi livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: # 配置挂载容器中控制器组件和nginx的日志输出路径 - name: logdir mountPath: /var/log/nginx_ingress_controller # 配置nginx日志的logrotate配置挂载路径 - name: logrotateconf mountPath: /etc/logrotate.d/nginx.log subPath: nginx.log volumes: # 控制器组件和nginx的日志输出路径为宿主机的hostpath - name: logdir hostPath: path: {{ user_nginx_log.host_path }} type: "" # nginx日志的轮转配置文件来自于configmap - name: logrotateconf configMap: name: nginx-ingress-logrotate items: - key: nginx.log path: nginx.log --- apiVersion: apps/v1 kind: DaemonSet metadata: name: default-http-backend namespace: kube-system labels: app: default-http-backend spec: selector: matchLabels: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 tolerations: - key: dedicated value: ingress-nginx effect: NoSchedule affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "system/ingress" operator: In values: - "true" containers: - name: default-http-backend # Any image is permissible as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: "{{ image_registry.addr }}/{{ image.http_backend }}" imagePullPolicy: IfNotPresent livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi ---
最后，有的人建议将initcontainer去掉，改为基于原有的nginx-ingress-controller镜像加一层layer，将配置路径权限的脚本放在该层执行。 个人认为这种方法既不美观，也不方便。唯一的好处仅在于deploy yaml仍然简洁（但少不了volumeMount之类的配置）。不过还是看个人使用感受吧~
到此这篇关于nginx-ingress-controller日志持久化方案的解决的文章就介绍到这了,更多相关nginx ingress controller日志持久化内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！
「点点赞赏，手留余香」
还没有人赞赏，快来当第一个赞赏的人吧！
除特别注明外，本站所有文章均为无双技术网原创，转载请注明出处来自https://www.bnskd.com/9309.html
