搜索文章 Ctrl K

3-15-2-ingress资源

熊 0 阅读 容器

1. Ingress资源基础

我们知道在k8s上service是用来解决Pod访问问题 , 它是通过kube-proxy在每个节点上创建iptables规则或ipvs规则 , 在用户请求某个pod时 , 用户的请求会被其service规则所捕获 , 从而实现访问对应pod。对于service来讲 , 用户请求直接在传输层就被捕获转发 , 效率很高效 , 但这同时也引入了一个新问题 ; 比如我们运行的pod对外客户端访问需要https通信 , 如果使用service这种4层调度 , 那就意味着每个pod上我们要配置证书 , 这很显然不是我们想要做的 ; 那有没有什么办法做到在用户访问pod对应的service时使用https , 而对应pod里又不用https协议呢？答案是有的 。

比如我们可以使用nginx来做https会话卸载器 ; 我们只需要在代理上配置证书即可 ; 又比如我们在k8s上运行了各种各样的pod , 这些pod的功能每个都不一样 , 有的是专门处理用户认证的 , 有的是专门处理站点主页的 , 有的专门处理支付的等等 , 而这些pod对外都是提供一个独有的url , 那么这些pod需要怎么才能被集群外部访问到呢？我们知道对于一个站点来讲 , 如果后端有多个server同时提供一种服务 , 我们可以把这些同功能的server定义成一个组 , 然后使用nginx代理将不同功能url的访问代理到不同组上即可 。 这样一来就解决了后端多server被负载访问的问题 。

那么对于k8s上这种同功能的pod怎么归并成一个组呢？用户访问不同url怎么调度到不同的组上呢？很显然要想实现这些功能 , 在k8s上应该有一个类似nginx一样的代理存在 , 这个代理就叫做ingress 控制器 。 ingress 控制器和k8s上的其他控制不一样 , ingress控制器并不能直接运行为kube-controller-manager的一部分 , 它类似k8s集群上的coredns , 需要在集群上单独部署 , 本质上就是一个pod , 我们可以使用k8s上的ds或deploy控制器来创建它 。

ingress controller pod的作用主要是引入集群外部流量 , 并实时监控着apiserver上ingress资源的变动 , 并将其ingress中定义的规则转化为对应ingress控制器对应应用程序的专有配置 , 然后动态的重载或重启对应守护进程来使其配置文件生效 。 在k8s上ingress是一种标准资源 , 它本质上就是我们定义的基于dns名称 ( host ) 或url路径把请求转发至指定service资源的规则 ; 简单讲ingress就是我们用来定义代理的配置所创建的资源 ; ingress控制器就是把对应ingress规则转换为对应ingress控制器中应用程序的专有配置 , 然后重启或重载对应配置文件使其生效的组件 ;

ingress和ingress controller pod的关系

kubernetes-service-ingress-1

如上图所示 , ingress就是ingress 控制器pod的代理规则 ; 用户请求某个后端pod所提供的服务时 , 首先会通过ingress controller pod把流量引入到集群内部 , 然后ingress controller pod根据ingress定义的规则 , 把对应ingress规则转化为对应ingress controller pod实现的对应应用的配置 ( ingress controller 可以由任何具有七层反向代理功能的服务实现 , 比如nginx,haproxy等等 ) 然后再适配用户请求 , 把对应请求反代到对应service上 ; 而对于pod的选择上 , ingress控制器可以基于对应service中的标签选择器 , 直接同pod直接通信 , 无须通过service对象api的再次转发 , 从而省去了用户请求到kube-proxy实现的代理开销 ( 本质上ingress controller 也是运行为一个pod , 和其他pod在同一网段中 ) ;

2. Ingress 部署

2.1 ingress controller部署

在k8s上ingress controller的实现有很多 , 比如基于nginx的 , 基于haproxy的等等 , 这里以nginx为例 ;

  1. 下载ingress-nginx包

    wget https://github.com/kubernetes/ingress-nginx/archive/nginx-0.28.0.tar.gz

  2. 解压包 , 找到对应的部署清单

    [root@master01 ~]# ll
total 92144
-rw------- 1 root root 65586688 Dec  8 15:16 flannel-v0.13.1-rc1.tar
drwxr-xr-x 2 root root     4096 Dec 21 21:04 manifests
-rw-r--r-- 1 root root 28760559 Dec 21 21:02 nginx-0.28.0.tar.gz
[root@master01 ~]# tar xf nginx-0.28.0.tar.gz
[root@master01 ~]# ls
flannel-v0.13.1-rc1.tar  ingress-nginx-nginx-0.28.0  manifests  nginx-0.28.0.tar.gz
[root@master01 ~]# cd ingress-nginx-nginx-0.28.0/
[root@master01 ingress-nginx-nginx-0.28.0]# ls
build         code-of-conduct.md  docs    hack      labels.yaml  mkdocs.yml      README.md              SECURITY_CONTACTS  version
Changelog.md  CONTRIBUTING.md     go.mod  images    LICENSE      OWNERS          requirements-docs.txt  test
cmd           deploy              go.sum  internal  Makefile     OWNERS_ALIASES  rootfs                 vendor
[root@master01 ingress-nginx-nginx-0.28.0]# cd deploy/
[root@master01 deploy]# ls
aws        cloud-generic  grafana   prometheus  static                       with-validating-webhook.yaml.tpl
baremetal  cluster-wide   minikube  README.md   validating-webhook.yaml.tpl
[root@master01 deploy]# cd static/
[root@master01 static]# ls
configmap.yaml  mandatory.yaml  namespace.yaml  provider  rbac.yaml  with-rbac.yaml
[root@master01 static]# pwd
/root/ingress-nginx-nginx-0.28.0/deploy/static
[root@master01 static]#

　　资源配置清单在ingress-nginx-nginx-0.28.0/deploy/static下 , 名为mandatory.yaml ;

  1. 资源配置清单内容

    apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
​
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
​
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
​
---
​
apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

    以上清单主要定义了一个名称ingress-nginx的名称空间 , 在其名称空间下创建了几个configmap , 最重要的是用deployment创建了一个ingress-nginx pod ;

这里说一下 , 对于ingress-nginx控制器 , 它本质还是运行为一个pod , 对于pod来说要想接入外部访问流量到集群内部来 , 有三种方式 , 一种是使用NodePort类型的service ; 第二种是使用ds或deploy控制器 , 在定义pod模板时使用hostPort把pod端口映射到宿主机方式 ; 第三种是定义pod模板时使用hostNetwork , 直接共享宿主机网络名称空间 ; 如下所示

使用专有NodePort service来引入外部流量

kubernetes-service-ingress-2

这种使用deploy控制管理ingress controller pod , 如果在pod模板中没有暴露端口 , 则需要创建一个service资源来暴露ingress controller pod的端口来引入外部流量到集群内部 ;

使用ds控制器管理ingress controller pod在pod模板中使用hostPort方式暴露端口

kubernetes-service-ingress-3

使用ds控制器能够保证每个节点上只运行一个ingress controller,所以我们可以把对应ingress controller pod端端口通过端口映射的方式映射到宿主机上的某一固定端口 ;

使用ds控制器在pod模板中使用hostNetwork方式共享宿主机网络名称空间

kubernetes-service-ingress-4

共享宿主机网络名称空间 , 也必须使用ds控制器来确保对应每个节点上只能运行一个ingress controller pod , 这样才能确保每个ingress controller pod能够正常把端口暴露出去 , 以供集群外部客户端访问 ;

选择上述其中一种方式暴露ingress controller pod的端口即可 ; 如果选择使用ds控制器来暴露端口 , 我们就需要修改其对应资源配置清单中的pod模板 , 如下所示

使用ds控制器来管理ingress controller pod在pod模板中使用hostPort方式暴露端口

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
​
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
​
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              hostPort: 30080
              protocol: TCP
            - name: https
              containerPort: 443
              hostPort: 30443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
​
---
​
apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

只需把对应控制器类型更改为DaemonSet , 在pod模板中spec字段下把replicas去掉 ; 在spec.template.spec.containers.ports字段中加上nodePort字段指定要把容器的端口映射到宿主机上某个端口 ; 如果暴露的端口是非标准端口 , 在对应k8s集群外部我们还需要部署反代 , 比如使用nginx , haproxy,lvs ;

使用ds控制器管理ingress controller pod在ds控制器资源配置中使用hostNetwork

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
​
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
​
---
​
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.28.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
​
---
​
apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - default:
    min:
      memory: 90Mi
      cpu: 100m
    type: Container

把对应控制器类型更改外DaemonSet , 在pod模板中spec字段下的replicas字段去掉 ; 在spec.template.spec字段下加上hostNetwork: true即可 ; 以上两种使用ds控制器管理ingress controller pod也可以使用node选择器 , 来筛选在某个节点上创建ingress controller pod ;

使用deploy控制器管理ingress controller pod , 就直接应用mandatory.yaml即可

[root@master01 ~]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/nginx-ingress-role created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
limitrange/ingress-nginx created
[root@master01 ~]#

查看应用资源清单创建的资源对象

[root@master01 ~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          80s
 
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-ingress-controller   1/1     1            1           80s
 
NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-ingress-controller-5466cb8999   1         1         1       80s
[root@master01 ~]#

可以看到在ingress-nginx名称空间下创建了一个deploy控制器 , 对应控制器创建了一个nginx-ingress-controller控制器pod ; 但是此pod现在不能被外部客户端访问到 , 我们需要创建一个service来引入外部流量到此pod上 ;

查看pod标签

[root@master01 ~]# kubectl get pod -n ingress-nginx --show-labels
NAME                                        READY   STATUS    RESTARTS   AGE     LABELS
nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          4m38s   app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx,pod-template-hash=5466cb8999
[root@master01 ~]#

根据上述标签来写一个创建ingress-service资源的配置清单

[root@master01 ~]# cat ingress-nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-svc
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - port: 80
      name: http
      nodePort: 30080
    - port: 443
      name: https
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
[root@master01 ~]#

以上配置清单主要把满足对应标签选择器的pod关联起来 ; 并把对应pod的80和443端口分别映射到对应主机上的30080和30443端口 ;

应用配置清单

[root@master01 ~]# kubectl apply -f ingress-nginx-service.yaml
service/ingress-nginx-svc created
[root@master01 ~]# kubectl get svc -n ingress-nginx
NAME                TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-svc   NodePort   10.98.4.208   <none>        80:30080/TCP,443:30443/TCP   13s
[root@master01 ~]#

访问集群任意节点ip的30080和30443端口 , 看看是否访问到对应pod？

kubernetes-service-ingress-5

30080是能够正常访问的 , 只是它显示404 , 是因为我们没有对应的主页 ;

访问30443端口

kubernetes-service-ingress-6
kubernetes-service-ingress-7
kubernetes-service-ingress-8

30443是一个https端口 , 所以访问必须用https协议访问 , 这里提示访问页面有风险是因为浏览器不信任证书引起的 , 我们可以点击高级 , 信任证书即可 ; 同样30443端口也是返回404 , 是因为没有主页的原因 ; 两个端口能够正常访问 , 说明我们在k8s上部署的ingress-nginx controller就部署好了 ;

2.2 ingress资源的使用

在k8s上创建一个deploy控制器 , 让其管理2个 ikubernetes/myapp:v1镜像运行的pod , 然后再创建一个对应的service

[root@master01 manifests]# cat myapp-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      rel: stable
  template:
    metadata:
      namespace: default
      labels:
        app: myapp
        rel: stable
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1
 
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: default
spec:
  selector:
    app: myapp
    rel: stable
  ports:
  - name: http
    port: 80
    targetPort: 80
[root@master01 manifests]#

一个清单中定义多个资源 , 需要用“---”来分割资源 ;

应用资源清单

[root@master01 manifests]# kubectl apply -f myapp-demo.yaml
deployment.apps/myapp created
service/myapp created
[root@master01 manifests]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE             NOMINATED NODE   READINESS GATES
myapp-6479b786f5-9d4mh   1/1     Running   0          11s   10.244.2.98   node02.k8s.org   <none>           <none>
myapp-6479b786f5-k252c   1/1     Running   0          11s   10.244.4.20   node04.k8s.org   <none>           <none>
[root@master01 manifests]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   4h52m
myapp        ClusterIP   10.105.208.218   <none>        80/TCP    21s
[root@master01 manifests]# kubectl describe svc myapp
Name:              myapp
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=myapp,rel=stable
Type:              ClusterIP
IP Families:       <none>
IP:                10.105.208.218
IPs:               10.105.208.218
Port:              http  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.98:80,10.244.4.20:80
Session Affinity:  None
Events:            <none>
[root@master01 manifests]#

创建ingress资源来反代以上资源

示例: 创建ingress资源

[root@master01 manifests]# cat ingress-myapp.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: /
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

创建ingress资源apiVersion的值要写成extensions/v1beta1 ,

  • kind为Ingress ;

  • metadata中的annotations的配置表示把ingress资源通知给那个类别的ingress controller , 如果k8s集群上有多个类别的ingress controller时 , 这一项特别有用

  • 在spec字段主要内嵌了三个字段

    rules字段用来定义反代规则列表 , 其值为一个对象列表 ; 其中rules字段里主要host和http字段 ;

    • host用来指定虚拟主机的fqdn名称 , 如果不写表示匹配任意虚拟主机名称

    • http是用来定义指向后端的http选择器列表 ; 其值为一个对象 , 里面只有一个paths字段 , 用于指定把请求映射到后端的某个路径 ; 其值为一个对象列表

      • paths字段中可以定义path , 用来指定映射后端的路径

      • backend用于指定后端pod的service , 其值为一个对象 ; serviceName用于指定对应pod的service名称 ; servicePort用于指定后端服务的端口

    • 以上配置表示把www.myapp.com这个虚拟主机的访问全部反代至服务名称为myapp端口为80的pod上 ;

应用配置清单

[root@master01 manifests]# kubectl apply -f ingress-myapp.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS           ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com             80      29s
[root@master01 manifests]#

查看ingress资源的详细信息

[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:         
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com 
                 /   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  81s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

可以看到对应满足service名称为myapp并且其端口为80的pod有两个 ;

进入ingress controller pod里 , 看看对应配置文件是否有www.myapp.com的配置

[root@master01 manifests]# kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-5466cb8999-4lsjc   1/1     Running   0          78m
[root@master01 manifests]# kubectl exec -it -n ingress-nginx pod/nginx-ingress-controller-5466cb8999-4lsjc -- /bin/sh
/etc/nginx $ cd /etc/nginx/
/etc/nginx $ ls
fastcgi.conf            koi-utf                 modsecurity             owasp-modsecurity-crs   uwsgi_params.default
fastcgi.conf.default    koi-win                 modules                 scgi_params             win-utf
fastcgi_params          lua                     nginx.conf              scgi_params.default
fastcgi_params.default  mime.types              nginx.conf.default      template
geoip                   mime.types.default      opentracing.json        uwsgi_params
/etc/nginx $ grep "www.myapp.com" nginx.conf
        ## start server www.myapp.com
                server_name www.myapp.com ;
        ## end server www.myapp.com
/etc/nginx $

可以看到在对应ingress-nginx 控制器pod中能够搜索到www.myapp.com的配置 ; 说明我们定义的ingress资源已经被ingress-nginx controller 捕获 ;

用浏览器访问www.myapp.com看看是否能够访问到内容？

kubernetes-service-ingress-9

使用www.myapp.com访问 , 需要确保对应域名能够正常解析到k8s集群任意一节点上 ; 可以看到访问www.myapp.com:30080能够访问到对应pod内容 ;

删除ingress代理规则

[root@master01 manifests]# kubectl delete -f ingress-myapp.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions "ingress-myapp" deleted
[root@master01 manifests]# kubectl get ingress
No resources found in default namespace.
[root@master01 manifests]#

2.3 基于url路径进行流量分发

示例: 配置基于url路径进行流量分发

[root@master01 manifests]# cat ingress-myapp1.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: /bbs
        backend:
          serviceName: myapp
          servicePort: 80
      - path: /blog
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

以上配置表示把www.myapp.com/bbs反代到service名称为myapp并且端口为80的pod上 ; 把www.myapp.com/blog反代到service名称为myapp并且端口为80的pod上 ; 我这里是因为k8s上只有这一种应用 , 生成环境中按照对应的业务逻辑来反代对应url到对应pod上即可 ;

应用配置清单

[root@master01 manifests]# kubectl apply -f ingress-myapp1.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS           ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com             80      5s
[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:         
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com 
                 /bbs    myapp:80 (10.244.2.98:80,10.244.4.20:80)
                 /blog   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     ingress.kubernetes.io/rewrite-target: /
                 kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  30s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

可以看到对应ingress上就有两个url分别指向后端service名称为myapp端口为80的pod上 ;

访问对应url , 看看是否访问到内容？

kubernetes-service-ingress-10

这里访问不到内容的原因是对应pod内部并没有对应url的页面 ;

进入ingress controller pod内部 , 查看是否有对应配置？

kubernetes-service-ingress-11
kubernetes-service-ingress-12

可以看到对应在ingress中定义的配置 , 都转为对应该ingress controller pod中的配置 , 说明我们定义基于url分发流量的ingress没有问题 ;

示例: 定义ingress规则基于主机名称的虚拟主机

[root@master01 manifests]# cat ingress-myapp2.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path:
        backend:
          serviceName: myapp
          servicePort: 80
  - host: blog.myapp.com
    http:
      paths:
      - path:
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

以上配置表示把www.myapp.com这个虚拟主机名称的访问流量分发至service名称为myapp端口为80的pod上 ; 把blog.myapp.com的流量分发至至service名称为myapp端口为80的pod上 ; 生成环境按照对应的service名称来分发即可 ;

应用配置清单

[root@master01 manifests]# kubectl apply -f ingress-myapp2.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp created
[root@master01 manifests]# kubectl get ingress
NAME            CLASS    HOSTS                          ADDRESS   PORTS   AGE
ingress-myapp   <none>   www.myapp.com,blog.myapp.com             80      16s
[root@master01 manifests]# kubectl describe ingress ingress-myapp
Name:             ingress-myapp
Namespace:        default
Address:         
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  www.myapp.com  
                     myapp:80 (10.244.2.98:80,10.244.4.20:80)
  blog.myapp.com 
                     myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:      ingress.kubernetes.io/rewrite-target: /
                  kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  32s   nginx-ingress-controller  Ingress default/ingress-myapp
[root@master01 manifests]#

验证配置信息

kubernetes-service-ingress-13
kubernetes-service-ingress-14

访问对应虚拟主机 , 看看是否能够访问对应pod？

kubernetes-service-ingress-15

可以看到两个虚拟主机名称都可以正常访问到 , 对应也做了调度 ;

2.4 创建tls类型的ingress资源

示例: 创建tls类型的ingress资源

创建证书

[root@master01 manifests]# openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
.........................................+++
........+++
e is 65537 (0x10001)
[root@master01 manifests]# openssl req -x509 -key tls.key -out tls.crt -subj /C=CN/ST=SiChuan/L=GuangYuan/O=Test/CN=www.myapp.com -days 3650 
[root@master01 manifests]#

以上两条命令创建了一个名为tls.key的私钥和一个自签名证书 , 其名为tls.crt ;

创建Secret资源

[root@master01 manifests]# kubectl create secret tls www-myapp-com-ingress-secret --cert=./tls.crt --key=./tls.key
secret/www-myapp-com-ingress-secret created
[root@master01 manifests]# kubectl get secret
NAME                           TYPE                                  DATA   AGE
default-token-xvd4c            kubernetes.io/service-account-token   3      13d
www-myapp-com-ingress-secret   kubernetes.io/tls                     2      21s
[root@master01 manifests]#

在ingress控制器上配置https主机时 , 不能直接使用私钥和证书文件 , 而是需要使用secret资源对象来传递相关数据 ;

定义tls类型ingress资源清单

[root@master01 manifests]# cat www-myapp-com-ingress-secret.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - www.myapp.com
    secretName: www-myapp-com-ingress-secret
  rules:
  - host: www.myapp.com
    http:
      paths:
      - path: /
        backend:
          serviceName: myapp
          servicePort: 80
[root@master01 manifests]#

定义tls类型ingress资源清单 , 需要在spec字段下用tls字段来指定对应主机名称 , 以及secret资源对象的名称 ;

应用资源清单

[root@master01 manifests]# kubectl apply -f www-myapp-com-ingress-secret.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/ingress-myapp-tls created
[root@master01 manifests]# kubectl get ingress
NAME                CLASS    HOSTS                          ADDRESS   PORTS     AGE
ingress-myapp       <none>   www.myapp.com,blog.myapp.com             80        31m
ingress-myapp-tls   <none>   www.myapp.com                            80, 443   8s
[root@master01 manifests]# kubectl describe ingress ingress-myapp-tls
Name:             ingress-myapp-tls
Namespace:        default
Address:         
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  www-myapp-com-ingress-secret terminates www.myapp.com
Rules:
  Host           Path  Backends
  ----           ----  --------
  www.myapp.com 
                 /   myapp:80 (10.244.2.98:80,10.244.4.20:80)
Annotations:     kubernetes.io/ingress.class: nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  26s   nginx-ingress-controller  Ingress default/ingress-myapp-tls
[root@master01 manifests]#

验证: 访问对应虚拟主机名称 , 看看对应的https端口是否能够正常访问到内容？

kubernetes-service-ingress-16

可以看到使用https协议访问对应的30443端口能够正常访问到对应后端pod提供的内容 ;

参考链接

容器编排系统k8s之Ingress资源 - Linux-1874 - 博客园 (cnblogs.com)