搜索文章 Ctrl K

6-1-k8s实际应用-nginx+tomcat+nfs实现动静分离web站点

熊 0 阅读 容器


1. 架构基础

基于Nginx+Tomcat+NFS实现通过域名转发动态请求到Tomcat Pod的动静分离架构 , 要求能够通过负载均衡的VIP访问到k8s集群中运行的Nginx+Tomcat+NFS中的web页面。 基于NFS实现动静分离：图片的上传由后端服务器tomcat完成 , 图片的读取由前端的nginx响应 , 就需要nginx与tomcat的数据保持一致性 , 因此需要将数据保存到k8s环境外部的存储服务器 , 然后再挂载到各nginx与tomcat 的容器中进行相应的操作。(nginx和tomcat的yaml定义的挂载指定同一个NFS)

1.1 业务镜像设计规划

kubernetes-project-web-1

根据业务的不同 , 我们可以导入官方基础镜像 , 在官方基础镜像的基础上自定义需要用的工具和环境 , 然后构建成自定义出自定义基础镜像 , 后续再基于自定义基础镜像 , 来构建不同服务的基础镜像 , 最后基于服务的自定义基础镜像构建出对应业务镜像 ; 最后将这些镜像上传至本地harbor仓库 , 然后通过k8s配置清单 , 将对应业务运行至k8s集群之上 ;

1.2 Nginx+Tomcat+NFS实现动静分离架构图

kubernetes-project-web-2

客户端通过负载均衡器的反向代理来访问k8s上的服务 , nginx pod和tomcat pod 由k8s svc 资源进行关联 ; 所有数据 ( 静态资源和动态资源) 通过存储挂载至对应pod中 ; nginx作为服务入口 , 它负责接收客户端的请求 , 同时响应静态资源 ( 到存储上读取 , 比如js文件 , css文件 , 图片等) ; 后端动态资源 , 由nginx将请求转发至后端tomcat server 完成 ( tomcat负责数据写入 , 比如用户的上传的图片等等) ;

2. 镜像准备

2.1 centos基础镜像构建

  1. 准备配置文件目录。主要用于存放各个项目的dockerfile和k8s服务的yaml文件目录

    #存放各个项目的dockerfile文件
root@harbor:~# mkdir -pv /opt/k8s-data/dockerfile/system/centos
​
#存放各个项目的yaml文件
root@harbor:~# mkdir -pv /opt/k8s-data/yaml/

  2. 下载filebeat的rpm文件, 准备repo文件

    root@harbor:~# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.13.4-x86_64.rpm
root@harbor:~# wget https://mirrors.aliyun.com/repo/Centos-7.repo
root@harbor:~# mv Centos-7.repo filebeat-8.13.4-x86_64.rpm /opt/k8s-data/dockerfile/system/centos/
root@harbor:~# cd /opt/k8s-data/dockerfile/system/centos/
root@harbor:/opt/k8s-data/dockerfile/system/centos# ls
Centos-7.repo  filebeat-8.13.4-x86_64.rpm

  3. 准备cento镜像的dockerfile文件

    root@k8s-master01:/opt/k8s-data/dockerfile/system/centos# cat Dockerfile 
#自定义Centos 基础镜像
FROM centos:7.9.2009 
ADD filebeat-8.13.4-x86_64.rpm /tmp
 
# 自定义安装工具和环境
RUN sed -e "s|^mirrorlist=|#mirrorlist=|g" \
         -e "s|^#baseurl=http://mirror.centos.org/centos/\$releasever|baseurl=https://mirrors.aliyun.com/centos-vault/7.9.2009|g" \
         -i.bak \
         /etc/yum.repos.d/CentOS-*.repo && rm -rf /etc/yum.repo.d/* && yum makecache &&yum install -y /tmp/filebeat-8.13.4-x86_64.rpm vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop &&  rm -rf /etc/localtime /tmp/filebeat-8.13.4-x86_64.rpm && ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime  && useradd nginx -u 2088
root@k8s-master01:~/k8s-data/dockerfile/system/centos# cat build-command.sh 
#!/bin/bash
nerdctl build -t harbor.cbear.top/baseimages/centos:7.9.2009 .
/usr/local/bin/nerdctl push harbor.cbear.top/baseimages/centos:7.9.2009
​

  4. 上传镜像

    nerdctl build -t  harbor.cbear.top/baseimages/centos79:v1 .
nerdctl push  harbor.cbear.top/baseimages/centos79:v1 .
    image-20240601154810655

  5. 验证构建镜像

    root@k8s-master01:/opt/k8s-data/dockerfile/system/centos# nerdctl run -it --rm harbor.cbear.top/baseimages/centos79:v1 bash
[root@a909808a9724 /]# filebeat --help
Usage:
  filebeat [flags]
  filebeat [command]
​
Available Commands:
  export      Export current config or index template
  generate    Generate Filebeat modules, filesets and fields.yml
  help        Help about any command
  keystore    Manage secrets keystore
  modules     Manage configured modules
  run         Run filebeat
  setup       Setup index template, dashboards and ML jobs
  test        Test config
  version     Show current version info
​
Flags:
  -E, --E setting=value              Configuration overwrite
  -M, --M setting=value              Module configuration overwrite
  -N, --N                            Disable actual publishing for testing
  -c, --c string                     Configuration file, relative to path.config (default "filebeat.yml")
      --cpuprofile string            Write cpu profile to file
  -d, --d string                     Enable certain debug selectors
  -e, --e                            Log to stderr and disable syslog/file output
      --environment environmentVar   set environment being ran in (default default)
  -h, --help                         help for filebeat
      --httpprof string              Start pprof http server
      --memprofile string            Write memory profile to this file
      --modules string               List of enabled modules (comma separated)
      --once                         Run filebeat only once until all harvesters reach EOF
      --path.config string           Configuration path (default "")
      --path.data string             Data path (default "")
      --path.home string             Home path (default "")
      --path.logs string             Logs path (default "")
      --plugin pluginList            Load additional plugins
      --strict.perms                 Strict permission checking on config files (default true)
  -v, --v                            Log at INFO level
​
Use "filebeat [command] --help" for more information about a command.
​

2.2 基于自定义centos基础镜像构建nginx镜像

  1. 准备nginx镜像目录环境

    root@k8s-master01:/opt/k8s-data/dockerfile# mkdir -pv web/put-images/nginx-base
​
root@k8s-master01:/opt/k8s-data/dockerfile# cd web/pub-images/nginx-base/

  2. 准备Dockerfile文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-base# cat Dockerfile 
# 导入自定义centos基础镜像
FROM harbor.cbear.top/baseimages/centos79:v1
# 安装编译环境
RUN yum install -y vim wget tree  lrzsz gcc gcc-c++ automake pcre pcre-devel zlib zlib-devel openssl openssl-devel iproute net-tools iotop
​
# 添加nginx源码至/usr/local/src/
ADD nginx-1.26.1.tar.gz /usr/local/src/
# 编译nginx
RUN cd /usr/local/src/nginx-1.26.1 && ./configure  && make && make install && ln -sv /usr/local/nginx/sbin/nginx /usr/sbin/nginx  &&rm -rf /usr/local/src/nginx-1.26.1.tar.gz 
​

  3. 构建镜像

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-base# nerdctl build -t harbor.cbear.top/pub-images/nginx:v1.26.1 .
root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-base# nerdctl push harbor.cbear.top/pub-images/nginx:v1.26.1

2.3 构建自定义nginx业务镜像

  1. 准备nginx.conf配置文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# cat nginx.conf 
user  nginx nginx;
worker_processes  auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;

daemon off;  ##关闭后台运行

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    #gzip  on;

#先关掉
#    upstream  tomcat_webserver {
#        server   linux39-tomcat-app1-service.linux39.svc.zzhz.local:80;
#    }

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        location /webapp {
            root   html;
            index  index.html index.htm;
        }
#先关掉
#        location /app1 {
#             proxy_pass  http://tomcat_webserver;
#             proxy_set_header   Host    $host;
#             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
#             proxy_set_header X-Real-IP $remote_addr;
#        }

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

  2. 准备访问页面

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# echo webapp.index.html > index.html
root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# tar zcvf app1.tar.gz index.html 
index.html
root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# echo nginx.index.html > index.html

  3. 配置Dockerfile文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# cat Dockerfile 
#Nginx 1.26.1
# 导入nginx基础镜像
FROM harbor.cbear.top/pub-images/nginx:v1.26.1
# 添加nginx配置文件
ADD nginx.conf /usr/local/nginx/conf/nginx.conf
# 添加业务代码
ADD app1.tar.gz  /usr/local/nginx/html/webapp/
ADD index.html  /usr/local/nginx/html/index.html
# 创建静态资源挂载路径
RUN mkdir -p /usr/local/nginx/html/webapp/static /usr/local/nginx/html/webapp/images 
# 暴露端口
EXPOSE 80 443
# 运行nginx
CMD ["nginx"]

  4. 准备构建镜像脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# cat build-command.sh 

#!/bin/bash
TAG=$1
nerdctl build -t harbor.cbear.top/pub-images/nginx-web1:${TAG} .
echo "镜像构建完成 , 即将上传到harbor"
sleep 1
nerdctl push harbor.cbear.top/pub-images/nginx-web1:${TAG}
echo "镜像上传到harbor完成"

  5. 开始构建镜像流程

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# bash build-command.sh v1

  6. 测试镜像效果

    root@k8s-master01:~# nerdctl run  -d -p 80:80 harbor.cbear.top/pub-images/nginx-web1:v1 
a4e962340ee5583b9b2c13754ee4499b45a0c117c628e72cdaae4f84bf482c90

root@k8s-master01:~# curl 192.168.71.29 
nginx.index.html
root@k8s-master01:~# curl 192.168.71.29/webapp/
webapp.index.html

2.4 基于自定义centos基础镜像构建tomcat镜像

2.4.1 准备jdk镜像

  1. 准备jdk二进制文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/jdk-17# wget https://download.oracle.com/java/17/latest/jdk-17_linux-x64_bin.tar.gz

  2. 准备profile文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/jdk-17# cat profile 
.....
export JAVA_HOME=/usr/local/jdk
export TOMCAT_HOME=/apps/tomcat
#export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$TOMCAT_HOME/bin:$PATH
export PATH=$JAVA_HOME/bin:$TOMCAT_HOME/bin:$PATH
export CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

  3. 准备jdk的Dockerfile文件

    #JDK Base Image
# 导入自定义centos基础镜像
FROM harbor.cbear.top/baseimages/centos79:v1
# 安装jdk环境
ADD jdk-17_linux-x64_bin.tar.gz /usr/local/src/
RUN ln -sv /usr/local/src/jdk-17.0.11 /usr/local/jdk 
ADD profile /etc/profile
ENV JAVA_HOME /usr/local/jdk
ENV CLASSPATH $JAVA_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin

  4. 配置构建脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/jdk-17# vi build-command.sh
#!/bin/bash
TAG=$1
nerdctl build -t harbor.cbear.top/pub-images/jdk-17-base:${TAG} .
echo "镜像构建完成 , 即将上传到harbor"
sleep 1
nerdctl push harbor.cbear.top/pub-images/jdk-17-base:${TAG} 
echo "镜像上传到harbor完成"

  5. 运行构建脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/jdk-17# bash build-command.sh v2

  6. 测试构建完成之后的镜像是否可用

    root@k8s-master01:~# nerdctl run -it --rm  harbor.cbear.top/pub-images/jdk-17-base:v2 bash
[root@402a2b29959d /]# java -version
java version "17.0.11" 2024-04-16 LTS
Java(TM) SE Runtime Environment (build 17.0.11+7-LTS-207)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.11+7-LTS-207, mixed mode, sharing)

2.4.2 基于jdk镜像准备tomcat镜像

  1. 下载二进制tomcat包

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat# wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.0-M20/bin/apache-tomcat-11.0.0-M20.tar.gz

  2. 配置Dockerfile文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat# cat Dockerfile 
#Tomcat 8.5.43基础镜像
# 导入自定义jdk镜像
FROM harbor.cbear.top/pub-images/jdk-17-base:v2
# 创建tomcat安装目录、数据目录和日志目录
RUN mkdir /apps /data/tomcat/webapps /data/tomcat/logs -pv 
# 安装tomcat
ADD apache-tomcat-11.0.0-M20.tar.gz  /apps
RUN useradd tomcat -u 2050 && ln -sv /apps/apache-tomcat-11.0.0-M20 /apps/tomcat && chown -R tomcat.tomcat /apps /data

  3. 准备构建镜像脚本文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat# cat build-command.sh 
#!/bin/bash
TAG=$1
nerdctl build -t harbor.cbear.top/pub-images/tomcat-base:11.0.0 .
echo "镜像构建完成 , 即将上传到harbor"
sleep 1
nerdctl push harbor.cbear.top/pub-images/tomcat-base:11.0.0 
echo "镜像上传到harbor完成"

  4. 运行构建脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat# bash build-command.sh

  5. 测试构建的tomcat镜像功能

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat# nerdctl run -it --rm -p 8080:8080  harbor.cbear.top/pub-images/tomcat-base:11.0.0
[root@d9cb5e8fd4f2 /]# ss -ntlp
State      Recv-Q Send-Q                          Local Address:Port                                         Peer Address:Port              
[root@d9cb5e8fd4f2 /]# /apps/tomcat/bin/catalina.sh start 
Using CATALINA_BASE:   /apps/tomcat
Using CATALINA_HOME:   /apps/tomcat
Using CATALINA_TMPDIR: /apps/tomcat/temp
Using JRE_HOME:        /usr/local/jdk
Using CLASSPATH:       /apps/tomcat/bin/bootstrap.jar:/apps/tomcat/bin/tomcat-juli.jar
Using CATALINA_OPTS:   
Tomcat started.
    image-20240601194738379

2.4.3 基于自定义tomcat镜像构建tomcat业务镜像

  1. 准备server.xml文件, 临时创建一个tomcat容器, 修改对应的server.xml文件然后拷贝出来即可

    
      <Host name="localhost"  appBase="/data/tomcat/webapps" unpackWARs="true" autoDeploy="true">

  2. 准备tomcat页面文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# echo "tomcat app1" > index.html
root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# tar -zcvf app1.tar.gz index.html 
index.html

  3. 启动脚本文件run_tomcat.sh

    #!/bin/bash
#echo "nameserver 223.6.6.6" > /etc/resolv.conf
#echo "192.168.7.248 k8s-vip.example.com" >> /etc/hosts

#/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &
su - nginx -c "/apps/tomcat/bin/catalina.sh start"
tail -f /etc/hosts

root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# chmod +x run_tomcat.sh

  4. 准备Dockerfile文件

    #tomcat web1
# 导入自定义tomcat镜像
FROM harbor.cbear.top/pub-images/tomcat-base:11.0.0
# 添加启动脚本和配置文件
#ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
# 添加业务代码
ADD app1.tar.gz /data/tomcat/webapps/app1/
#ADD filebeat.yml /etc/filebeat/filebeat.yml 
RUN chown  -R nginx.nginx /data/ /apps/
#ADD filebeat-7.5.1-x86_64.rpm /tmp/
#RUN cd /tmp && yum localinstall -y filebeat-7.5.1-amd64.deb
# 暴露端口
EXPOSE 8080 8443
# 启动tomcat

root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# cat Dockerfile 
#tomcat web1
# 导入自定义tomcat镜像
FROM harbor.cbear.top/pub-images/tomcat-base:11.0.0
# 添加启动脚本和配置文件
#ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
# 添加业务代码
ADD app1.tar.gz /data/tomcat/webapps/app1/
#ADD filebeat.yml /etc/filebeat/filebeat.yml 
RUN chown  -R nginx.nginx /data/ /apps/
# 暴露端口
EXPOSE 8080 8443
# 启动tomcat
CMD ["/apps/tomcat/bin/run_tomcat.sh"]

  5. 准备构建脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# cat build-command.sh 
#!/bin/bash
TAG=$1
nerdctl build -t harbor.cbear.top/pub-images/tomcat-app1:${TAG} .
echo "镜像构建完成 , 即将上传到harbor"
sleep 1
nerdctl push harbor.cbear.top/pub-images/tomcat-app1:${TAG} 
echo "镜像上传到harbor完成"

  6. 运行构建脚本

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# bash build-command.sh v1

  7. 测试容器, 并访问业务

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/tomcat-app# nerdctl run -it --rm -p 8080:8080 harbor.cbear.top/pub-images/tomcat-app1:v1
    image-20240601204058307

3. 在k8s环境中运行tomcat

3.1 准备nfs环境

  1. 准备NFS共享目录

    root@harbor:~# mkdir -pv /data/k8sdata/cbear/nginx/images
root@harbor:~# mkdir -pv /data/k8sdata/cbear/nginx/static

  2. 安装nfs

    root@harbor:/usr/local/harbor# apt install nfs-server

  3. 配置nfs共享目录

    在NFS (Network File System) 配置中 , subtree_check 和 no_subtree_check 是两个重要的选项 , 它们控制NFS服务器如何处理子目录的权限检查。

    在早期版本的 nfs-utils 中 , 默认行为是 subtree_check , 这意味着NFS服务器会检查每个子目录的权限 , 以确保客户端有权访问。然而 , 从 nfs-utils 版本 1.0.x 开始 , 默认行为改为了 no_subtree_check , 这意味着服务器不会进行子目录的权限检查。

    root@harbor:~# cat /etc/exports 
# /etc/exports: the access control list for filesystems which may be exported
#		to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#
/data/k8sdata/cbear/nginx/static *(rw,no_root_squash,no_subtree_check)
/data/k8sdata/cbear/nginx/images *(rw,no_root_squash,no_subtree_check)

  4. 重新识别nfs共享配置

    root@harbor:/usr/local/harbor# exportfs  -rv
exporting *:/data/k8sdata/cbear/nginx/images
exporting *:/data/k8sdata/cbear/nginx/static

  5. 在其他节点检查识别可以识别到nfs共享

    root@ha1:~# showmount -e 192.168.71.34
Export list for 192.168.71.34:
/data/k8sdata/cbear/nginx/images *
/data/k8sdata/cbear/nginx/static *

3.2 配置tomcat 相关资源

  1. 准备存放两个资源的ymal目录

    root@k8s-master01:/opt/k8s-data/yaml# mkdir /opt/k8s-data/yaml/cbear/tomcat-app1
root@k8s-master01:/opt/k8s-data/yaml# mkdir /opt/k8s-data/yaml/cbear/nginx

  2. 创建对应项目的namespaces

    root@k8s-master01:/opt/k8s-data/yaml/namespaces# cat namespaces-cbear.yaml 
apiVersion: v1
kind: Namespace
metadata: 
  name: cbear
root@k8s-master01:/opt/k8s-data/yaml/namespaces# kubectl get ns
NAME              STATUS   AGE
default           Active   5d4h
kube-node-lease   Active   5d4h
kube-public       Active   5d4h
kube-system       Active   5d4h
root@k8s-master01:/opt/k8s-data/yaml/namespaces# kubectl apply -f namespaces-cbear.yaml 
namespace/cbear created

3.2.1 tomcat镜像对应的yaml文件

这里通过创建对应deployment实现对底层pod的管理。同时通过service实现对pod的网络负载均衡管理。因此需要创建这两种资源。

  1. tomcat的deployment的yaml资源文件

    root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# cat tomcat-app1-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cbear-tomcat-app1-deployment-label
  name: cbear-tomcat-app1-deployment
  namespace: cbear
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cbear-tomcat-app1-selector
  template:
    metadata:
      labels:
        app: cbear-tomcat-app1-selector
    spec:
      containers:
      - name: cbear-tomcat-app1-container
        image: harbor.cbear.top/pub-images/tomcat-app1:v1
        #command: ["/apps/tomcat/bin/run_tomcat.sh"]
        imagePullPolicy: IfNotPresent
        #imagePullPolicy: Always
        ports:
        - containerPort: 8080
          protocol: TCP
          name: http
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "18"
        resources:
          limits:
            cpu: 1
            memory: "512Mi"
          requests:
            cpu: 500m
            memory: "512Mi"
        volumeMounts:
        - name: cbear-images
          mountPath: /data/tomcat/webapps/app1/images
          readOnly: false
        - name: cbear-static
          mountPath: /data/tomcat/webapps/app1/static
          readOnly: false
      volumes:
      - name: cbear-images
        nfs:
          server: 192.168.71.34
          path: /data/k8sdata/cbear/nginx/images
      - name: cbear-static
        nfs:
          server: 192.168.71.34
          path: /data/k8sdata/cbear/nginx/static
#      nodeSelector:
#        project: cbear
#        app: tomcat

  2. tomcat的service的yaml资源文件

    root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# cat tomcat-app1-services.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cbear-tomcat-app1-service-label
  name: cbear-tomcat-app1-service
  namespace: cbear
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30092
  selector:
    app: cbear-tomcat-app1-selector

3.2.2 启用对应的yaml文件

  1. 启用具体资源

    root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# ls
tomcat-app1-deployment.yaml  tomcat-app1-services.yaml
root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl apply -f ./
deployment.apps/cbear-tomcat-app1-deployment created
service/cbear-tomcat-app1-service created
root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl get pods -n cbear -o wide
NAME                                            READY   STATUS    RESTARTS   AGE   IP            NODE                   NOMINATED NODE   READINESS GATES
cbear-tomcat-app1-deployment-6655c4f55b-97xqn   1/1     Running   0          19s   172.20.4.9    k8s-node02.cbear.top   <none>           <none>
cbear-tomcat-app1-deployment-6655c4f55b-b95db   1/1     Running   0          19s   172.20.3.16   k8s-node01.cbear.top   <none>           <none>
cbear-tomcat-app1-deployment-6655c4f55b-svcv4   1/1     Running   0          19s   172.20.5.15   k8s-node03.cbear.top   <none>           <none>
root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl get svc -n cbear
NAME                        TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
cbear-tomcat-app1-service   NodePort   10.68.3.58   <none>        80:30092/TCP   5m47s
root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl describe  svc/cbear-tomcat-app1-service -n cbear
Name:                     cbear-tomcat-app1-service
Namespace:                cbear
Labels:                   app=cbear-tomcat-app1-service-label
Annotations:              <none>
Selector:                 app=cbear-tomcat-app1-selector
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.68.3.58
IPs:                      10.68.3.58
Port:                     http  80/TCP
TargetPort:               8080/TCP
NodePort:                 http  30092/TCP
Endpoints:                172.20.3.16:8080,172.20.4.9:8080,172.20.5.15:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

  2. 验证集群外30092端口是否可以正常访问

    image-20240602004736239

  3. 由于之后的tomcat不对集群外提供服务。因此这里将tomcat的services服务, 设置为ClusterIP类型。仅对集群内存提供服务

    root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# cat tomcat-app1-services.yaml 
apiVersion: v1
kind: Service
metadata:
  labels:
    app: cbear-tomcat-app1-service-label
  name: cbear-tomcat-app1-service
  namespace: cbear
spec:
  type: ClusterIP                    #这里将NodePort修改为ClusterIP。
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 30092
  selector:
    app: cbear-tomcat-app1-selector

#然后重新应用服务
root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl apply -f ./
deployment.apps/cbear-tomcat-app1-deployment configured
service/cbear-tomcat-app1-service configured

root@k8s-master01:/opt/k8s-data/yaml/cbear/tomcat-app1# kubectl get svc -n cbear
NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
cbear-tomcat-app1-service   ClusterIP   10.68.3.58   <none>        80/TCP    64m

4. 在k8s环境中运行nginx

4.1 配置nginx相关资源

实现一个通用的nginx+tomcat动静分离web架构，即用户访问的静态页面和图片在由nginx直接响应，而动态请求则基于location转发至tomcat。

4.1.1 重构nginx业务镜像

这里重构nginx镜像过程中, 需要重新配置nginx服务的配置文件 , 使其支持后端的tomcat容器

  1. 重新配置nginx.conf文件

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# cat nginx.conf 
user  nginx nginx;
worker_processes  auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;
#pid        logs/nginx.pid;

daemon off;  ##关闭后台运行

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;

    keepalive_timeout  65;

    #gzip  on;

    #开启以下内容
    upstream  tomcat_webserver {
        server   cbear-tomcat-app1-service.cbear.svc.cbear.local:80;
    }

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root   html;
            index  index.html index.htm;
        }

        location /webapp {
            root   html;
            index  index.html index.htm;
        }
        #开启以下内容
        location /app1 {
             proxy_pass  http://tomcat_webserver;
             proxy_set_header   Host    $host;
             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
             proxy_set_header X-Real-IP $remote_addr;
        }

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

  2. 开始构建镜像

    root@k8s-master01:/opt/k8s-data/dockerfile/web/pub-images/nginx-web# bash build-command.sh v2

4.1.2 配置对应的yaml文件

  1. nginx的deployment的yaml资源文件

    apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cbear-nginx-deployment-label
  name: cbear-nginx-deployment
  namespace: cbear
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cbear-nginx-selector
  template:
    metadata:
      labels:
        app: cbear-nginx-selector
    spec:
      containers:
      - name: cbear-nginx-container
        image: harbor.cbear.top/pub-images/nginx-web1:v2
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        #imagePullPolicy: None
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
        - containerPort: 443
          protocol: TCP
          name: https
        env:
        - name: "password"
          value: "123456"
        - name: "age"
          value: "20"
        resources:
          limits:
            cpu: 1
            memory: 512Mi
          requests:
            cpu: 200m
            memory: 246Mi

        volumeMounts:
        - name: cbear-images
          mountPath: /usr/local/nginx/html/webapp/images 
          readOnly: false
        - name: cbear-static
          mountPath: /usr/local/nginx/html/webapp/static 
          readOnly: false
      volumes:
      - name: cbear-images
        nfs:
          server: 192.168.71.34
          path: /data/k8sdata/cbear/nginx/images 
      - name: cbear-static
        nfs:
          server: 192.168.71.34
          path: /data/k8sdata/cbear/nginx/static
      #nodeSelector:
      #  group: linux39

  2. nginx的services的yaml文件

    apiVersion: v1
kind: Service
metadata:
  labels:
    app: cbear-nginx-service-label
  name: cbear-nginx-service
  namespace: cbear
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30090
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 30091
  selector:
    app: cbear-nginx-selector

  3. 启用对应资源

    root@k8s-master01:/opt/k8s-data/yaml/cbear/nginx# kubectl apply -f ./
deployment.apps/cbear-nginx-deployment created
service/cbear-nginx-service created
root@k8s-master01:/opt/k8s-data/yaml/cbear/nginx# ls
nginx-deployment.yaml  nginx-service.yaml

4.2测试配置结果

root@k8s-master01:/opt/k8s-data/yaml/cbear/nginx# curl 192.168.71.35:30090
nginx.index.html
root@k8s-master01:/opt/k8s-data/yaml/cbear/nginx# curl 192.168.71.35:30090/webapp/
webapp.index.html
root@k8s-master01:/opt/k8s-data/yaml/cbear/nginx# curl 192.168.71.35:30090/app1/
tomcat app1

5. 在负载均衡器上代理nginx服务

  1. 配置haproxy。之后重启haproxy服务

    root@ha2:~# cat /etc/haproxy/haproxy.cfg 
global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http
listen stats 
       mode http 
       bind 0.0.0.0:9999 
       stats enable 
       log global 
       stats uri     /haproxy-status 
       stats auth    haadmin:123456 
 
listen k8s-6443 
       bind 192.168.71.150:6443 
       mode tcp 
       balance roundrobin 
       server 192.168.71.29 192.168.71.29:6443 check inter 2s fall 3 rise 5 
       server 192.168.71.30 192.168.71.30:6443 check inter 2s fall 3 rise 5 
       server 192.168.71.31 192.168.71.31:6443 check inter 2s fall 3 rise 5 
listen nginx-svc-80
       bind 192.168.71.150:80
       mode tcp
       server k8s-node01 192.168.71.37:30090 check inter 2000 fall 3 rise 5
       server k8s-node02 192.168.71.35:30090 check inter 2000 fall 3 rise 5
       server k8s-node03 192.168.71.36:30090 check inter 2000 fall 3 rise 5

  2. 配置挂载的静态页面, 之后也会通过tomcat访问

    root@harbor:/opt/k8s-data/dockerfile/system/centos# echo test-tomcat-static > /data/k8sdata/cbear/nginx/static/index.html
    image-20240602024709866

参考链接

k8s实战案例之部署Nginx+Tomcat+NFS实现动静分离_nginx nfs-CSDN博客

#