Devops - Jenkins CICD / kubernetes 动态 slave

2022.11.10

作者： Ju4t

按照官方流程走：https://plugins.jenkins.io/kubernetes/

Agent：jenkins 5000端口设置好即可，http://xxxx/configureSecurity

环境

k8s

containerd

安装 Jenkins

helm

$ cat <<EOF > values.yaml:
extraEnvVars:
  - name: JENKINS_JNLP_PORT_NUMBER
    value: "50000"
service:
  extraPorts:
  - name: tunnel
    port: 50000
    protocol: TCP
    targetPort: 50000
EOF
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm install jenkins -f helm-values.yaml --namespace devops \
  --set jenkinsUser=admin \
  --set jenkinsPassword=JXBING \
  bitnami/jenkins

插件

kubernetes https://plugins.jenkins.io/kubernetes/
Localization: Chinese (Simplified)
Blue Ocean
Build Timestamp (设置Timezone:Asia/Shanghai Pattern：yyyyMMddHHmmss)
Generic Webhook Trigger

Jenkins设置

配置通信端口

在安全中配置 TCP port for inbound agents

Fixed: 50000

helm安装时设置的 JENKINS_JNLP_PORT_NUMBER 实测不生效

配置Clouds

Kubernetes 地址: https://kubernetes.default.svc.cluster.local (可选)

Jenkins 地址：http://jenkins.devops.svc.cluster.local/

Jenkins 通道：jenkins.devops.svc.cluster.local:50000

其他pod模版可以不设置，在Jenkinsfile里自定义

凭据

token或者kube-config二选一

kubeconfig

上传~/.kube/config 到凭据里即可在pipline里调用

Token

创建账户

$ kubectl create ns devops

$ kubectl create sa jenkins --namespace devops
$ kubectl create clusterrolebinding jenkins --clusterrole cluster-admin --serviceaccount=devops:jenkins

k8s token

# 1.查看sa
$ kubectl get sa -n devops
NAME      SECRETS   AGE
default   1         18d
jenkins   1         1m

# 2.查看name
$ kubectl describe sa jenkins -n devops
$ kubectl -n devops get serviceaccount jenkins -o go-template --template='{{range .secrets}}{{.name}}{{"\n"}}{{end}}'
Name:                jenkins
Namespace:           devops
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   jenkins-token-mpznq
Tokens:              jenkins-token-mpznq
Events:              <none>

# 获取token
$ kubectl describe secrets jenkins-token-mpznq -n devops
$ kubectl -n devops get secrets jenkins-token-mpznq -o go-template --template '{{index .data "token"}}' | base64 -d
Name:         jenkins-token-mpznq
Namespace:    devops
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: jenkins
              kubernetes.io/service-account.uid: 789022ab-8251-4ac5-b502-729028026517

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1099 bytes
namespace:  6 bytes
# 复制token即可，不用再解密
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImtPOGhjMGVMcVBiYjhuNkRSUW5aZmpoY0dVWUxRaWNHbENERkFiYUhnc1EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXZvcHMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiamVua2lucy10b2tlbi1tcHpucSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJqZW5raW5zIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOTZhNDJkNTEtMDJiNy00MDhjLThhNTUtZTI3ODNhM2VjZjQ5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRldm9wczpqZW5raW5zIn0.F-r82qQzbBH2akkRA0Zn4Nopyonzqtd0Lbq69q8rxANA8J8OBgOINBcThg8ypr9Ia8b3TrDHMC5vvaTPwkfLfy8ZcwCKGxJaoPOnOoe3fJyEyQiuX36Eam-Ga098SiGpmxrtsOkxeLFGL_G9tO6TQDyxUrjnRxtdO6Z9O5KSLmA2jZXxBx2rL0pR8-izrO-h6wDFlx_qg2xGH2cHdeaiZptx4jFznyBpwo5zZeUHAzLA7hqCK_cekPAytIIPlret6FrJJ0A1CaccX4KlyIhUtlzIHzP5L9KaAxXRYKlEA_Nm19oE594pCKfHHt8zu908c5qGaIAUSNc4d4UbrBNkFQ

证书（外部链接的方式，可选）

# 1、查看kubernetes的config文件
$ cat ~/.kube/config

# 2、根据配置文件生成证书.替换引号内部的信息为config内相关value
$ echo "certificate-authority-data" | base64 -d > ca.crt
$ echo "client-certificate-data" | base64 -d > client.crt
$ echo "client-key-data" | base64 -d > client.key
# 3、生成jenkins使用的cert.pfx，此处需要设置一个4位数以上的密码
$ openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt

# 添加评据，上传证书
# 密码即为 制作证书的过程的密码

Demo

首次执行：需要先部署应用，否则 “部署应用” 这步操作将失败

执行前：先把maven-setting、docker-config、kube-config 这些文件准备好，以及pvc：maven-local-repo

env.GIT_URL    = "https://code.aliyun.com/Ju4t/helloword-springboot.git"
env.APP_NAME   = "helloword"
env.NAMESPACE  = "default"
env.IMAGE_NAME = "ju4t/$APP_NAME"
env.IMAGE_TAG  = "v$BUILD_TIMESTAMP"

// podTemplate help doc https://plugins.jenkins.io/kubernetes/
podTemplate(
  cloud: 'kubernetes',
  namespace: 'devops',
  // idleMinutes: 10, // 执行后，释放时间(分)
  // imagePullSecrets: ''
  containers: [
    containerTemplate(name: 'maven', image: 'maven:3.8.1-jdk-8', command: 'sleep', args: '99d'),
    containerTemplate(name: 'buildkit', image: 'moby/buildkit:master', privileged: 'true'),
    containerTemplate(name: 'kubectl', image: 'ju4t/kubectl', command: 'cat', ttyEnabled: 'true')
  ],
  volumes: [
    // 挂载 docker-config，buildctl build 推送镜像
    configMapVolume(configMapName: 'docker-config', mountPath: '/root/.docker/'),
    // maven settings.xml 加速
    configMapVolume(configMapName: 'maven-setting', mountPath: '/root/.m2/'),
    // maven repository
    persistentVolumeClaim(claimName: 'maven-local-repo', mountPath: '/root/.m2/repository/'),
    // 本例中使用 credentialsId 覆盖 /root/.kube/config文件
    // configMapVolume(configMapName: 'kube-config', mountPath: '/root/.kube/'),
    // more
    // emptyDirVolume(mountPath: '/etc/mount1', memory: false),
    // secretVolume(mountPath: '/etc/mount2', secretName: 'my-secret'),
    // configMapVolume(mountPath: '/etc/mount3', configMapName: 'my-config'),
    // hostPathVolume(mountPath: '/etc/mount4', hostPath: '/mnt/my-mount'),
    // nfsVolume(mountPath: '/etc/mount5', serverAddress: '127.0.0.1', serverPath: '/', readOnly: true),
    // persistentVolumeClaim(mountPath: '/etc/mount6', claimName: 'myClaim', readOnly: true)
  ]
) { 
  node(POD_LABEL) {

    stage('Git Clone') {
      git url: "$GIT_URL"
    }

    stage('Maven project') {
      container('maven') {
        sh 'mvn -B clean package -Dmaven.test.skip=true'
      }
    }

    stage('Build & Push') {
      container('buildkit') {
        // sh "buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=ju4t/helloword:springboot_${BUILD_ID},push=true"
        sh '''
        buildctl build --frontend dockerfile.v0 \
        --local context=. --local dockerfile=. \
        --output type=image,name=$IMAGE_NAME:$IMAGE_TAG,push=true
        '''
        // 推送镜像到仓库，添加以下命令
        // `--output type=image,name=docker.io/username/image,push=true`
        // 推送镜像需要创建 `~/.docker/config.json`，包含密钥信息（yaml/configmap-docker-config.yaml）
      }
    }

    // 部署方式一：更新镜像
    // stage('Deployment') {
    //   // 上传.kube/config 到 凭据 中，命名为：kube-admin
    //   withCredentials([file(credentialsId: 'kube-admin', variable: 'KUBECONFIG')]) {
    //     container('kubectl') {
    //       // sh 'mkdir -p ~/.kube && cp $KUBECONFIG ~/.kube/config'
    //       // sh 'kubectl set image deployment/helloword helloword=ju4t/helloword:v1.0 -n default'
    //       // sh 'kubectl set image deployment/<deployment_name> <containers_name>=ju4t/helloword:tag'
    //       sh 'kubectl --kubeconfig $KUBECONFIG set image deployment/$APP_NAME $APP_NAME=$IMAGE_NAME:$IMAGE_TAG -n $NAMESPACE'
    //       sh 'kubectl --kubeconfig $KUBECONFIG get pod -o wide -n $NAMESPACE'
    //       sh 'kubectl --kubeconfig $KUBECONFIG get svc -o wide -n $NAMESPACE'
    //     }
    //   }
    // }

    // 部署方式二
    stage('Deployment') {
      // 上传.kube/config 到 凭据 中，命名为：kube-admin
      withCredentials([file(credentialsId: 'kube-admin', variable: 'KUBECONFIG')]) {
        container('kubectl') {
          sh '''
cat > ./deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: $APP_NAME
  namespace: $NAMESPACE
spec:
  selector:
    matchLabels:
      app: $APP_NAME
  template:
    metadata:
      labels:
        app: $APP_NAME
    spec:
      containers:
      - name: $APP_NAME
        image: $IMAGE_NAME:$IMAGE_TAG
        imagePullPolicy: IfNotPresent
        ports:
          - containerPort: 8080
        livenessProbe:
          initialDelaySeconds: 3
          periodSeconds: 3
          httpGet:
            path: /
            port: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: $APP_NAME
  namespace: $NAMESPACE
  labels:
    app: $APP_NAME
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  - port: 443
    targetPort: 8080
    protocol: TCP
    name: https
  selector:
    app: $APP_NAME
EOF'''
              // ' 比 " 安全
          sh 'kubectl --kubeconfig $KUBECONFIG apply -f ./deployment.yaml'
          sh 'kubectl --kubeconfig $KUBECONFIG get pod -l app=$APP_NAME -o wide -n $NAMESPACE'
          sh 'echo $APP_NAME URL http://$(kubectl --kubeconfig $KUBECONFIG get svc $APP_NAME --namespace $NAMESPACE --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")/'
        }
      }
    }

  }
}

webhook

设置token后在git里设置webhook触发即可

token，不能包含字母（待验证）？

http://220.171.2.38:8880/generic-webhook-trigger/invoke?token=123

测试

$ git status
$ git add .
$ git commit -m "update .."
$ git push origin master

其他资料

buildctl 构建镜像

containerTemplate(name: 'buildkit', image: 'moby/buildkit:master', privileged: 'true')

sh "buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=. --output type=image,name=docker.io/username/${JOB_NAME}:${BUILD_ID},push=true"

https://github.com/moby/buildkit/tree/master/examples/kubernetes

buildctl 推送镜像

cat <<EOF >  /root/.docker/config.json
{
    "auths": {
        "registry.cn-beijing.aliyuncs.com": {
            "auth": "CHENGEME"
        }
    }
}
EOF