Traefik2.2从坑出发

极客运维圈

发布于 2020-06-01 11:54:30

3.2K0

Traefik2.2安装

Traefik 2.2新增的功能如下： 1. 支持了udp 2. traefik2.2 支持使用K/V存储做为动态配置的源，分别是 consul, etcd, Redis, zookeeper 3. 能够使用kubernetes CRD自定义资源定义UDP负载平衡 IngressRouteUDP。 4. 能够使用 rancher， consul catalog， docker和 marathon中的标签定义UDP的负载平衡 5. 增加了对ingress注解的主持 6. 将TLS存储功能 TLSStores添加到Kubernetes CRD中，使kubernetes用户无需使用配置文件和安装证书即可提供默认证书。 7. 在日志中增加了http的请求方式,是http还是https 8. 因为TLS的配置可能会影响CPU的使用率，因此增加了 TLS version和 TLS cipher使用的指标信息 9. 当前的WRR算法对于权重不平衡端点存在严重的偏差问题，将EDF调度算法用于WeightedRoundRobin， Envoy也是使用了 EOF调度算法 10. 支持请求主体用于流量镜像 11. 增加了 ElasticAPM作为traefik的tracing系统。 12. Traefik的Dashboard增加了UDP的页面 13. Traefik也增加了黑暗主题

下面进行安装过程。

注：我们这里是将traefik部署在ingress-traefik命名空间，如果你需要部署在其他命名空间，需要更改资源清单，如果你是部署在和我同样的命令空间中，你需要创建该命名空间。

创建命名空间：

# kubectl create ns ingress-traefik

1、创建CRD资源

Traefik 2.0版本后开始使用CRD来对资源进行管理配置，所以我们需要先创建CRD资源。 traefik-crd.yaml

## IngressRoute
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
---
## IngressRouteTCP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
---
## Middleware
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
---
## TraefikService
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice

---
## TraefikTLSStore
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsstores.traefik.containo.us
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSStore
    plural: tlsstores
    singular: tlsstore

---
## IngressRouteUDP
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressrouteudps.traefik.containo.us 
spec:
  scope: Namespaced
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteUDP
    plural: ingressrouteudps
    singular: ingressrouteudp

创建资源清单：

# kubectl apply -f traefik-crd.yaml
# kubectl get crd
NAME                                    CREATED AT
ingressroutes.traefik.containo.us       2019-12-13T05:40:30Z
ingressroutetcps.traefik.containo.us    2019-12-13T05:40:30Z
middlewares.traefik.containo.us         2019-12-13T05:40:30Z

2、创建RBAC权限

traefik-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: ingress-traefik 
  name: traefik-ingress-controller
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups: [""]
    resources: ["services","endpoints","secrets"]
    verbs: ["get","list","watch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get","list","watch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses/status"]
    verbs: ["update"]
  - apiGroups: ["traefik.containo.us"]
    resources: ["middlewares"]
    verbs: ["get","list","watch"]
  - apiGroups: ["traefik.containo.us"]
    resources: ["ingressroutes","traefikservices"]
    verbs: ["get","list","watch"]
  - apiGroups: ["traefik.containo.us"]
    resources: ["ingressroutetcps","ingressrouteudps"]
    verbs: ["get","list","watch"]
  - apiGroups: ["traefik.containo.us"]
    resources: ["tlsoptions","tlsstores"]
    verbs: ["get","list","watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: ingress-traefik

创建RBAC资源清单：

# kubectl apply -f traefik-rbac.yaml

3、创建traefik配置文件

traefik-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-config
  namespace: ingress-traefik
data:
  traefik.yaml: |-
    serversTransport:
      insecureSkipVerify: true
    api:
      insecure: true
      dashboard: true
      debug: true
    metrics:
      prometheus: ""
    entryPoints:
      web:
        address: ":80"
      websecure:
        address: ":443"
    providers:
      kubernetesCRD: ""
    log:
      filePath: ""
      level: error
      format: json
    accessLog:
      filePath: ""
      format: json
      bufferingSize: 0
      filters:
        retryAttempts: true
        minDuration: 20
      fields:
        defaultMode: keep
        names:
          ClientUsername: drop
        headers:
          defaultMode: keep
          names:
            User-Agent: redact
            Authorization: drop
            Content-Type: keep

创建ConfigMap:

# kubectl apply -f traefik-config.yaml

4、给节点打标签

由于是 Kubernetes DeamonSet 这种方式部署 Traefik，所以需要提前给节点设置 Label，这样当程序部署时 Pod 会自动调度到设置 Label 的点上。 节点设置 Label 标签

格式：kubectl label nodes [节点名] [key=value]

kubectl label nodes 172.16.0.33 IngressProxy=true

查看节点标签：

# kubectl get nodes --show-labels 
NAME          STATUS                     ROLES    AGE   VERSION   LABELS
172.16.0.33   Ready,SchedulingDisabled   master   20d   v1.15.0   IngressProxy=true,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.0.33,kubernetes.io/os=linux,kubernetes.io/role=master
172.16.0.52   Ready                      node     20d   v1.15.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/fluentd-ds-ready=true,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=172.16.0.52,kubernetes.io/os=linux,kubernetes.io/role=node

5、部署traefik

很多时候我们会采用DS方式部署，并且设置网络为hostNetwork=True，这样方便流量进入。但是在这里我采用的是service的nodeport进行暴露。

（1）、Service的配置清单 traefik-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: traefik
  namespace: ingress-traefik
spec:
  type: NodePort
  ports:
    - name: web
      port: 80
    - name: websecure
      port: 443
    - name: admin
      port: 8080
  selector:
    app: traefik

（2）、DS的配置清单 traefik-ds.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: traefik-ingress-controller
  namespace: ingress-traefik
  labels:
    app: traefik
spec:
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      name: traefik
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 1
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/rookieops/traefik:v2.2.0
          name: traefik-ingress-lb
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          resources:
            limits:
              cpu: 2000m
              memory: 1024Mi
            requests:
              cpu: 1000m
              memory: 1024Mi
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
          args:
            - --configfile=/config/traefik.yaml
          volumeMounts:
            - mountPath: "/config"
              name: "config"
      volumes:
        - name: config
          configMap:
            name: traefik-config 
      tolerations:              #设置容忍所有污点，防止节点被设置污点
        - operator: "Exists"
      nodeSelector:             #设置node筛选器，在特定label的节点上启动
        IngressProxy: "true"

3) 配置traefik路由规则

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-route
  namespace: ingress-traefik
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`traefik.coolops.cn`)
      kind: Rule
      services:
        - name: traefik
          port: 8080

然后创建资源：

# kubectl apply -f traefik-svc.yaml
# kubectl apply -f traefik-ds.yaml
# kubectl get svc -n ingress-traefik 
NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                     AGE
traefik   NodePort   10.102.225.66   <none>        80:30161/TCP,443:30629/TCP,8080:32359/TCP   31m
# kubectl get pod -n ingress-traefik 
NAME                               READY   STATUS    RESTARTS   AGE
traefik-ingress-controller-whbjm   1/1     Running   0          22m

然后本地配置hosts

10.1.10.128 traefik.coolops.cn

由于我们ingress是通过svc的nodeport暴露的，所以输入以下访问

并且可以看到我配置的HTTP

6、配置SSL

这里使用Let's Encrypt 来进行自动化 HTTPS。修改配置文件，新增如下内容：

    certificatesresolvers:
      default:
        acme:
          tlsChallenge: {}
          email: "coolops@163.com"
          storage: "acme.json"

更新配置文件。新增一个https的ingressRoute，如下：

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-webui-tls
  namespace: ingress-traefik 
spec:
  entryPoints:
  - websecure  # 注意这里是websecure这个entryPoint，监控443端口
  routes:
  - match: Host(`traefik.coolops.cn`)
    kind: Rule
    services:
    - name: traefik
      port: 8080
  tls:
    certResolver: default

查看pod的日志信息，报错如下：

{"level":"error","msg":"Unable to obtain ACME certificate for domains \"traefik.coolops.cn\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": dial tcp 172.65.32.248:443: connect: connection refused","providerName":"default.acme","routerName":"ingress-traefik-traefik-webui-tls-fcde00a088d29eefb3a6@kubernetescrd","rule":"Host(`traefik.coolops.cn`)","time":"2020-05-26T02:49:49Z"}

这是因为pod里的网络和https://acme-v02.api.letsencrypt.org/directory 不通造成的。部署成功后可以使用HTTPS访问了。

上面是自动生成证书，如果有自己的域名证书，那么一切都简单了，你只需要配置一个secret，然后在ingressRoute中引用即可，比如下面来自官方的例子：

apiVersion: v1
kind: Secret
metadata:
  name: supersecret
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0=

---

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroutetls
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`foo.com`) && PathPrefix(`/bar`)
    kind: Rule
    services:
    - name: whoami
      port: 443
  tls:
    secretName: supersecret

用Let's Encrypt的话，虽然免费，用起来还是有坑，这就需要自己去踩了~~！

完

本文参与腾讯云自媒体同步曝光计划，分享自微信公众号。

原始发表：2020-05-26，如有侵权请联系 cloudcommunity@tencent.com 删除

SSL 证书