2020/06/29

K8sで動くWorkflow管理ツール「Argo」をEKSに導入してみた

k8seksargo

概要

定期的に実行するjobなどの管理が増えてきそうな感じがしました。

以前、apache airflowの導入を行った実績があるので、ariflowでも良いかなと思ったのですが

管理系のツールはEKSを立てて運用している背景もあり、せっかくなので、k8s nativeなworkflow管理ができるArgoを導入することに決めました。

以下実施内容です。

実施内容

クライアントツールのインストール

% brew install argoproj/tap/argo
% argo version
argo: 2.8.2+8a151ae.dirty
  BuildDate: 2020-06-18T23:53:01Z
  GitCommit: 8a151aec6538c9442cf2380c2544ba3efb60ff60
  GitTreeState: dirty
  GitTag: 2.8.2
  GoVersion: go1.13
  Compiler: gc
  Platform: darwin/amd64

初期設定

namespaceの作成

apiVersion: v1
kind: Namespace
metadata:
  name: argo

manifestの作成

% kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
  • https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml の中身です。
## see https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterworkflowtemplates.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: ClusterWorkflowTemplate
    plural: clusterworkflowtemplates
    shortNames:
      - clusterwftmpl
      - cwft
  scope: Cluster
  version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: cronworkflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: CronWorkflow
    plural: cronworkflows
    shortNames:
      - cronwf
      - cwf
  scope: Namespaced
  version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  additionalPrinterColumns:
    - JSONPath: .status.phase
      description: Status of the workflow
      name: Status
      type: string
    - JSONPath: .status.startedAt
      description: When the workflow was started
      format: date-time
      name: Age
      type: date
  group: argoproj.io
  names:
    kind: Workflow
    plural: workflows
    shortNames:
      - wf
  scope: Namespaced
  version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: workflowtemplates.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: WorkflowTemplate
    plural: workflowtemplates
    shortNames:
      - wftmpl
  scope: Namespaced
  version: v1alpha1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-role
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
  name: argo-aggregate-to-admin
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workflows/finalizers
      - workflowtemplates
      - workflowtemplates/finalizers
      - cronworkflows
      - cronworkflows/finalizers
      - clusterworkflowtemplates
      - clusterworkflowtemplates/finalizers
    verbs:
      - create
      - delete
      - deletecollection
      - get
      - list
      - patch
      - update
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: argo-aggregate-to-edit
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workflows/finalizers
      - workflowtemplates
      - workflowtemplates/finalizers
      - cronworkflows
      - cronworkflows/finalizers
      - clusterworkflowtemplates
      - clusterworkflowtemplates/finalizers
    verbs:
      - create
      - delete
      - deletecollection
      - get
      - list
      - patch
      - update
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: argo-aggregate-to-view
rules:
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workflows/finalizers
      - workflowtemplates
      - workflowtemplates/finalizers
      - cronworkflows
      - cronworkflows/finalizers
      - clusterworkflowtemplates
      - clusterworkflowtemplates/finalizers
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-cluster-role
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/exec
    verbs:
      - create
      - get
      - list
      - watch
      - update
      - patch
      - delete
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - ""
    resources:
      - persistentvolumeclaims
    verbs:
      - create
      - delete
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workflows/finalizers
    verbs:
      - get
      - list
      - watch
      - update
      - patch
      - delete
      - create
  - apiGroups:
      - argoproj.io
    resources:
      - workflowtemplates
      - workflowtemplates/finalizers
      - clusterworkflowtemplates
      - clusterworkflowtemplates/finalizers
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - serviceaccounts
    verbs:
      - get
      - list
  - apiGroups:
      - argoproj.io
    resources:
      - cronworkflows
      - cronworkflows/finalizers
    verbs:
      - get
      - list
      - watch
      - update
      - patch
      - delete
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
  - apiGroups:
      - policy
    resources:
      - poddisruptionbudgets
    verbs:
      - create
      - get
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-server-cluster-role
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/exec
      - pods/log
    verbs:
      - get
      - list
      - watch
      - delete
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
  - apiGroups:
      - argoproj.io
    resources:
      - workflows
      - workflowtemplates
      - cronworkflows
      - clusterworkflowtemplates
    verbs:
      - create
      - get
      - list
      - watch
      - update
      - patch
      - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-role
subjects:
  - kind: ServiceAccount
    name: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-cluster-role
subjects:
  - kind: ServiceAccount
    name: argo
    namespace: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-server-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-server-cluster-role
subjects:
  - kind: ServiceAccount
    name: argo-server
    namespace: argo
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
---
apiVersion: v1
kind: Service
metadata:
  name: argo-server
spec:
  ports:
    - name: web
      port: 2746
      targetPort: 2746
  selector:
    app: argo-server
---
apiVersion: v1
kind: Service
metadata:
  name: workflow-controller-metrics
spec:
  ports:
    - name: metrics
      port: 9090
      protocol: TCP
      targetPort: 9090
  selector:
    app: workflow-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
    spec:
      containers:
        - args:
            - server
          image: argoproj/argocli:v2.8.2
          name: argo-server
          ports:
            - containerPort: 2746
              name: web
          readinessProbe:
            httpGet:
              path: /
              port: 2746
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 20
      serviceAccountName: argo-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: workflow-controller
spec:
  selector:
    matchLabels:
      app: workflow-controller
  template:
    metadata:
      labels:
        app: workflow-controller
    spec:
      containers:
        - args:
            - --configmap
            - workflow-controller-configmap
            - --executor-image
            - argoproj/argoexec:v2.8.2
          command:
            - workflow-controller
          image: argoproj/workflow-controller:v2.8.2
          name: workflow-controller
      serviceAccountName: argo

argo cli用のservice accountの作成

  • default という名前の service accountを作成して、admin権限を設定します。
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-admin
  namespace: argo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: admin
subjects:
  - kind: ServiceAccount
    name: default
    namespace: argo

Hello World

hello-world.yaml として作成します。

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: false
spec:
  entrypoint: whalesay
  templates:
    - name: whalesay
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["hello world"]

このファイルを以下のように実行します。

% argo submit -n argo --watch hello-world.yaml

以下のように実行結果が確認できました。

% argo logs $(argo get hello-world-bmd24 -n argo -o name) -c main -n argo
hello-world-bmd24:  _____________ 
hello-world-bmd24: < hello world >
hello-world-bmd24:  ------------- 
hello-world-bmd24:     \
hello-world-bmd24:      \
hello-world-bmd24:       \     
hello-world-bmd24:                     ##        .            
hello-world-bmd24:               ## ## ##       ==            
hello-world-bmd24:            ## ## ## ##      ===            
hello-world-bmd24:        /""""""""""""""""___/ ===        
hello-world-bmd24:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
hello-world-bmd24:        \______ o          __/            
hello-world-bmd24:         \    \        __/             
hello-world-bmd24:           \____\______/   

Access to Argo UI

HTTPSを有効にしたかったのと、IP制限などの細かい要求に答えるために、公式で記載のある以下のやり方は採用しませんでした。

% kubectl patch svc argo-ui -n argo -p '{"spec": {"type": "LoadBalancer"}}'

実際には argo-server:2746 にproxyするhttpsを有効にしたnginxのdeploymentを作成してdeployします。
さらに、network load balancer(nlb)を作成する serviceを作成して、外部からアクセスできるようにしました。

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: argo
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https
  selector:
    app: nginx

以上になります。