2020/06/29
K8sで動くWorkflow管理ツール「Argo」をEKSに導入してみた
概要
定期的に実行するjobなどの管理が増えてきそうな感じがしました。
以前、apache airflowの導入を行った実績があるので、ariflowでも良いかなと思ったのですが
管理系のツールはEKSを立てて運用している背景もあり、せっかくなので、k8s nativeなworkflow管理ができるArgoを導入することに決めました。
以下実施内容です。
実施内容
クライアントツールのインストール
- Releases · argoproj/argo · GitHubをみてインストールします。
% brew install argoproj/tap/argo
% argo version
argo: 2.8.2+8a151ae.dirty
BuildDate: 2020-06-18T23:53:01Z
GitCommit: 8a151aec6538c9442cf2380c2544ba3efb60ff60
GitTreeState: dirty
GitTag: 2.8.2
GoVersion: go1.13
Compiler: gc
Platform: darwin/amd64
初期設定
namespaceの作成
apiVersion: v1
kind: Namespace
metadata:
name: argo
manifestの作成
% kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yamlの中身です。
## see https://raw.githubusercontent.com/argoproj/argo/stable/manifests/install.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: clusterworkflowtemplates.argoproj.io
spec:
group: argoproj.io
names:
kind: ClusterWorkflowTemplate
plural: clusterworkflowtemplates
shortNames:
- clusterwftmpl
- cwft
scope: Cluster
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronworkflows.argoproj.io
spec:
group: argoproj.io
names:
kind: CronWorkflow
plural: cronworkflows
shortNames:
- cronwf
- cwf
scope: Namespaced
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: workflows.argoproj.io
spec:
additionalPrinterColumns:
- JSONPath: .status.phase
description: Status of the workflow
name: Status
type: string
- JSONPath: .status.startedAt
description: When the workflow was started
format: date-time
name: Age
type: date
group: argoproj.io
names:
kind: Workflow
plural: workflows
shortNames:
- wf
scope: Namespaced
version: v1alpha1
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: workflowtemplates.argoproj.io
spec:
group: argoproj.io
names:
kind: WorkflowTemplate
plural: workflowtemplates
shortNames:
- wftmpl
scope: Namespaced
version: v1alpha1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argo-role
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
name: argo-aggregate-to-admin
rules:
- apiGroups:
- argoproj.io
resources:
- workflows
- workflows/finalizers
- workflowtemplates
- workflowtemplates/finalizers
- cronworkflows
- cronworkflows/finalizers
- clusterworkflowtemplates
- clusterworkflowtemplates/finalizers
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
rbac.authorization.k8s.io/aggregate-to-edit: "true"
name: argo-aggregate-to-edit
rules:
- apiGroups:
- argoproj.io
resources:
- workflows
- workflows/finalizers
- workflowtemplates
- workflowtemplates/finalizers
- cronworkflows
- cronworkflows/finalizers
- clusterworkflowtemplates
- clusterworkflowtemplates/finalizers
verbs:
- create
- delete
- deletecollection
- get
- list
- patch
- update
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: argo-aggregate-to-view
rules:
- apiGroups:
- argoproj.io
resources:
- workflows
- workflows/finalizers
- workflowtemplates
- workflowtemplates/finalizers
- cronworkflows
- cronworkflows/finalizers
- clusterworkflowtemplates
- clusterworkflowtemplates/finalizers
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: argo-cluster-role
rules:
- apiGroups:
- ""
resources:
- pods
- pods/exec
verbs:
- create
- get
- list
- watch
- update
- patch
- delete
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- watch
- list
- apiGroups:
- ""
resources:
- persistentvolumeclaims
verbs:
- create
- delete
- apiGroups:
- argoproj.io
resources:
- workflows
- workflows/finalizers
verbs:
- get
- list
- watch
- update
- patch
- delete
- create
- apiGroups:
- argoproj.io
resources:
- workflowtemplates
- workflowtemplates/finalizers
- clusterworkflowtemplates
- clusterworkflowtemplates/finalizers
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- get
- list
- apiGroups:
- argoproj.io
resources:
- cronworkflows
- cronworkflows/finalizers
verbs:
- get
- list
- watch
- update
- patch
- delete
- apiGroups:
- ""
resources:
- events
verbs:
- create
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- create
- get
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: argo-server-cluster-role
rules:
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- watch
- list
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- pods/exec
- pods/log
verbs:
- get
- list
- watch
- delete
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- argoproj.io
resources:
- workflows
- workflowtemplates
- cronworkflows
- clusterworkflowtemplates
verbs:
- create
- get
- list
- watch
- update
- patch
- delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: argo-role
subjects:
- kind: ServiceAccount
name: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: argo-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: argo-cluster-role
subjects:
- kind: ServiceAccount
name: argo
namespace: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: argo-server-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: argo-server-cluster-role
subjects:
- kind: ServiceAccount
name: argo-server
namespace: argo
---
apiVersion: v1
kind: ConfigMap
metadata:
name: workflow-controller-configmap
---
apiVersion: v1
kind: Service
metadata:
name: argo-server
spec:
ports:
- name: web
port: 2746
targetPort: 2746
selector:
app: argo-server
---
apiVersion: v1
kind: Service
metadata:
name: workflow-controller-metrics
spec:
ports:
- name: metrics
port: 9090
protocol: TCP
targetPort: 9090
selector:
app: workflow-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: argo-server
spec:
selector:
matchLabels:
app: argo-server
template:
metadata:
labels:
app: argo-server
spec:
containers:
- args:
- server
image: argoproj/argocli:v2.8.2
name: argo-server
ports:
- containerPort: 2746
name: web
readinessProbe:
httpGet:
path: /
port: 2746
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 20
serviceAccountName: argo-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: workflow-controller
spec:
selector:
matchLabels:
app: workflow-controller
template:
metadata:
labels:
app: workflow-controller
spec:
containers:
- args:
- --configmap
- workflow-controller-configmap
- --executor-image
- argoproj/argoexec:v2.8.2
command:
- workflow-controller
image: argoproj/workflow-controller:v2.8.2
name: workflow-controller
serviceAccountName: argo
argo cli用のservice accountの作成
defaultという名前の service accountを作成して、admin権限を設定します。
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: default-admin
namespace: argo
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- kind: ServiceAccount
name: default
namespace: argo
Hello World
hello-world.yaml として作成します。
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-world-
labels:
workflows.argoproj.io/archive-strategy: false
spec:
entrypoint: whalesay
templates:
- name: whalesay
container:
image: docker/whalesay:latest
command: [cowsay]
args: ["hello world"]
このファイルを以下のように実行します。
% argo submit -n argo --watch hello-world.yaml
以下のように実行結果が確認できました。
% argo logs $(argo get hello-world-bmd24 -n argo -o name) -c main -n argo
hello-world-bmd24: _____________
hello-world-bmd24: < hello world >
hello-world-bmd24: -------------
hello-world-bmd24: \
hello-world-bmd24: \
hello-world-bmd24: \
hello-world-bmd24: ## .
hello-world-bmd24: ## ## ## ==
hello-world-bmd24: ## ## ## ## ===
hello-world-bmd24: /""""""""""""""""___/ ===
hello-world-bmd24: ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
hello-world-bmd24: \______ o __/
hello-world-bmd24: \ \ __/
hello-world-bmd24: \____\______/
Access to Argo UI
HTTPSを有効にしたかったのと、IP制限などの細かい要求に答えるために、公式で記載のある以下のやり方は採用しませんでした。
% kubectl patch svc argo-ui -n argo -p '{"spec": {"type": "LoadBalancer"}}'
実際には argo-server:2746 にproxyするhttpsを有効にしたnginxのdeploymentを作成してdeployします。
さらに、network load balancer(nlb)を作成する serviceを作成して、外部からアクセスできるようにしました。
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: argo
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
selector:
app: nginx
以上になります。
関連する記事
[小〜中規模向け]GKEにTiDBをデプロイする
MySQL互換のNewSQLであるTiDBをGKEにデプロイしてみました。
NATS JetStream Controllerを使ってNATSをGKEにデプロイする
helm chartのnackを使って、NATS JetStreamサーバーをデプロイして、Stream/Consumerをk8sリソースとして管理する
GKEにDragonflydbをデプロイする
redis互換のdragonflydbをGKEにデプロイしました
[GKE]Kafka Strimziをアップグレードする
GKEにデプロイしているKafka Strimzi 0.26.0を0.30.0にアップグレードする