Airflow Kubernetes Secret: Streamlining Your Workflow

As you explore the world of Apache Airflow and its integration with Kubernetes, one crucial aspect that often goes unnoticed is the creation of custom pod template files. These templates enable you to recycle base values between multiple tasks, making your workflow more efficient and streamlined.

One such example of a pod template file is provided below:

```yaml

apiVersion: v1

kind: Pod

metadata:

spec:

containers:

- env:

- name: AIRFLOW_CORE_EXECUTOR

value: LocalExecutor

- name: AIRFLOW_CORE_FERNET_KEY

valueFrom:

secretKeyRef:

key: fernet-key

- name: AIRFLOW_DATABASE_SQL_ALCHEMY_CONN

valueFrom:

secretKeyRef:

key: connection

- name: AIRFLOW_CONN_AIRFLOW_DB

valueFrom:

secretKeyRef:

key: connection

image: dummy_image

imagePullPolicy: IfNotPresent

volumeMounts:

- mountPath: /opt/airflow/logs

- mountPath: /opt/airflow/airflow.cfg

readOnly: true

subPath: airflow.cfg

restartPolicy: Never

securityContext:

runAsUser: 50000

fsGroup: 50000

serviceAccountName: 'RELEASE-NAME-worker-serviceaccount'

volumes:

- emptyDir:

```

This template file provides a foundation for your pods, allowing you to customize and recycle the same base values between multiple tasks.

In addition to custom pod template files, configuration information specific to the Kubernetes Executor also needs to be specified in the Airflow Configuration file. This includes worker namespace and image information, which can be customized on a per-task basis using the Executor config.

One example of an Airflow deployment running on a distributed set of five nodes in a Kubernetes cluster is shown below. Consistent with the regular Airflow architecture, the Workers need access to the DAG files to execute the tasks within those DAGs and interact with the Metadata repository.

The worker pod then runs the task, reports the result, and terminates. When a DAG submits a task, the KubernetesExecutor requests a worker pod from the Kubernetes API. The KubernetesExecutor requires a non-SQLite database in the backend, which does not necessarily need to be running on Kubernetes but does need access to a Kubernetes cluster.

As of Airflow 2.7.0, you can install the cncf.kubernetes provider package to use this feature. You can also install Airflow with the cncf.kubernetes extras by installing apache-airflow-providers-cncf-kubernetes>=7.4.0.

In conclusion, custom pod template files and Kubernetes Executor configuration are essential components of a successful Airflow deployment on a distributed set of nodes in a Kubernetes cluster. By streamlining your workflow and utilizing these features, you can take your data processing and analytics to the next level.

Airflow kubernetes secret