Skip to main content

Flyte Task Execution Deployment Model

This deployment diagram illustrates the runtime execution architecture of a Flyte task authored using the Flyte Python SDK.

The workflow begins on the Developer Workstation, where the user authors tasks with task environments, images, and Kubernetes pod configurations (PodTemplate). Using the Flyte CLI/SDK (ImageBuildEngine), task container images are built locally (or remotely via RemoteImageBuilder) and pushed to a Container Registry (such as GitHub Container Registry, ECR, or Docker Hub).

When a task execution or run is triggered:

  1. The Flyte Python SDK serializes task templates, native inputs, and execution metadata, uploading input payloads to Cloud Storage (S3, GCS, or Azure Blob Storage).
  2. The SDK's RemoteController submits an action request to the Flyte Backend / Control Plane (FlyteAdmin / Union gRPC API).
  3. The Flyte Backend orchestrates execution on a Kubernetes Cluster, dispatching the task to run within a dedicated Kubernetes Pod.
  4. The Kubernetes Pod pulls the task image from the Container Registry, runs the Flyte task entrypoint container (alongside any sidecars or custom mounts defined in PodTemplate), downloads inputs from Cloud Storage, executes the user logic, and writes output literals/artifacts back to Cloud Storage.
  5. Execution status and completion signals are reported back to the Flyte Backend.

Key Architectural Findings:

  • Developers author tasks decorated with @env.task using Image (Debian base, UV scripts, or custom Docker images) and optional PodTemplate (Kubernetes V1PodSpec).
  • The Flyte CLI / SDK builds images via ImageBuildEngine / DockerImageBuilder / RemoteImageBuilder and pushes them to a container registry (e.g., ghcr.io, ECR, GCR).
  • RemoteController and ControllerClient handle communication with Flyte Backend / Control Plane gRPC endpoints for task submission and lifecycle monitoring.
  • Task inputs and outputs are serialized/deserialized and stored in Object/Cloud Storage (flyte.storage.S3, GCS, ABFS, or local sandbox storage) via remote filesystems (fsspec/RemoteFSPathResolver).
  • The Flyte backend orchestrates the task as a Kubernetes Pod executing the Flyte entrypoint in the primary container with configured resources and secrets.
Loading diagram...