Skip to main content

Workflow Registration and Execution Flow

This sequence diagram illustrates the dual flows of Workflow Registration and Workflow Execution within the Flyte SDK.

Registration Flow​

The registration flow begins when a user calls flyte.deploy(). The SDK packages the environment and its tasks into a DeploymentPlan. Each task, represented by a TaskTemplate, is translated into a wire-format TaskSpec. The ClientSet (acting as the Remote Orchestration & Management) then communicates with the remote Flyte Admin service via the TaskService to register the task definition.

Execution Flow​

The execution flow is triggered by flyte.run(). The _Runner component manages the lifecycle of the remote execution:

  1. Image & Code Bundling: It ensures that the necessary container images are built and the source code is bundled (either as a .tgz or .pkl file).
  2. Type Validation: The TypeEngine is invoked to validate the user's native Python arguments against the task's NativeInterface. It transforms these arguments into Flyte Core Type System Architecture.
  3. Input Upload: Before triggering the run, the SDK uploads the transformed inputs to the Flyte backend using the DataProxyService.
  4. Execution Trigger: Finally, the ClientSet calls RunService.create_run() to initiate the execution on the Flyte cluster. The call returns a Run object (representing the Core Execution Framework), which the user can use to track status, logs, and outputs.

Key architectural components discovered include the ClientSet for unified API access, the TypeEngine for extensible type transformations, and the TaskTemplate as the primary unit of definition for both tasks and workflows in this SDK version.

Key Architectural Findings:

  • The SDK uses a ClientSet class as a unified entry point for all Flyte Admin services (Task, Run, DataProxy, etc.).
  • TaskTemplate serves as the core definition entity, encompassing what was traditionally split between tasks and workflows.
  • The TypeEngine handles the complex mapping between native Python types and Flyte's protobuf-based Literal system.
  • Remote execution involves a multi-step process: image building, code bundling, input transformation, input uploading via DataProxy, and finally run creation.
  • The Run class acts as the handle for an active or completed execution, providing methods to wait for completion, fetch logs, and retrieve outputs.
Loading diagram...