Workflow Execution and Registration Flow

This diagram illustrates the sequence of events for both registering a new workflow and executing it using the Flytekit SDK and CLI.

Registration Flow:

1. A developer uses the flyte deploy command to register their workflow code.
2. The CLI invokes the flyte.deploy function within the Flytekit SDK.
3. The SDK then plans the deployment, which involves analyzing the tasks and their dependencies.
4. The apply function orchestrates the deployment, calling _deploy_task_env and then _deploy_task for each individual task within the environment.
5. Finally, a DeployTaskRequest is sent to the Flyte Backend API to create or update the task registration.

Execution Flow:

1. A user or developer initiates a remote execution using the flyte run --remote command, specifying the task and its inputs.
2. The CLI uses the with_runcontext and run methods in the SDK to prepare the execution.
3. The internal _Runner class's _run_remote method is called.
4. The SDK first sends an UploadInputsRequest to the Flyte backend's dataproxy service to offload any large input data.
5. With the inputs uploaded, the SDK sends a CreateRunRequest to the backend's run service to launch the workflow.
6. The backend returns a CreateRunResponse, and the SDK provides a Run object back to the CLI, which then displays the execution URL and status to the user.

Key Architectural Findings:

• The flyte deploy command initiates the registration process, which involves a series of internal SDK calls culminating in a DeployTaskRequest to the backend.
• The core registration logic is found in flyte._deploy.py, with _deploy_task being the function that makes the final API call.
• The flyte run --remote command triggers the execution flow.
• Execution involves a _Runner class within flyte._run.py that handles the remote execution.
• Remote execution is a two-step process: first uploading inputs via dataproxy_service.upload_inputs and then creating the run via run_service.create_run.

Loading diagram...