Skip to main content

Local vs. Remote Building

Local vs. Remote Image Building​

Flyte workflows often require custom container images to execute tasks. The flyte-sdk provides flexible mechanisms for building these images, offering both local and remote strategies. This choice allows developers to optimize for development speed, resource utilization, and consistency across different environments. The central component orchestrating this process is the ImageBuildEngine.

ImageBuildEngine: The Orchestrator​

The ImageBuildEngine class (flyte.extend.ImageBuildEngine) serves as the primary interface for building container images within flyte-sdk. It intelligently determines whether an image needs to be built (by checking its existence in a registry) and then dispatches the build request to the appropriate builder implementation.

The core method for initiating an image build is ImageBuildEngine.build:

@classmethod
async def build(
    cls,
    image: Image,
    builder: ImageBuildEngine.ImageBuilderType | None = None,
    dry_run: bool = False,
    force: bool = False,
    wait: bool = True,
) -> "ImageBuild":
    pass

This method takes an Image object (representing the desired container image specification) and optional parameters to control the build process, such as dry_run to simulate a build, force to skip existence checks and rebuild, and wait to await build completion. If the image already exists in the registry and force is not True, the build is skipped.

The ImageBuildEngine selects the concrete builder implementation via its internal _get_builder method. This method inspects the builder argument or the global configuration to return either a DockerImageBuilder for local builds or a RemoteImageBuilder for remote builds:

@classmethod
def _get_builder(cls, builder: ImageBuildEngine.ImageBuilderType | None = "local") -> ImageBuilder:
    if builder is None:
        builder = "local"
    if builder == "remote":
        from flyte._internal.imagebuild.remote_builder import RemoteImageBuilder
        return RemoteImageBuilder()
    elif builder == "local":
        from flyte._internal.imagebuild.docker_builder import DockerImageBuilder
        return DockerImageBuilder()
    # ... custom builder loading ...

While flyte.build.aio is commonly used for parallel image builds, you can directly invoke the engine, as seen in examples/advanced/fractal.py:

import flyte
from flyte._internal.imagebuild.image_builder import ImageBuildEngine

# Assuming 'env.image' is an Image object defined elsewhere
async def main_builder():
    result = await ImageBuildEngine.build(env.image)
    print(f"Built image URI: {result.uri}")
    return result

Local Image Building with DockerImageBuilder​

The DockerImageBuilder (flyte._internal.imagebuild.docker_builder.DockerImageBuilder) handles image builds directly on your local machine using Docker and BuildKit. This approach is ideal for local development and testing, leveraging your existing Docker environment.

When DockerImageBuilder is used, it performs several key steps:

  1. BuildKit Setup: It ensures that a Docker BuildKit builder named flytex is available and correctly configured with network=host via _ensure_buildx_builder. If not, it creates or reconfigures it.
  2. Dockerfile Generation: For images defined without an explicit Dockerfile, it dynamically generates one. This involves starting from a base image, setting up a Python virtual environment, and then processing various image layers (e.g., with_pip_packages, with_apt_packages, with_source) to add dependencies and source code. This process uses templates like DOCKER_FILE_UV_BASE_TEMPLATE and DOCKER_FILE_BASE_FOOTER.
  3. Docker Build Execution: It constructs and executes a docker buildx build command, passing the generated Dockerfile (or a provided one), build context, tags, platforms, and any necessary secrets.

The build_image method orchestrates this process:

async def build_image(
    self, image: Image, dry_run: bool = False, wait: bool = True, force: bool = False
) -> "ImageBuild":
    pass

Here's an example from tests/flyte/imagebuild/test_docker_builder.py demonstrating its usage:

from flyte.core.image import Image
from flyte._internal.imagebuild.docker_builder import DockerImageBuilder

async def test_basic_image():
    img = Image.from_debian_base(registry="localhost:30000", name="test_image", install_flyte=False).with_pip_packages(
        "requests"
    )
    builder = DockerImageBuilder()
    await builder.build_image(img, dry_run=False)

During the build process, DockerImageBuilder logs the temporary directory used for the build context, which can be helpful for debugging: Temporary directory: {tmp_dir}.

Remote Image Building with RemoteImageBuilder​

The RemoteImageBuilder (flyte._internal.imagebuild.remote_builder.RemoteImageBuilder) offloads the image building process to a remote Flyte service. This is particularly useful for ensuring consistent build environments, leveraging cloud resources, and avoiding local machine resource contention.

Instead of executing Docker commands locally, RemoteImageBuilder interacts with a Flyte backend. Its build_image method performs the following:

  1. Configuration Validation: It validates the image specification and build context for remote execution.
  2. Remote Task Invocation: It retrieves a predefined Flyte task (identified by IMAGE_TASK_NAME, IMAGE_TASK_PROJECT, IMAGE_TASK_DOMAIN) and executes it remotely using flyte.with_runcontext(...).run.aio. This remote task is responsible for performing the actual image build.
  3. Status Monitoring: It monitors the remote build task's progress. If wait is True, it awaits the completion of the remote task and raises an ImageBuildError if the build fails. If wait is False, it returns immediately with a remote_run object that can be used to track the build status asynchronously.
async def build_image(
    self, image: Image, dry_run: bool = False, wait: bool = True, force: bool = False
) -> "ImageBuild":
    pass

An example from tests/flyte/imagebuild/test_image_build_engine.py illustrates its use:

from flyte.core.image import Image
from flyte._internal.imagebuild.remote_builder import RemoteImageBuilder

async def test_remote_build_example():
    builder = RemoteImageBuilder()
    img = Image.from_debian_base()
    await builder.build_image(img)

If the remote image builder is not enabled or configured correctly in the Flyte backend, attempting to use it will result in a flyte.errors.ImageBuildError indicating that the remote builder is not enabled.

Configuring the Image Builder​

You can specify which image builder flyte-sdk should use by setting the image_builder configuration option. This can be done through flyte.init_from_config or environment variables.

  • local (default): Uses the DockerImageBuilder to build images on your local machine.
  • remote: Uses the RemoteImageBuilder to offload image builds to a configured Flyte service.

For example, to configure flyte-sdk to use the remote builder:

import flyte

# Configure Flyte to use the remote image builder
flyte.init_from_config(config_dict={
    "image_builder": "remote",
    # ... other configuration ...
})

# Or via environment variable:
# export FLYTE_IMAGE_BUILDER=remote

Tradeoffs and Considerations​

Choosing between local and remote image building involves several tradeoffs:

  • Local (DockerImageBuilder):

    • Pros: Faster iteration during local development, direct control over the build environment, no dependency on a remote Flyte cluster for builds. Requires only Docker to be installed locally.
    • Cons: Consumes local machine resources (CPU, memory, disk), builds might not be consistent with production environments if local Docker setup differs, potential for build failures due to local environment issues.

  • Remote (RemoteImageBuilder):

    • Pros: Offloads resource-intensive builds to a dedicated service, ensures consistent build environments (as builds happen in the same remote service), leverages cloud scalability, ideal for CI/CD pipelines.
    • Cons: Requires a configured Flyte backend with the remote image building service enabled, introduces network latency for build submission and status retrieval, debugging build issues can be more complex as they occur remotely.

Regardless of the chosen builder, if an image existence check fails (e.g., the image is not found in the registry), flyte-sdk will log a warning such as Image not found: {repository}:{tag}.