Skip to main content

How Images are Built and Cached

Define an image specification first​

When you write a task environment with flyte.Image, you are describing the image that Flyte should build; you are not constructing an already-built container object. Start with one of the from_* constructors, then add immutable layers with with_* methods:

from pathlib import Path

import flyte
from flyte import Image

image = (
    Image.from_debian_base(install_flyte=False)
    .with_apt_packages("vim", "wget")
    .with_pip_packages("mypy", "httpx")
    .with_env_vars({"hello": "world1"})
    .with_dockerignore(Path(__file__).parent / ".dockerignore")
    .with_local_v2()
)

env = flyte.TaskEnvironment(name="t1", image=image)

This is the pattern used in examples/image/base_image.py. Image is a frozen dataclass in src/flyte/_image.py; every with_* method calls clone(addl_layer=...) and returns a new Image. The original specification is therefore unchanged when you add a package, environment variable, source file, or other layer.

Do not call Image(...) directly. Image.__post_init__() rejects direct construction, and the private _new() constructor supplies the guard used internally. Use constructors such as Image.from_debian_base(), Image.from_base(), Image.from_uv_script(), Image.from_dockerfile(), or Image.from_ref_name().

Make a pre-built base buildable​

A base URI created by from_base() has no image name. Give it a target name with clone() before adding layers or invoking the normal build flow:

from flyte import Image

image = Image.from_base("ghcr.io/flyteorg/flyte:py3.12-v2.0.0b56").clone(
    name="my-flyte-image",
)

Image.clone() sets the new object's internal _is_cloned flag to True, which moves it into the normal existence-check/build path. Adding a layer also requires the source image to be extendable and to have a name. Ordinary clones default to extendable=False; create an extendable clone when you intend to add layers:

extendable_image = image.clone(name="my-extendable-image", extendable=True)
image_with_packages = extendable_image.with_pip_packages("requests", "pandas")

Dockerfile images are intentionally not extendable. Image.from_dockerfile() sets extendable=False, and clone() rejects additional layers for an image with a Dockerfile. Put all setup in the Dockerfile instead. The constructor accepts a registry, name, and one or more platforms, as in examples/basics/using_dockerfiles.py:

from pathlib import Path

import flyte

image = flyte.Image.from_dockerfile(
    file=Path(__file__).parent / "Dockerfile.example",
    name="flyte-temp-dockerfile-example",
    registry="ghcr.io/flyteorg",
    platform=("linux/amd64", "linux/arm64"),
)

How the content tag is calculated​

For a cloned image without an explicit internal tag, Image._final_tag is the hexadecimal digest returned by _get_hash_digest(). The digest is an MD5 hash updated in this order:

  1. The base_image string, when present.
  2. The contents of the Dockerfile, when present; the Dockerfile path itself is not hashed.
  3. Each layer's update_hash() representation, in layer order.

File-backed layers hash file contents. Source-directory hashing also applies the .dockerignore selected by a DockerIgnore layer, or the .dockerignore under the initialized root directory when no such layer is present. Consequently, changing a copied source file changes the tag, and changing layer order or relevant ignored files can change it as well. The registry and name are not part of this digest, so the same specification can retain its content tag when moved to another target.

The resulting URI comes from Image.uri. An unmodified image returns its original base_image. A cloned image requires a name and becomes <registry>/<name>:<final_tag>, or <name>:<final_tag> when no registry is set. If _tag is set, it overrides the digest; if neither a tag nor hash content is available, _final_tag falls back to latest.

The content-sensitive behavior is covered by tests/flyte/test_image.py:

f = tmp_path / "version.md"
f.write_text("first")
base = Image.from_debian_base(registry="localhost", name="test-image", flyte_version="0.2.0b14")
h1 = base.with_source_file(f)._get_hash_digest()
f.write_text("second")
h2 = base.with_source_file(f)._get_hash_digest()
assert h1 != h2

Although _get_hash_digest() and _tag are internal members, the tag override is used by the repository's CI image script when a release or CI target must have a supplied tag:

from flyte import Image
from flyte.extend import ImageBuildEngine

registry, name, tag = parse_target(to_target)
image = Image.from_base(from_image).clone(registry=registry, name=name)
object.__setattr__(image, "_tag", tag)
result = await ImageBuildEngine.build(image, builder=builder, force=force)

This is the exact assignment used in examples/image/ci_build_image.py; it is an advanced, internal-tag override rather than the usual application API.

The build decision and cache lookup​

Build directly through the public synchronous API when you have an Image and want an ImageBuild result:

if __name__ == "__main__":
    flyte.init_from_config()
    result = flyte.build(image, force=True, wait=False)
    print(f"URI: {result.uri}")
    print(f"Remote run: {result.remote_run}")

flyte.build() is defined in src/flyte/_build.py as a syncified wrapper around an async function. It forwards image, dry_run, force, and wait to ImageBuildEngine.build(). Async callers use flyte.build.aio(image, ...).

ImageBuildEngine is publicly available as flyte.extend.ImageBuildEngine, although its implementation is in src/flyte/_internal/imagebuild/image_builder.py. Its cached build() method makes decisions in this order:

  • With force=True or dry_run=True, it skips the existence check and uses image.uri as the candidate URI.
  • Otherwise it calls the cached image_exists() method. A non-None result is a cache hit: the engine returns ImageBuild(uri=existing_uri, remote_run=None) and does not call a builder.
  • On a miss, it calls image.validate(), which validates every layer, selects a builder, and invokes that builder's build_image() method.

The central cache behavior is explicit in tests/flyte/imagebuild/test_image_build_engine.py: an existing URI causes build_image not to be called, while force=True does not call image_exists() and passes force=True through to the builder.

Existence-check special cases​

image_exists() does not query a registry for every image:

  • An unmodified Flyte default image is treated as an image published with the Flyte release and returns image.uri immediately.
  • An image with no name returns its URI without checking. This is why a from_base() image must be cloned and named to participate in the normal target-image flow.
  • A latest tag skips checking. The normal build path consequently always builds images tagged latest.

For other cloned images, the engine forms the repository from image.registry and image.name, then asks the selected builder's checkers whether the tag exists for all requested platforms. If a checker raises, the engine logs the error and tries the next checker. If every checker fails, it assumes the image exists and returns image.uri; this behavior can turn Docker, Podman, registry, or network failures into a skipped build.

Both image_exists() and build() use @alru_cache. Tests clear the build cache with ImageBuildEngine.build.cache_clear() when isolating cases. If you change initialization configuration or checker behavior in a long-running process, account for those cached results.

Validation is deferred until the build call. Missing requirements files, source paths, UV metadata, project files, or lock files may therefore remain unnoticed while constructing the fluent specification. with_source_file() does perform one immediate check: a list containing duplicate filenames for the same destination raises ValueError rather than allowing the files to overwrite each other.

Local and remote builders​

The engine chooses the builder from the per-call builder argument, the configured image_builder, or the local default. _get_builder() recognizes "local" and "remote"; other names are loaded from the flyte.plugins.image_builders entry-point group.

The local builder uses Docker/BuildKit and Podman-compatible image checkers. Its checks use the requested image.platform values: manifest lists must contain every requested platform, while the current checker accepts a single-platform manifest regardless of the requested architecture. The local builder returns an ImageBuild with the built URI and remote_run=None.

The remote builder submits Flyte's system build-image task. With wait=True, the result contains the completed fully qualified URI and the remote run. With wait=False, the call returns immediately with uri=None and a remote_run that identifies the submitted build:

result = flyte.build(image, force=True, wait=False)
print(f"URI: {result.uri}")
print(f"Remote run: {result.remote_run}")

The ImageBuild dataclass in src/flyte/_build.py contains only uri and remote_run. A local result, an existence-cache hit, and a completed build have remote_run=None; a non-waiting remote build can have uri=None. force=True also reaches remote builders as the overwrite-cache request described by ImageBuildEngine.build().

For lower-level integrations, select a builder and use the async engine directly:

from flyte.extend import ImageBuildEngine

result = await ImageBuildEngine.build(image, builder="remote", wait=False)

Remote operation requires initialized Flyte client configuration and an enabled system build-image task. The task domain defaults to production and can be controlled by FLYTE_IMAGEBUILDER_TASK_DOMAIN.

Deployment and configured image references​

Deployment uses the same build engine, but aggregates results rather than returning one result to the caller. _deploy.build_images() resolves deferred code-bundle layers, starts builds for environment images, gathers them, and returns an ImageCache containing image_lookup and any asynchronous build_run_ids.

A reference image can bypass building entirely. Image.from_ref_name() stores its name as _ref_name; deployment resolves that name against configured image references, substitutes the configured URI as base_image, and, when there are no additional layers, records that URI directly in image_lookup. An unknown reference raises ValueError and lists the available names. This supports pre-built-image configuration without asking a builder to rebuild it.

Finally, the direct-construction example still present in the flyte.build() docstring (flyte.Image("example_image")) is stale: the guarded Image implementation rejects that form. Use a from_* constructor, and clone a pre-built base when you need to add layers or build a named target.