Skip to main content

Building Container Images

When you deploy tasks to remote clusters, container images must contain all binary libraries, Python dependencies, and environment configurations your code requires. In flyte-sdk, flyte.Image provides an immutable, fluent Python API to construct container images without writing boilerplate Dockerfiles.

Building and Defining Container Images​

To define a container image, initialize a base image using a factory constructor like flyte.Image.from_debian_base() and chain layers with with_* methods. You can assign the resulting Image directly to @flyte.task or a flyte.TaskEnvironment, or build it explicitly using flyte.build().

import flyte

# Define an immutable image specification
image = (
    flyte.Image.from_debian_base(name="data-processor", registry="ghcr.io/my-org")
    .with_apt_packages("curl", "git")
    .with_pip_packages("pandas", "scikit-learn")
    .with_env_vars({"ENVIRONMENT": "production"})
)

env = flyte.TaskEnvironment(
    name="data_processing_env",
    image=image,
)

@env.task
async def process_data(data_path: str) -> int:
    import pandas as pd
    df = pd.read_csv(data_path)
    return len(df)

To build an image programmatically, pass the flyte.Image instance to flyte.build() (synchronous) or flyte.build.aio() (asynchronous):

import flyte

image = (
    flyte.Image.from_debian_base(name="worker", registry="ghcr.io/my-org")
    .with_pip_packages("requests", "numpy")
)

# Returns an ImageBuild instance with the final URI
build_result = flyte.build(image)
print(build_result.uri)

Direct instantiation via flyte.Image(...) is disallowed and raises a TypeError. You must create images using one of the factory methods on flyte.Image.

Base Image Constructors​

flyte.Image provides factory constructors targeting different image sources and package configurations:

Debian Base​

flyte.Image.from_debian_base() creates an extendable image starting from a Debian slim base image with Python pre-installed and configured with a default virtual environment (/opt/venv).

import flyte

image = flyte.Image.from_debian_base(
    name="analytics-image",
    registry="ghcr.io/my-org",
    python_version=(3, 12),
    install_flyte=True,
    platform=("linux/amd64", "linux/arm64"),
)

Pre-Built Existing Image​

flyte.Image.from_base() points to an existing image URI in a registry. You can extend this image with additional layers if configured:

import flyte

image = flyte.Image.from_base(
    image_uri="python:3.12-slim"
).clone(name="extended-python", registry="ghcr.io/my-org", extendable=True)

Dockerfile-Based Image​

flyte.Image.from_dockerfile() builds an image directly from an existing Dockerfile. Because flyte-sdk does not parse the internal layout or package manager of arbitrary Dockerfiles, Dockerfile-based images are marked non-extendable (extendable=False) and cannot be chained with additional with_* layer methods.

from pathlib import Path
import flyte

image = flyte.Image.from_dockerfile(
    file=Path("./Dockerfile"),
    registry="ghcr.io/my-org",
    name="custom-container",
    platform=("linux/amd64",),
)

Self-Contained UV Scripts​

flyte.Image.from_uv_script() constructs an image from a script containing inline metadata compliant with PEP 723:

import flyte

image = flyte.Image.from_uv_script(
    script="scripts/etl_pipeline.py",
    name="uv-script-image",
    registry="ghcr.io/my-org",
)

Customizing Image Layers​

flyte.Image is an immutable dataclass (frozen=True). Every with_* method returns a new Image instance with an updated layer sequence.

Python Dependencies​

Install packages via individual package specifications, requirements.txt, uv, or Poetry:

from pathlib import Path
import flyte

# Pip packages
image_pip = (
    flyte.Image.from_debian_base(name="pip-env", registry="localhost:30000")
    .with_pip_packages("numpy>=1.26.0", "torch", pre=False)
)

# Requirements file with custom PyPI index
image_reqs = (
    flyte.Image.from_debian_base(name="reqs-env", registry="localhost:30000")
    .with_requirements(
        file=Path("requirements.txt"),
        index_url="https://pypi.org/simple",
        extra_index_urls=["https://download.pytorch.org/whl/cpu"],
    )
)

# UV project workspace
image_uv = (
    flyte.Image.from_debian_base(name="uv-env", registry="localhost:30000")
    .with_uv_project(
        pyproject_file=Path("pyproject.toml"),
        uvlock=Path("uv.lock"),
        project_install_mode="dependencies_only",
    )
)

# Poetry project workspace
image_poetry = (
    flyte.Image.from_debian_base(name="poetry-env", registry="localhost:30000")
    .with_poetry_project(
        pyproject_file=Path("pyproject.toml"),
        poetry_lock=Path("poetry.lock"),
        project_install_mode="install_project",
    )
)

System Packages and Shell Commands​

Add apt packages or run shell commands during image build:

import flyte

image = (
    flyte.Image.from_debian_base(name="sys-env", registry="localhost:30000")
    .with_apt_packages("ffmpeg", "libsm6", "libxext6")
    .with_commands(["mkdir -p /app/data", "chmod 755 /app/data"])
)

Files, Folders, and Working Directory​

Copy source assets and set container directories:

from pathlib import Path
import flyte

image = (
    flyte.Image.from_debian_base(name="app-image", registry="localhost:30000")
    .with_workdir("/workspace")
    .with_dockerignore(Path(".dockerignore"))
    .with_source_file(Path("configs/app.yaml"), dst="/workspace/app.yaml")
    .with_source_folder(Path("./src"), dst="./src", copy_contents_only=False)
)

Build Secrets and Private Repositories​

When installing dependencies from private PyPI indexes or private Git repositories, use secret_mounts to pass secret references into the build context without embedding secrets into container layers. Secrets can be passed as strings (which resolve to secret keys) or Secret instances.

import flyte

private_repo_pkg = "git+https://$GITHUB_PAT@github.com/my-org/private-repo.git@main"

image = (
    flyte.Image.from_debian_base(name="secure-worker", registry="ghcr.io/my-org")
    .with_pip_packages(
        private_repo_pkg,
        secret_mounts=["GITHUB_PAT"],
    )
    .with_apt_packages(
        "custom-pkg",
        secret_mounts=["apt-secret"],
    )
)

Build Engine and Content-Addressable Tagging​

When an image is modified using with_* methods or clone(), flyte-sdk computes a deterministic MD5 hash (_get_hash_digest()) across all base settings, attached files, package names, and layers.

The resulting URI format is <registry>/<name>:<digest>.

import flyte

image = (
    flyte.Image.from_debian_base(name="worker", registry="ghcr.io/my-org")
    .with_pip_packages("pandas")
)

# Check URI before build
print(image.uri)
# Example: ghcr.io/my-org/worker:a1b2c3d4e5f6...

Build Execution and Registry Checks​

When executing flyte.build(image) or deploying a task environment:

  1. flyte.extend.ImageBuildEngine queries registered ImageChecker implementations (such as registry API checkers).
  2. If an image with the calculated hash tag already exists in the target container registry, the build is skipped automatically.
  3. If the image is not found, ImageBuildEngine invokes the configured builder (DockerImageBuilder for local Docker buildx, or RemoteImageBuilder for remote build tasks).
  4. Setting force=True bypasses existence checks and forces a rebuild:
import flyte

image = (
    flyte.Image.from_debian_base(name="worker", registry="ghcr.io/my-org")
    .with_pip_packages("requests")
)

# Force a rebuild of an image
result = flyte.build(image, force=True)

Troubleshooting​

TypeError on Direct Construction​

  • Symptom: TypeError: Direct instantiation of Image not allowed, please use one of the various from_...() methods instead
  • Fix: Replace flyte.Image(...) with flyte.Image.from_debian_base(...), flyte.Image.from_base(...), or flyte.Image.from_dockerfile(...).

ValueError: Cannot Add Layers to Non-Extendable Image​

  • Symptom: ValueError: Flyte current cannot add additional layers to a Dockerfile-based Image. Please amend the dockerfile directly.
  • Fix: Dockerfiles created with Image.from_dockerfile() cannot be combined with with_pip_packages() or other layer methods. Place all installation commands directly inside your Dockerfile.

Duplicate Files in with_source_file​

  • Symptom: ValueError: Multiple files with the same name would overwrite each other at destination '.'
  • Fix: When passing a list of Path objects to with_source_file(), ensure each file has a unique filename or call with_source_file() individually with explicit destination paths.

Missing Image Name for Custom Layers​

  • Symptom: ValueError: Cannot add additional layer ... to an image without name. Please first clone().
  • Fix: When starting from Image.from_base("ubuntu:22.04") without a name, call .clone(name="my-image", registry="ghcr.io/my-org", extendable=True) before appending layers.