Skip to main content

Security & Secret Management

Flyte-sdk provides the Secret class to securely inject sensitive information, such as API keys or database credentials, into your tasks and image build contexts. Secrets can be exposed as environment variables or mounted as files within the task container.

Injecting Secrets into Tasks​

You can request secrets for a task by passing them to the @task decorator. By default, Flyte-sdk makes these secrets available as environment variables.

Using Environment Variables​

If you pass a string to the secrets parameter, flyte-sdk automatically converts it to a Secret object. The environment variable name is derived from the secret name by converting it to uppercase and replacing hyphens with underscores.

import os
import flyte

@flyte.task(secrets="my-api-key")
async def my_task():
    # If the secret name is "my-api-key", the env var is "MY_API_KEY"
    api_key = os.environ["MY_API_KEY"]
    ...

To use a specific environment variable name, use the as_env_var parameter:

from flyte import Secret, task

@task(secrets=Secret(key="openai-key", as_env_var="OPENAI_API_KEY"))
async def my_task():
    api_key = os.environ["OPENAI_API_KEY"]
    ...

Using TaskEnvironment​

For more complex configurations or when sharing settings across multiple tasks, use TaskEnvironment:

import flyte

env = flyte.TaskEnvironment(
    "production-env",
    secrets=[
        flyte.Secret(key="db-password", as_env_var="DB_PASSWORD"),
        flyte.Secret(group="aws", key="s3-token") # Becomes AWS_S3_TOKEN
    ]
)

@env.task
def query_db():
    password = os.environ["DB_PASSWORD"]
    ...

Mounting Secrets as Files​

Secrets can also be mounted as files. Currently, flyte-sdk only supports mounting secrets under the /etc/flyte/secrets directory.

When a secret is mounted, it is placed in a subdirectory named after the secret group (if provided) and a file named after the secret key (in lowercase).

import pathlib
import flyte

GROUP = "user-credentials"
KEY = "SSH_KEY"
SECRET_PATH = "/etc/flyte/secrets"

@flyte.task(
    secrets=flyte.Secret(group=GROUP, key=KEY, mount=pathlib.Path(SECRET_PATH))
)
def read_secret_file() -> str:
    # The file path follows the pattern: /etc/flyte/secrets/<group>/<key_lowercase>
    path = pathlib.Path(f"{SECRET_PATH}/{GROUP}/{KEY.lower()}")
    return path.read_text()

Using Secrets in Image Builds​

You can use secrets during the image build process to provide credentials for private package managers or repositories. This is common when installing private Python packages via pip.

from flyte import Image, Secret

# Use a secret to provide a GitHub Personal Access Token for a private repo
private_package = "git+https://$GITHUB_PAT@github.com/org/private-repo.git"

image = (
    Image.from_debian_base(install_flyte=True)
    .with_pip_packages(
        private_package, 
        secret_mounts=Secret("GITHUB_PAT")
    )
)

Managing Secrets Programmatically​

The flyte.remote.Secret class allows you to manage secrets on the Flyte backend directly from Python. This is useful for CI/CD pipelines or setup scripts.

Creating and Deleting Secrets​

You can create "regular" secrets for task injection or "image_pull" secrets for private container registries.

from flyte.remote import Secret

# Create a regular secret
Secret.create(name="my-secret", value="super-secret-value")

# Create an image pull secret (must be organization-level, no project/domain)
Secret.create(
    name="my-registry-auth", 
    value=b'{"auths": {...}}', 
    type="image_pull"
)

# Delete a secret
Secret.delete(name="my-secret")

Listing and Retrieving Secrets​

from flyte.remote import Secret

# Get a specific secret's metadata
s = Secret.get("my-api-key")
print(f"Secret {s.name} was created at {s.pb2.secret_metadata.created_time}")

# List all secrets in the current project/domain
for s in Secret.listall(limit=50):
    print(s.name, s.type)

Troubleshooting​

Environment Variable Naming​

Environment variable names must match the regex ^[A-Z_][A-Z0-9_]*$. If you provide a custom as_env_var that does not match this pattern, flyte-sdk will raise a ValueError.

Mount Path Restrictions​

The mount parameter in flyte.Secret currently only accepts pathlib.Path("/etc/flyte/secrets"). Attempting to use any other path will result in a ValueError: "Only /etc/flyte/secrets is supported as secret mount path today."

Image Pull Secret Scope​

When creating an image_pull secret using Secret.create, ensure that your configuration does not have a project or domain set. These secrets are scoped to the organization level to allow the Flyte platform to pull images across different projects. Attempting to create one with a project/domain set will raise a ValueError.