Skip to main content

Secure Secret Management

When you need to use sensitive information like API keys or database credentials in your Flyte tasks, flyte-sdk provides a secure way to request and manage these values without hardcoding them. You can request secrets in your task definitions and manage them on the cluster using the programmatic API or the CLI.

Requesting Secrets in Tasks​

To use a secret in a task, you specify it in the @task decorator. flyte-sdk automatically makes these secrets available to your task at runtime, typically as environment variables.

Using Environment Variables​

By default, secrets are injected as environment variables. The environment variable name is derived from the secret key by converting it to uppercase and replacing hyphens with underscores.

import os
from flyte import task, Secret

# Simple usage: secret "my-api-key" becomes "MY_API_KEY"
@task(secrets="my-api-key")
def my_task():
    api_key = os.environ.get("MY_API_KEY")
    ...

# Explicitly naming the environment variable
@task(secrets=Secret(key="openai-key", as_env_var="OPENAI_API_KEY"))
def my_task_with_custom_env():
    api_key = os.environ.get("OPENAI_API_KEY")
    ...

Mounting Secrets as Files​

You can also mount secrets as files. Currently, flyte-sdk only supports mounting secrets under the /etc/flyte/secrets directory.

import pathlib
from flyte import task, Secret

SECRET_PATH = "/etc/flyte/secrets"

@task(secrets=Secret(key="db-config", mount=pathlib.Path(SECRET_PATH)))
def my_file_task():
    # The secret will be available at /etc/flyte/secrets/db-config
    config_path = pathlib.Path(SECRET_PATH) / "db-config"
    config_data = config_path.read_text()
    ...

Managing Secrets Programmatically​

The flyte.remote.Secret class allows you to manage secrets on a Flyte cluster programmatically. This is useful for automation or setting up environments.

Creating and Deleting Secrets​

You can create "regular" secrets (scoped to a project and domain) or "image_pull" secrets (scoped to the organization).

from flyte.remote import Secret

# Create a regular secret for the current project/domain
Secret.create(name="my-secret", value="super-secret-value")

# Create a secret from binary data
Secret.create(name="binary-secret", value=b"\x01\x02\x03")

# Delete a secret
Secret.delete(name="my-secret")

Retrieving and Listing Secrets​

You can retrieve metadata about secrets, such as their creation time and status across clusters.

from flyte.remote import Secret

# Get a specific secret
secret = Secret.get(name="my-secret")
print(f"Secret {secret.name} was created at {secret.pb2.secret_metadata.created_time}")

# List all secrets in the current project/domain
for secret in Secret.listall(limit=20):
    print(f"Found secret: {secret.name} (Type: {secret.type})")

Secret Types and Scoping​

flyte-sdk distinguishes between two types of secrets:

  1. Regular Secrets: These are the most common and are scoped to a specific project and domain. They are used by tasks running within that project and domain.
  2. Image Pull Secrets: These are used by the Flyte platform to authenticate with private container registries. They are organization-level and must not have a project or domain set.
# Creating an image pull secret (ensure project/domain are NOT set in config)
Secret.create(name="registry-creds", value=docker_config_json, type="image_pull")

CLI Management​

You can also manage secrets using the flyte CLI, which uses the same underlying flyte.remote.Secret implementation.

# Create a secret (will prompt for value)
flyte create secret my_secret

# Create a secret with an explicit value
flyte create secret my_secret --value "my_value"

# Create a secret from a file
flyte create secret my_secret --from-file ./secret_file.bin

# Create an image pull secret for a registry
flyte create secret regcred --type image_pull --registry ghcr.io --username myuser

Troubleshooting​

Environment Variable Naming​

If you request a secret with key="my-secret-key" and don't specify as_env_var, flyte-sdk will look for MY_SECRET_KEY. If your secret key contains characters other than alphanumeric and hyphens, it is recommended to provide an explicit as_env_var name that matches the regex ^[A-Z_][A-Z0-9_]*$.

Mount Path Restrictions​

Attempting to mount a secret to any path other than /etc/flyte/secrets will raise a ValueError:

# This will raise: ValueError("Only /etc/flyte/secrets is supported as secret mount path today.")
Secret(key="my-key", mount=pathlib.Path("/tmp/secrets"))

Image Pull Secret Scoping​

If you attempt to create an image_pull secret while a project or domain is configured in your environment, Secret.create will raise a ValueError. Ensure these are cleared before managing organization-level secrets.