Skip to main content

Scaling, Domains, and Timeouts

Configure scaling, domains, and request timeouts​

Configure these behaviors on the public flyte.app.AppEnvironment declaration. For example, this app keeps one replica available, permits up to five replicas, scales on request rate, waits five minutes before scaling down, and uses a custom subdomain:

from flyte.app._app_environment import AppEnvironment
from flyte.app._types import Domain, Scaling

app_env = AppEnvironment(
    name="panel-textual-app-test-1",
    image="auto",
    port=8080,
    scaling=Scaling(),
    domain=Domain(),
)

AppEnvironment supplies Scaling(), Domain(), and Timeouts() defaults. Import those types from flyte.app in application code; flyte.app._types and flyte.app._app_environment are private implementation modules. The private imports above identify the defining modules used by the SDK source.

Choose replica bounds​

Scaling.replicas is either a fixed integer or a (minimum, maximum) tuple. The constructor normalizes an integer to a pair, and get_replicas() returns the pair used by app serialization.

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Scaling' does not accept keyword(s): replicas

from flyte.app._types import Scaling

scale_to_zero = Scaling()                 # (0, 1), the default
always_on = Scaling(replicas=(1, 1))
burstable = Scaling(replicas=(1, 5))
high_availability = Scaling(replicas=(2, 10))
fixed = Scaling(replicas=3)               # normalized to (3, 3)

assert scale_to_zero.get_replicas() == (0, 1)
assert fixed.replicas == (3, 3)

The default (0, 1) allows no idle replica and a maximum of one replica when demand arrives. A fixed Scaling(replicas=3) is always the (3, 3) policy. Be careful with Scaling(replicas=0): it is normalized to (0, 0), whereas a tuple (0, 1) is the documented scale-to-zero policy. Tuple input cannot use (0, 0) because tuple maxima must be at least 1.

The repository uses the (0, 1) pattern for model-serving environments, with a five-minute idle delay and authentication disabled:

from flyte.app._types import Scaling

scaling = Scaling()

A model-serving app can pass this object as its inherited scaling configuration and set requires_auth=False, as the vLLM example does.

Select an autoscaling metric​

Set metric to one of the nested Scaling metric classes:

  • Scaling.Concurrency(val) uses concurrent requests per replica as the threshold.
  • Scaling.RequestRate(val) uses requests per second per replica as the threshold.

Both thresholds must be at least 1:

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Scaling' does not accept keyword(s): metric, replicas 'flyte.app._types.Scaling.Concurrency' does not accept keyword(s): val

from flyte.app._types import Scaling

concurrency_scaling = Scaling(
    replicas=(1, 10),
    metric=Scaling.Concurrency(val=5),
)

request_rate_scaling = Scaling(
    replicas=(1, 10),
    metric=Scaling.RequestRate(val=100),
)

Concurrency is a frozen dataclass, so its val cannot be reassigned after construction. RequestRate is a regular dataclass. At serialization time, flyte-sdk converts the first metric to the AppIDL concurrency.target_value field and the second to request_rate.target_value. Omitting metric leaves the scaling metric unset.

scaledown_after accepts integer seconds or a datetime.timedelta. Positive integer values are normalized to a timedelta before serialization:

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Scaling' does not accept keyword(s): scaledown_after

from datetime import timedelta

from flyte.app._types import Scaling

five_minutes_from_seconds = Scaling(scaledown_after=300)
five_minutes_from_delta = Scaling(scaledown_after=timedelta(minutes=5))

assert five_minutes_from_seconds.scaledown_after == timedelta(seconds=300)
assert five_minutes_from_delta.scaledown_after == timedelta(minutes=5)

The app serializer includes a scaledown period only when app_env.scaling.scaledown_after is truthy. Consequently, use a positive duration for an explicit delay; an integer 0 is not normalized by Scaling.__post_init__ and is omitted by the serializer.

Set an app subdomain or custom domain​

Pass a Domain to AppEnvironment:

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Domain' does not accept keyword(s): subdomain

import flyte

from flyte.app._app_environment import AppEnvironment
from flyte.app._types import Domain

app_env = AppEnvironment(
    name="streamlit-hello-custom-domain",
    image="auto",
    command="streamlit hello --server.port 8080",
    domain=Domain(subdomain="custom-subdomain"),
)

if __name__ == "__main__":
    flyte.init_from_config()
    d = flyte.deploy(app_env)
    print(d[0])

This follows the declaration and deployment pattern in examples/apps/custom_domain.py. Domain(subdomain=...) supplies the ingress subdomain. To supply a custom-domain/CNAME value, use custom_domain; both values can be supplied together:

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Domain' does not accept keyword(s): custom_domain, subdomain

from flyte.app._app_environment import AppEnvironment
from flyte.app._types import Domain

app_env = AppEnvironment(
    name="domain-app",
    image="auto",
    domain=Domain(
        subdomain="my-custom-subdomain",
        custom_domain="example.com",
    ),
)

Domain only stores subdomain and custom_domain; it does not validate hostname syntax, subdomain format, or ownership. During translate_app_env_to_idl(), flyte-sdk places these values in IngressConfig.subdomain and IngressConfig.cname. An empty Domain() and domain=None produce empty ingress values, and the serialized ingress has private=False.

Resolve the running endpoint​

AppEnvironment.endpoint is context-sensitive:

  • In local mode it returns http://localhost:<port>.
  • Outside local mode, if INTERNAL_APP_ENDPOINT_PATTERN is set, it formats that value with app_fqdn=self.name.
  • Otherwise it initializes the Flyte client and obtains the endpoint through flyte.remote.App.get(name=self.name).

For internal app-to-app traffic, set the pattern with an {app_fqdn} placeholder, for example:

INTERNAL_APP_ENDPOINT_PATTERN=http://{app_fqdn}.internal:8080

The same placeholder convention is used by private AppEndpoint resolution. A pattern without the placeholder does not provide the upstream-name substitution expected by that path.

Configure request timeouts​

Use Timeouts(request=...) on the app environment. Integers mean seconds, and datetime.timedelta values are accepted directly:

Unverified example

This example could not be verified against this version of the codebase and may not work as shown. Validator finding: 'flyte.app._types.Timeouts' does not accept keyword(s): request

from flyte.app._app_environment import AppEnvironment
from flyte.app._types import Timeouts

app_env = AppEnvironment(
    name="timeout-app",
    image="auto",
    timeouts=Timeouts(request=60),
)

assert app_env.timeouts.request.total_seconds() == 60

Timeouts.request is normalized to a timedelta. None leaves the timeout configuration out of the app specification; 0 is valid and is serialized as a zero-second duration. The accepted range is zero through one hour (3600 seconds), inclusive. Negative values, values over one hour, floats, and strings raise validation errors.

The runtime serializer converts a configured request timeout to a protobuf Duration and places it in Spec.timeouts.request_timeout. For example, Timeouts(request=30) produces seconds == 30 and nanos == 0 in the serialized duration.

Troubleshoot validation failures​

  • Replica range: fixed counts and minimums must be nonnegative. A tuple must contain exactly two values; its maximum must be at least 1 and no smaller than its minimum. Thus (5, 2) and (0, 0) are rejected.
  • Metric threshold: Scaling.Concurrency(val=0) and Scaling.RequestRate(val=0) are rejected. metric must be one of those two metric types.
  • Scale-down units: scaledown_after=300 means 300 seconds, not 300 minutes. Use timedelta when expressing the duration in named units.
  • Timeout zero behavior: Timeouts(request=0) is retained and serialized, while Scaling(scaledown_after=0) is falsey and omitted from the autoscaling configuration.
  • App name: names must be lowercase alphanumeric strings with hyphens, beginning and ending with an alphanumeric character. Names such as My-App, my_app, -app, and app- are rejected.
  • Reserved ports: integer ports are converted to Port objects, but 8012, 8022, 8112, 9090, and 9091 cannot be used.
  • Parameters and commands: an app with parameters cannot use a custom command unless that command starts with fserve. Use args instead, or define the process with @app_env.server.

How deployment receives the settings​

flyte.app._runtime.app_serde.translate_app_env_to_idl() is the serialization entry point. It calls Scaling.get_replicas(), converts the selected metric and scale-down duration into AutoscalingConfig, maps Domain into IngressConfig, and maps Timeouts.request into TimeoutConfig. flyte.deploy(app_env) therefore receives one declarative AppEnvironment whose scaling, ingress, and request-timeout fields are carried into the deployed app specification.