Skip to main content

Real-time Log Observation

To observe logs from remote Flyte executions in real-time, flyte-sdk provides interactive log viewers and streaming utilities. These tools allow you to monitor progress, debug failures, and filter out system noise directly from your terminal or Jupyter notebook.

Tailing Logs from a Run or Action​

The most common way to observe logs is using the show_logs() method on a Run or Action object. By default, this launches an interactive viewer that maintains a rolling window of the most recent log lines.

from flyte.remote import FlyteRemote

remote = FlyteRemote(...)
run = remote.get_run(run_id="my-run-id")

# Launch the interactive log viewer
run.show_logs()

The show_logs method is decorated with @syncify, meaning you can call it synchronously in a script or asynchronously using .aio() in an async context:

# Asynchronous usage
await run.show_logs.aio(max_lines=50, show_ts=True)

Customizing the Log Viewer​

The show_logs method (implemented in src/flyte/remote/_run.py and src/flyte/remote/_action.py) accepts several parameters to control the output:

  • max_lines: The number of lines to keep in the rolling buffer (default is 30 for actions, 100 for runs).
  • show_ts: Set to True to prefix each line with an ISO-8601 timestamp.
  • filter_system: Set to True to hide Flyte internal logs and system-generated messages.
  • attempt: Specify a specific execution attempt (defaults to the latest attempt).
# View logs for the first attempt with timestamps and system filtering
run.show_logs(attempt=1, show_ts=True, filter_system=True)

Streaming Raw Logs​

In non-interactive environments like CI/CD pipelines or when you want to pipe logs to another process, use the raw=True flag. This disables the rich-based interactive viewer and prints log lines directly to the console as they arrive.

# Stream logs as plain text
run.show_logs(raw=True)

Internally, Logs.create_viewer in src/flyte/remote/_logs.py handles the logic for switching between the AsyncLogViewer and raw console output.

Programmatic Log Access​

If you need to process log lines programmatically, use the get_logs() method. This returns an iterator (or async iterator) of strings.

# Process logs line by line
for line in run.get_logs(filter_system=True):
    if "CRITICAL" in line:
        print(f"Found critical error: {line}")

For lower-level access to the raw protobuf objects (payload_pb2.LogLine), you can use the Logs.tail method directly:

from flyte.remote._logs import Logs

# action_id is an identifier_pb2.ActionIdentifier
async for log_line in Logs.tail.aio(action_id=action.action_id):
    print(f"Source: {log_line.originator}, Message: {log_line.message}")

Troubleshooting and Requirements​

Logs Not Yet Available​

When a run first starts, logs may not be immediately available in the backend. The show_logs implementation in src/flyte/remote/_action.py automatically calls self.wait(wait_for="logs-ready") to ensure the log stream is ready before attempting to connect. If you call Logs.tail directly and the stream is missing, it will retry for a limited duration before raising a LogsNotYetAvailableError.

Jupyter Notebook Support​

The interactive viewer uses rich.live.Live to update the display. In Jupyter or IPython environments, this requires ipywidgets to be installed. If ipywidgets is missing, flyte-sdk will log a warning and default to raw text output to ensure logs are still visible.

System Log Filtering​

When filter_system=True is used, the _format_line function in src/flyte/remote/_logs.py applies the following rules:

  1. Lines with originator == SYSTEM are dropped.
  2. Lines containing the string [flyte] are dropped, unless they also contain flyte.errors (ensuring critical framework errors remain visible).