AsyncConnector
This is the base class for all async connectors, and it defines the interface that all connectors must implement. The connector service is responsible for invoking connectors. The executor will communicate with the connector service to create tasks, get the status of tasks, and delete tasks.
Attributes
| Attribute | Type | Description |
|---|---|---|
| name | str = "Async Connector" | Display name identifying the async connector. |
| task_type_name | str | Task type name used by the connector service to look up the connector in the ConnectorRegistry. |
| task_type_version | int = 0 | Version number of the task type used for connector resolution in the ConnectorRegistry. |
| metadata_type | [ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | Type of resource metadata associated with the connector for tracking and managing task operations. |
Methods
create()
@abstractmethod
async def create(
self,
task_template: tasks_pb2.TaskTemplate,
output_prefix: str,
inputs: Optional[Dict[str, typing.Any]] = None,
task_execution_metadata: Optional[TaskExecutionMetadata] = None,
**kwargs,
) -> ResourceMeta: ...
Return a resource meta that can be used to get the status of the task.
Parameters
| Name | Type | Description |
|---|---|---|
| task_template | tasks_pb2.TaskTemplate | The task template definition containing the specification and configuration for the task. |
| output_prefix | str | The storage prefix path where task execution outputs should be written. |
| inputs | Optional[Dict[str, typing.Any]] = None | Dictionary of input key-value pairs required for the task execution. |
| task_execution_metadata | Optional[TaskExecutionMetadata] = None | Contextual metadata and execution settings for the task run. |
| kwargs | Any | Additional connector-specific keyword arguments. |
Returns
| Type | Description |
|---|---|
[ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | A resource meta that can be used to get the status of the task. |
get()
@abstractmethod
async def get(self, resource_meta: ResourceMeta, **kwargs) -> Resource: ...
Return the status of the task, and return the outputs in some cases. For example, bigquery job can't write the structured dataset to the output location, so it returns the output literals to the propeller, and the propeller will write the structured dataset to the blob store.
Parameters
| Name | Type | Description |
|---|---|---|
| resource_meta | [ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | Metadata identifying the specific task resource to query. |
| kwargs | Any | Additional connector-specific keyword arguments. |
Returns
| Type | Description |
|---|---|
[Resource](resource.md?sid=flyte_connectors__connector_resource) | The current status of the task resource along with any output literals if available. |
delete()
@abstractmethod
async def delete(self, resource_meta: ResourceMeta, **kwargs): ...
Delete the task. This call should be idempotent. It should raise an error if fails to delete the task.
Parameters
| Name | Type | Description |
|---|---|---|
| resource_meta | [ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | Metadata identifying the specific task resource to delete. |
| kwargs | Any | Additional connector-specific keyword arguments. |
get_metrics()
async def get_metrics(self, resource_meta: ResourceMeta, **kwargs) -> GetTaskMetricsResponse: ...
Return the metrics for the task.
Parameters
| Name | Type | Description |
|---|---|---|
| resource_meta | [ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | Metadata identifying the specific task resource whose metrics are requested. |
| kwargs | Any | Additional connector-specific keyword arguments. |
Returns
| Type | Description |
|---|---|
GetTaskMetricsResponse | The response containing execution metrics for the specified task. |
get_logs()
async def get_logs(self, resource_meta: ResourceMeta, **kwargs) -> GetTaskLogsResponse: ...
Return the metrics for the task.
Parameters
| Name | Type | Description |
|---|---|---|
| resource_meta | [ResourceMeta](resourcemeta.md?sid=flyte_connectors__connector_resourcemeta) | Metadata identifying the specific task resource whose logs are requested. |
| kwargs | Any | Additional connector-specific keyword arguments. |
Returns
| Type | Description |
|---|---|
GetTaskLogsResponse | The response containing log output information and links for the task execution. |