gigl.common.services.dataflow#

Cancel Dataflow jobs by name through the google-cloud-dataflow-client library.

Attributes#

logger

Classes#

DataflowService

Lists and cancels Dataflow jobs in one project and region.

Module Contents#

class gigl.common.services.dataflow.DataflowService(project, region)[source]#

Lists and cancels Dataflow jobs in one project and region.

Example

>>> service = DataflowService(project="p", region="us-central1")
>>> service.cancel_jobs_with_name_prefix("gigl-my-job-")
['2026-10-28_09_00_00-123']
Parameters:

  • project (str) – GCP project that owns the jobs.

  • region (str) – Dataflow regional endpoint the jobs run in.

cancel_jobs_with_name_prefix(name_prefix)[source]#

Request a cancel for every active job whose name starts with name_prefix.

The match is a plain prefix. Build the prefix with gigl.src.common.utils.dataflow.get_sanitized_dataflow_job_name() and end it with - so the full GiGL job name must match.

A job too new to accept a cancel is retried for about two minutes.

A failed cancel does not stop the loop. Every matching job gets a cancel request before the errors are raised.

Parameters:

name_prefix (str) – Sanitized Dataflow job name prefix.

Returns:

Ids of the jobs a cancel was requested for.

Raises:

ExceptionGroup – If one or more cancels failed, for example a job that still refuses the cancel after the last retry. It holds one error for each failed job.

Return type:

list[str]

gigl.common.services.dataflow.logger[source]#