gigl.src.common.utils.spark_job_manager#

Attributes#

IDLE_TTL_DEFAULT_S

IDLE_TTL_DEV_DEFAULT_S

logger

Classes#

DataprocClusterInitData

SparkJobManager

Functions#

build_split_generator_cluster_name(job_name)

Return the Dataproc cluster name the split generator uses for job_name.

build_subgraph_sampler_cluster_name(job_name)

Return the Dataproc cluster name the subgraph sampler uses for job_name.

Module Contents#

class gigl.src.common.utils.spark_job_manager.DataprocClusterInitData[source]#
cluster_name: str[source]#
debug_cluster_owner_alias: str | None = None[source]#
init_script_uri: gigl.common.GcsUri | None = None[source]#
is_debug_mode: bool[source]#
labels: dict[str, str] | None = None[source]#
machine_type: str[source]#
num_local_ssds: int[source]#
num_workers: int[source]#
project: str[source]#
region: str[source]#
service_account: str[source]#
temp_assets_bucket: str[source]#
class gigl.src.common.utils.spark_job_manager.SparkJobManager(project, region, cluster_name)[source]#
Parameters:

  • project (str)

  • region (str)

  • cluster_name (str)

create_dataproc_cluster(cluster_init_data, use_spark35=False)[source]#
Parameters:
delete_cluster()[source]#
static get_sanitized_dataproc_cluster_name(cluster_name)[source]#

Make a name valid for Dataproc: _ to -, at most 50 characters, no trailing -.

Dataproc cluster names must match (?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?). This is the single source of the rule; the subgraph sampler, the split generator, and job cleanup all build cluster names through it.

Example

>>> SparkJobManager.get_sanitized_dataproc_cluster_name("sgs_my_job")
'sgs-my-job'
Parameters:

cluster_name (str) – Raw cluster name.

Returns:

The sanitized cluster name.

Return type:

str

submit_and_wait_scala_spark_job(main_jar_file_uri, max_job_duration, runtime_args=[], extra_jar_file_uris=[], use_spark35=False)[source]#
Parameters:

  • main_jar_file_uri (gigl.common.Uri)

  • max_job_duration (datetime.timedelta)

  • runtime_args (list[str])

  • extra_jar_file_uris (list[str])

  • use_spark35 (bool)

gigl.src.common.utils.spark_job_manager.build_split_generator_cluster_name(job_name)[source]#

Return the Dataproc cluster name the split generator uses for job_name.

Example

>>> build_split_generator_cluster_name("my_job")
'split-my-job'
Parameters:

job_name (str) – The GiGL applied task identifier.

Returns:

The sanitized cluster name.

Return type:

str

gigl.src.common.utils.spark_job_manager.build_subgraph_sampler_cluster_name(job_name)[source]#

Return the Dataproc cluster name the subgraph sampler uses for job_name.

Example

>>> build_subgraph_sampler_cluster_name("my_job")
'sgs-my-job'
Parameters:

job_name (str) – The GiGL applied task identifier.

Returns:

The sanitized cluster name.

Return type:

str

gigl.src.common.utils.spark_job_manager.IDLE_TTL_DEFAULT_S = 600[source]#
gigl.src.common.utils.spark_job_manager.IDLE_TTL_DEV_DEFAULT_S = 36000[source]#
gigl.src.common.utils.spark_job_manager.logger[source]#