gigl.common.beam.sharded_read#

Attributes#

logger

Classes#

BigQueryShardedReadConfig

ShardedExportRead

A transform object used to modify one or more PCollections.

Module Contents#

class gigl.common.beam.sharded_read.BigQueryShardedReadConfig[source]#
num_shards: int = 20[source]#
project_id: str[source]#
shard_key: str[source]#
temp_dataset_name: str[source]#
class gigl.common.beam.sharded_read.ShardedExportRead(table_name, sharded_read_info, **kwargs)[source]#

Bases: apache_beam.PTransform

A transform object used to modify one or more PCollections.

Subclasses must define an expand() method that will be used when the transform is applied to some arguments. Typical usage pattern will be:

input | CustomTransform(…)

The expand() method of the CustomTransform object passed in will be called with input as an argument.

Parameters:
expand(pbegin)[source]#
Parameters:

pbegin (apache_beam.pvalue.PBegin)

gigl.common.beam.sharded_read.logger[source]#