gigl.src.data_preprocessor.lib.transform.feature_quantization#

Attributes#

EDGE_PACKED_FEATURE_KEY

NODE_PACKED_FEATURE_KEY

logger

Functions#

apply_feature_quantization_transform(logical_features, ...)

Quantizes selected feature columns and bit-packs each record's values.

Module Contents#

gigl.src.data_preprocessor.lib.transform.feature_quantization.apply_feature_quantization_transform(logical_features, logical_metadata, logical_feature_keys, quantization_spec, quantization_metadata_path, packed_feature_key)[source]#

Quantizes selected feature columns and bit-packs each record’s values.

Stores packed bytes under packed_feature_key and computes global quantization statistics with Beam. Node preprocessing uses node_packed_features; main-edge preprocessing uses edge_packed_features.

Side Effects:

Writes the quantization statistics JSON that data_preprocessor.py reads and serializes into the preprocessing metadata protobuf.

Parameters:

  • logical_features (apache_beam.PCollection[pyarrow.RecordBatch]) – RecordBatches containing the logical feature columns.

  • logical_metadata (tensorflow_transform.tf_metadata.dataset_metadata.DatasetMetadata | apache_beam.PCollection[tensorflow_transform.tf_metadata.dataset_metadata.DatasetMetadata]) – Metadata for the logical schema. When reusing a pretrained TFTransform model, its schema is available immediately from the saved model as a DatasetMetadata object. When analyzing a new TFTransform model, Beam produces its output metadata during pipeline execution, after this pipeline has been built. It is therefore supplied as a PCollection for downstream transforms.

  • logical_feature_keys (list[str]) – Logical feature columns in original feature-vector order.

  • quantization_spec (gigl.src.data_preprocessor.lib.types.FeatureQuantizationSpec) – Feature keys and bit width to quantize.

  • quantization_metadata_path (str) – Destination for the quantization statistics JSON.

  • packed_feature_key (str) – Reserved physical field used for packed values.

Returns:

Quantized RecordBatches and eager or deferred physical I/O metadata. That metadata removes quantized feature columns and adds packed_feature_key. It affects serialized-record I/O only; the logical model schema remains unchanged.

Raises:

ValueError – If the reserved packed key already exists, a selected feature is absent or non-scalar, or feature values cannot be quantized.

Return type:

tuple[apache_beam.PCollection[pyarrow.RecordBatch], tensorflow_transform.tf_metadata.dataset_metadata.DatasetMetadata | apache_beam.pvalue.AsSingleton]

gigl.src.data_preprocessor.lib.transform.feature_quantization.EDGE_PACKED_FEATURE_KEY: Final[str] = 'edge_packed_features'[source]#
gigl.src.data_preprocessor.lib.transform.feature_quantization.NODE_PACKED_FEATURE_KEY: Final[str] = 'node_packed_features'[source]#
gigl.src.data_preprocessor.lib.transform.feature_quantization.logger[source]#