pyspark.pipelines.append_flow#

pyspark.pipelines.append_flow(*, target, name=None, spark_conf=None)[source]#

Return a decorator on a query function to define a flow in a pipeline.

Parameters

name – The name of the flow. If unspecified, the query function’s name will be used.
target – The name of the dataset this flow writes to. Must be specified.
spark_conf – A dict whose keys are the conf names and values are the conf values. These confs will be set when the flow is executed; they can override confs set for the destination, for the pipeline, or on the cluster.