fileflow.operators package

fileflow.operators.dive_operator module

class fileflow.operators.dive_operator.DiveOperator(*args, **kwargs)

Bases: airflow.models.BaseOperator

An operator that sets up storage and assigns data dependencies to the operator class.

This is intended as a base class for eg fileflow.operators.DivePythonOperator.

storage

Lazy load a storage property on access instead of on class instantiation. Something in the storage attribute is not deep-copyable which causes errors with airflow clear and airflow backfill which both try to deep copy a target DAG and all its operators, so we only want this property when we actually use it.

fileflow.operators.dive_python_operator module

class fileflow.operators.dive_python_operator.DivePythonOperator(python_object, python_method='run', *args, **kwargs)

Bases: fileflow.operators.dive_operator.DiveOperator, python_operator.PythonOperator

Python operator that can send along data dependencies to its callable. Generates the callable by initializing its python object and calling its method.

pre_execute(context)