Record Storers

Base Class

All record storers must inherit from/implement the following base class (with some repeated from FlowStep)

class flow.record_storer.RecordStorer[RecordType]

An abstract interface for retrieving and storing records.

Parent Class: FlowStep

abstract class property description: str

A high-level description of the step

abstract class property config_types: list[tuple[str, Type[ValidConfigTypes], str]]

The types of the configurations for the step. Each tuple in the list should include the name, type, and description of a configuration (respectively). See FlowStep

abstractmethod validate() → None

This function should inspect the object’s configs attribute, and perform any static validation possible.

abstractmethod get_records(logger: Callable[[str], None], debug: bool = False) → list[RecordType]

Get the records from wherever they’re stored.

Parameters:

• logger (Callable[[str], None]) – A logger for recording notable events

• debug (bool) – Whether to run in debug mode. Defaults to False

Returns:

The retrieved records

Return type:

list[RecordType]

abstractmethod set_records(rec_list: list[RecordType], logger: Callable[[str], None], debug: bool = False) → None

Store the updated records wherever they’re stored.

Parameters:

• rec_list (list[RecordType]) – The records to store

• logger (Callable[[str], None]) – A logger for recording notable events

• debug (bool) – Whether to run in debug mode. Defaults to False

Implementations

class utils.basic_steps.BasicRecordStorer

A basic record storer that stores integers in a file.

Supported record types: int

configs.file_path: str

The path to a file to source and store integers from

validate() → None

Validate the configurations for the step.

Specifically, we make sure that the supplied path doesn’t represent a pre-existing non-file object, and that we have permissions to read from it

class google_steps.spreadsheet_storer.SpreadsheetStorer[SpreadsheetRecord]

An abstraction of getting and storing records in a Google Sheet.

If multiple flows are accessing the spreadsheet at the same time, there is the possibility of data being miscommunicated if one flow reads at the same time that another clears the spreadsheet while updating. To address this, a global lock is implemented using POSIX file locks and fcntl.lockf to ensure proper synchronization between flows (running on the same system, which is assumed).

Supported record types: SpreadsheetRecord

configs.sheet_id: str

The ID of the Google Sheet to access

Service Account access

Make sure that your service account has access to the Google Sheet with this ID!

configs.tag: str

The Google Sheet tab to access for records

validate() → None

Validate the configurations for the spreadsheet storer.

Specifically, we make sure that the service account can access the specified Google Sheet