sources.sql_database_pipeline

load_select_tables_from_database

def load_select_tables_from_database() -> None

[view_source]

Use the sql_database source to reflect an entire database schema and load select tables from it.

This example sources data from the public Rfam MySQL database.

load_entire_database

def load_entire_database() -> None

[view_source]

Use the sql_database source to completely load all tables in a database

load_standalone_table_resource

def load_standalone_table_resource() -> None

[view_source]

Load a few known tables with the standalone sql_table resource, request full schema and deferred table reflection

select_columns

def select_columns() -> None

[view_source]

Uses table adapter callback to modify list of columns to be selected

select_with_end_value_and_row_order

def select_with_end_value_and_row_order() -> None

[view_source]

Gets data from a table withing a specified range and sorts rows descending

my_sql_via_pyarrow

def my_sql_via_pyarrow() -> None

[view_source]

Uses pyarrow backend to load tables from mysql

create_unsw_flow

def create_unsw_flow() -> None

[view_source]

Uploads UNSW_Flow dataset to postgres via csv stream skipping dlt normalizer. You need to download the dataset from https://github.com/rdpahalavan/nids-datasets

test_connectorx_speed

def test_connectorx_speed() -> None

[view_source]

Uses unsw_flow dataset (~2mln rows, 25+ columns) to test connectorx speed

use_type_adapter

def use_type_adapter() -> None

[view_source]

Example use of type adapter to coerce unknown data types

specify_columns_to_load

def specify_columns_to_load() -> None

[view_source]

Run the SQL database source with a subset of table columns loaded