SQL upsert loader

November 28, 2020

Overview of SQL upsert loader

• jupyter notebook link

• source code

I hope to present a Proof Of Concept version of a multiprocessed data loader that I use extensively in my data integration pipelines.

Couple of problems that it is attempting to address are

1. Ability to operate on any update flat file associated with a SQL table, with minor configuration.
2. Ability to perform upsert operations on datasets without a primary key column. Although a combination key has to be identified using multiple columns to dedup the records.

Use the link above to view the example in a jupyter notebook.

• jupyter
• SQL
• ETL
• integration