settingsLogin | Registersettings
Scalefree
Show Menu

Load data from source incremental without changing source

0 votes
Hi,

When the source database or system can't be changed and there isn't a way to know if data has changed (date modified or other options to check this), in that case checking all the rows is always needed when copying data.

The only rights I will get on the source is read tables. So there is no option to configure cdc or add a trigger or change the source system etc. Just reading tables, that's it.
Rows in the source table are updated, created and deleted. I need an exact copy of this table in the staging.

I hoped that DataVault would solve this, but I couldn't find an answer.
Also tried several options, but nothing beats just copy the complete table every time to the staging area.



Thanks for any advice on this matter.

Tijs

ps. I am using Kimball methods right now and I am new to Data Vault. :-)
asked Jan 7, 2016 in Stage Area by trins (120 points)

1 Answer

0 votes
Hi,

check out the last seen date (page 100 of "Building a Scalable Data Warehouse with Data Vault 2.0"). It can be used to reduce the number of keys to check against the stage area. But in your specific scenario, you need to load all your data into stage first.

One way to reduce the load is to load a full load of your business keys only. This will be sufficient to detect deletes and is a thin table, small enough to handle. The other data comes as delta (new and changed records only).

Hope that helps,

Mike
answered Feb 25, 2016 by molschimke (1,890 points)
Scalefree Scalefree

Upcoming Trainings

  • July 01 to 03 - Amsterdam (English) Data Vault 2.0 Boot Camp and Certification

  • August 05 to 07 - Berlin (German) Data Vault 2.0 Boot Camp and Certification

  • September 9 to 13 - Hanover WWDVC EU

  • September 16 to 18 - Brussels (English) Data Vault 2.0 Boot Camp and Certification

  • October 21 to 23 - Hanover (German) Data Vault 2.0 Boot Camp and Certification

  • November 18 to 20 - Amsterdam (English) Data Vault 2.0 Boot Camp and Certification

  • December 02 to 04 - Dusseldorf (German) Data Vault 2.0 Boot Camp and Certification

  • December 16 to 18 - Zurich (English) Data Vault 2.0 Boot Camp and Certification

  • Contact us for inhouse training engagements.
    Visual Data Vault: Logical Modelling for Data Vault - Free Download!
    Scalefree
    DataVault.guru is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License.
    Permissions beyond the scope of this license are available in our license.

    DataVault.guru | DWH Wiki | Recent questions RSS feed | Imprint | Provided by Michael Olschimke
    ...