In QlikView,incremental load is used when large trasactional tables are involved. Incremental load is a mechanism by which a data is appended or updated in large transactional tables.Incremental load require the use of QVD files for data storage.
Why use incremental load
In many business environments, there will be many large transaction tables. These tables may receive 1000′s or millions of rows of data daily or monthly. Since the data is huge and it may be sensitive also from the history standpoint, you may not want to delete the existing records, instead you will load only if the data is new. In such scenarios incremental load is used.
3 Options are available for Incremental Load
1. Insert Only
2. Insert and Update
3. Insert, Update and Delete
In this Part I ,i will show you Insert Only option
To show the concept of Incremental load, i will use and load table Incre_Table1. Before starting with incremental load, you have to do a initial load, which is just storing all the data present in the table to QVD.
Insert only is the process which just takes new data from the source and adds it to the existing QVD
1. Load all the data from a db table and store it into QVD.This is referred as initial load
After the initial load, i have added a row in my table so that we can load this updated row. With new row the table will look like the following, notice the LastUpdate date in the last row.
2. Load all New data from the db. There can be many ways to identify New data. One way is to have a LastUpdate date in table and while loading the table check for the last update date.
This completes Incremental Load with Insert Only option. In the next blog i will show you how to perform incremental load with Insert and Update And with Insert,Update and delete option.