Incremental Load in QlikView – Part1

In QlikView, incremental load is used  when large transactional tables are involved. Incremental load is a mechanism by which updated/new data is loaded in QlikView. Incremental load require the use of QVD files for data storage.

Why use incremental load

In many business environments, there will be many large transaction tables. These tables may receive 1000’s or millions of rows of data daily or monthly. Since the data is huge and it may be sensitive also from the history standpoint, you may not want to delete the existing records, instead you will load only if the data is new. In such scenarios incremental load is used.

3 Options are available for Incremental Load

1. Insert Only

2. Insert and Update

3. Insert, Update and Delete

In this Part I ,i will show you Insert Only option

To show the concept of Incremental load, i will use and load table Incre_Table1. Before starting with incremental load, you have to do a initial load, which is just storing all the data present in the table to QVD.

InitialTable_1

Insert Only

Insert only is the process which just takes new data from the source and adds it to the existing QVD

Steps :

1. Load all the data from a db table and store it into QVD. This is referred as initial load

Incremental1

After the initial load, i have added a row in my table so that we can load this updated row. With new row the table will look like the following, notice the LastUpdate date in the last row.

InitialTable_AddedOnerow

2. Load all New data from the db. There can be many ways to identify New data. One way is to have a LastUpdate date in table and while loading the table check for the last update date.

Incre_NewUpdatedRow3. Concatenate this data with the Load of all the data from the QVD file and store the entire table into the QVD. This will overwrite the existing QVD.

Incre_Concatenate2

This completes Incremental Load with Insert Only option. In the next blog i will show you how to perform incremental load with Insert and Update And with Insert,Update and delete option.

5 thoughts on “Incremental Load in QlikView – Part1

  1. Varun Prakash says:

    What if there is no Date field then?

    • Chandraish Sinha says:

      Varun:
      You will always need a way to identify new records added to the table. In any warehouse environment date is the best method to keep track
      of data load. Load date or last update date should always be part of your table.

      • Thanks you very much Chandraish Sinha,
        You deserve it,

        This is the best blog which I have seen for the past 6 months,
        I’m very new to qlikview, your explanation is so clear.
        Thanks a lot.

        I will share this site to all my friends, who are learning qlikview.
        We request you to please keep on posting on this site..

  2. After concatenate, when STORE TO QVD statement is performed, does it overwrite/replace the data in QVD or it just appends the new data to the existing data in QVD?

    • Taj Mohamed says:

      Hi Ashok,

      Its better to have a new QVDs after incremental values have been added to qlikview.

      So you will be having old and new QVDs which are created from same table but naming different.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.