Infinidb error during ETL process

3 posts / 0 new
Last post
jessieapollo
jessieapollo's picture
Offline
Last seen: 1 year 1 month ago
Joined: Aug 1 2013
Junior Boarder

Posts: 2

Jessie Evangelista
Infinidb error during ETL process

We are using Calpont InfiniDB 2.2.11 Community on Ubuntu Server 12.04LTS
Hardware: 2 x Quad Core Xeon, 48GB mem, LSI RAID card 512MB w/ BBU in writeback mode, 8TB RAID10
we processed a 315MB sales data file containing several days of data
we first split the file by date and process each one in turn:

119M sales_transaction_20130601.csv
134M sales_transaction_20130602.csv
30M sales_transaction_20130603.csv
33M sales_transaction_20130531.csv
29K sales_transaction_20130530.csv
13K sales_transaction_20130527.csv
1.6K sales_transaction_20130526.csv
12K sales_transaction_20130525.csv
104 sales_transaction_20130516.csv
20K sales_transaction_20130523.csv
46 sales_transaction_20130524.csv
471K sales_transaction_20130528.csv
1.2M sales_transaction_20130529.csv
4.8K sales_transaction_20130522.csv
2.3K sales_transaction_20130515.csv
33K sales_transaction_20130510.csv

we check if there is existing data for the particular date, if none we simple load the data using cpimport
if data already exists, we load the data into an adjustment table and detect duplicates between the target fact table and the adjustment table
we delete the duplicate data in the fact table and load the adjustments into the fact table
we then generate aggregates on the loaded fact table which encountered the error described below.

we then restarted infinidb and tried running the same query, no more error.

should we restart infinidb everytime we try to do ETL?

#idbmysql -BN suppliers_portal <

bdempsey
bdempsey's picture
Offline
Last seen: 1 month 3 weeks ago
Joined: Oct 27 2009
Platinum Boarder

Posts: 194

Robert Dempsey
Re: Infinidb error during ETL process

If it happens again, try [code]select calflushcache();[/code] and see if the problem goes away.

jessieapollo
jessieapollo's picture
Offline
Last seen: 1 year 1 month ago
Joined: Aug 1 2013
Junior Boarder

Posts: 2

Jessie Evangelista
Re: Infinidb error during ETL process

I removed infinidb restart commands and replace with select calflushcache();

so far so good

can you please explain why this works?