Issue with importing from CVS

4 posts / 0 new
Last post
deano
deano's picture
Offline
Last seen: 1 year 2 months ago
Joined: Sep 22 2011
Junior Boarder

Posts: 15

Dean Kamali
Issue with importing from CVS

It works fine when I do select start from table and pipe the output to cpimport tool, however it doesn't work when I'm trying to load it from cvs, message suggest to increase read buffer, the only think I found was to increase the number of read threads, however it doesn't work, can someone help me with that?

[code][root@archive bin]# ./cpimport -r 100 -j40
number of read threads : 100

Bulkload root directory : /usr/local/Calpont/data/bulk
Job description file : /usr/local/Calpont/data/bulk/job/Job_40.xml
Log file for this job: /usr/local/Calpont/data/bulk/log/Job_40.log
2012-01-25 10:54:59 (21595) INFO : successfully loaded job file /usr/local/Calpont/data/bulk/job/Job_40.xml
2012-01-25 10:54:59 (21595) INFO : Job file loaded, run time for this step : 0.0782552 seconds
2012-01-25 10:54:59 (21595) INFO : PreProcessing check starts
2012-01-25 10:55:00 (21595) INFO : PreProcessing check completed
2012-01-25 10:55:00 (21595) INFO : preProcess completed, run time for this step : 0.813608 seconds
2012-01-25 10:55:00 (21595) INFO : No of Read Threads Spawned = 100
2012-01-25 10:55:00 (21595) INFO : No of Parse Threads Spawned = 3
2012-01-25 10:55:00 (21595) ERR : Error reading import file /usr/local/Calpont/data/bulk/data/import/reportingdetail.tbl; near line 1; Single row fills read buffer; try larger read buffer. [1456]
2012-01-25 10:55:00 (21595) CRIT : Bulkload Read (thread 0) Failed for Table cms_archive.reportingdetail. Terminating this job. [1456]
2012-01-25 10:55:00 (21595) INFO : Bulkload Parse (thread 2) Stopped parsing Tables. BulkLoad::parse() responding to job termination
2012-01-25 10:55:00 (21595) INFO : Bulkload Parse (thread 0) Stopped parsing Tables. BulkLoad::parse() responding to job termination
2012-01-25 10:55:00 (21595) INFO : Bulkload Parse (thread 1) Stopped parsing Tables. BulkLoad::parse() responding to job termination
2012-01-25 10:55:00 (21595) INFO : Table cms_archive.reportingdetail (OID-4292) was not successfully loaded. Rolling back.
2012-01-25 10:55:01 (21595) INFO : Bulk load completed, total run time : 1.87133 seconds

Error in loading job data[/code]

Thank you

radams
radams's picture
Offline
Last seen: 2 hours 2 min ago
Joined: Jan 3 2011
Administrator

Posts: 492

Robert Adams
Re: Issue with importing from CVS

Hi,

You do not need to change the number of read threads. Please check your data file. The error is indicating that it has read in a million bytes of data with out finding the end of the first row. How big are the rows you are trying to import?

Thank you,

Robert
Calpont

deano
deano's picture
Offline
Last seen: 1 year 2 months ago
Joined: Sep 22 2011
Junior Boarder

Posts: 15

Dean Kamali
Re: Issue with importing from CVS

there should be around 1 million rows, I guess I need to terminate the rows properly when exporting data to cvs file.

I'm only terminating fields with | I guess the lines should be terminated with t

I will try that

deano
deano's picture
Offline
Last seen: 1 year 2 months ago
Joined: Sep 22 2011
Junior Boarder

Posts: 15

Dean Kamali
Re: Issue with importing from CVS

thanks for your help!

problem solved after exporting fields to be terminated by | and lines by ' backslash n'

I tried it from mysql console and via sqlyog and it worked!

Thanks