I am currently getting started with InfiniDB and geared towards making it my go-to database solution. I am using it in combination with in-house written software that produces and queries the data that ends up in the database tables.
The concrete use case it that, within my C program, data is collected in a buffer for a time frame of roughly 5 minutes and then flushed out from memory directly into the database. I have come to realize that this is not as straightforward in InfiniDB because for efficient bulk loading, cpimport is used. However, I would like to avoid multiple disk reads and writes as in the following scenario:
In-Memory buffer (my program) --> Temp file on disk --> memory (cpimport) --> Database on disk
Instead, it would be beneficial to do something programmatic along the lines of the following posts:
In-Memory buffer (my program) --> API (operating on memory) --> Database on disk
Are there any solutions other than cpimport? The following post talks about piping data to STDOUT, however, this still involves multiple on-disk operations and the use of cpimport:
If I need to bulk insert roughly 10 000 entries every 5 minutes, would it be possible to use standard MySQL "INSERT" statements instead, while using the underlying InfiniDB engine?