Joining large time series table with itself

2 posts / 0 new
Last post
phong
phong's picture
Offline
Last seen: 1 month 2 weeks ago
Joined: Aug 8 2014
Junior Boarder

Posts: 1

Philip Hong
Joining large time series table with itself

Hi,

I'm extremely new to infinidb so please bear with me. I'm working with a time series data table in a single server setting with hundreds of millions of rows, and I need to join the table with itself for some analysis. I will be updating the table too with the calculated results. I've read on the tuning guide that I can tweak the PmMaxMemorySmallSide field for joining smaller tables, but what's the best way for joining large tables like in my case? Also, I keep running out of memory on the server - is NumBlocksPct the correct field to fix that?

 

I'm running infinidb community on a Linux server with about 100G of memory.

Here are my settings for hash join:

    <HashJoin>
        <MaxBuckets>128</MaxBuckets>
        <MaxElems>128K</MaxElems>
        <FifoSizeLargeSide>128</FifoSizeLargeSide>
        <PmMaxMemorySmallSide>4000M</PmMaxMemorySmallSide>
        <TotalUmMemory>32G</TotalUmMemory>
        <CPUniqueLimit>100</CPUniqueLimit>
    </HashJoin>
 

Thanks very much in advance

mikikg
mikikg's picture
Offline
Last seen: 1 month 1 week ago
Joined: Jul 24 2014
Junior Boarder

Posts: 3

Aleksandar Markovic
RE: Joining large time series table with itself