performance - Improve memory consumption of a map of tuples without losing speed in C++ -

To store costs for 5-Tulips, I have a map like this:

  studs :: map & lt; Std :: tr1 :: tuple & lt; U32, u32, u32, u32, u8 & gt ;, f32 & gt; CostMap_;  

There are several such table loaded in memory during the runtime, where each has approximately 2.5 * 10⁷ entries in each entry

  2.16345 # 5875 396 # 47512 26445 # 2  

and std :: inputstream . After this, the entries remain unchanged.

Many Tuples are only different in 1 value, meaning there are several values ​​that appear in some entries. I thought about using this fact and thought about unnecessary collection of values.

I have tried a map map map like

  std :: map & lt; U32, std :: maps & lt; U32, std :: map & lt; U32, std :: map & lt; U32, std :: map & lt; U8, f32 & gt; & Gt; & Gt; & Gt; & Gt; CostMap_;  

But this is very slow, because there are millions of look-ups in only a few minutes and therefore the look up should be fast enough (this is not fully optimized on the runtime yet. ).

Can the tables be loaded in memory, such as at the same time without losing too much speed during the runtime, memory consumption drops considerably below (at least one aspect of at least 2)?

You can reduce the overlap by dividing the tubal on the length of the most common prefix instead of splitting It's at each level.

For example, suppose your data has a lot of overlap on the first three members of Tupl, i.e. if you consider the first three members of each tuple in the map, the number of specific entries are some orders of magnitude In that situation, you can divide the map into maps map. I also recommend using unordered_map , because it is asymptotically fast.

  std :: unordered_map & lt; Std :: tr1 :: tuple & lt; U32, u32, u32 & gt; Std:: unordered_map & lt; Std :: tr1 :: Tuple & LT; U32, u8 & gt ;, F32 & gt; & Gt; CostMap_;  

This approach lowers the duplication of significant storage at the cost of additional lookups. The result is in saving, when you have more number of duplicates in the prefix of Tuples, it is enough to justify additional look.

Keep in mind that this approach is the generalization of the map expansion approach that you have in your post, suggested in the "fuse" tuples with the first three and last two positions. You can choose another point for the partition - for example, if the first four keys are repeated often, you can

  std :: unordered_map & lt; Std :: tr1 :: tuple & lt; U32, u32, u32, u32 & gt ;, std :: unordered_map & lt; U8, F32 & gt; & Gt; CostMap_;  

with all but the last search keys in "fuse" in Tupal.