[Enhancement] Reducing memory expansion when flushing memtable

[sollhui]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Background

We observed that a large amount of memory expansion occurred during the down flushing of memtable, causing imports to be easily cancelled due to resource constraints or large data volumes.

Solution

The blocks of memtable are organized by columns in memory, and the memory expansion of memtable during down flushing is greatly optimized by flushing memtable and releasing related memory(such as memtable block and pages of column writer) columns by columns：

• implement freeing memory column by column for partial update [opt](memory) implement freeing memory column by column for partial update #58275
• set source content column by column when flush memtable for partial update [opt](memory) set source content column by column when flush memtable for partial update #58782
• release memtable block column by column

Solution

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct