Custom Steering of nodes extraction

[ParisNeo]

In general, the graph contains alot of noise (many completely out of scope or useless nodes are generated along useful nodes).
How about we add in the interface a user input that details the kind of nodes we want to extract or to suppress.
this can be done by making a part of the extraction prompt modifiable. We can use the webui to steer the extraction with two texts that can either be set together or just one of them:

• What kind of information we want to extract
• What kind of information to ignore

This can be cool and allow enhancing the quality of te extraction and qurying.

@LarFii @danielaskdd @ArnoChenFx .
What do you think?

[LarFii]

I think it might be possible to add an optional customizable prompt section, allowing users to control which content to focus on extracting.

[danielaskdd]

I think after indexing the documents, we should perform an optimization scan of nodes and relationships to mitigate issues. During this phase, we can do the following:

• Remove unnecessary nodes
• Community detection and community summarization
• Identify and merge nodes representing the same entities
• Node and edge summary compression (to prevent LLM context overflow during queries)

[ntsarb]

Community Detection and Summarisation are available in Microsoft's GraphRAG and my understanding is that LightRAG intentionally avoided that to reduce the LLM usage and API calls.

Also, I haven't seen how de-duplication works in LightRAG but it's important to keep in mind that two entities may have the same name but different meaning. Their descriptions (and vector embeddings) should reflect that. Vice-versa, two entities may have slightly different names and correspond to the same entity. Hopefully the relationships can be used to determine that.