How to incrementally evaluate LLM?

Question

I have a question regarding optimizing LLM for usage in application like generative agents: https://github.com/joonspk-research/generative_agents

In such agentic environment, each agent repeatedly query the LLM API to evaluate context provided in prompt to generate new information or make decisions. These prompt calls have a lot of content in common, meaning that if an agent made 10 consecutive calls with 10 prompts, each prompt would look like the previous one with new information appended. This begs the questions -- is it possible to only send the new information and ask the LLM to evaluate the delta prompt incrementally?

On the LLM side, it only needs to cache the embeddings so that it can continue the same train of thoughts with new information later. This would result in dramatic speed up for AI agents. Any one has experience implementing this?

Answer 1

On the LLM side, it only needs to cache the embeddings so that it can continue the same train of thoughts with new information later

That's somehow similar to KV caching:

When generating text with a transformer model, we can significantly optimize the process by caching the Key (K) and Value (V) matrices. Let’s understand this visually:

In this visualization:

1. q_new represents the Query for the latest token.
2. K_prev and V_prev are cached from previous computations
3. k_new and v_new are computed only for the new token
4. Blue arrows show how attention is computed using both cached and new values