Record Managers
Record Managers keep track of your indexed documents, preventing duplicated vector embeddings in Vector Store.
When document chunks are upserting, each chunk will be hashed using SHA-1 algorithm. These hashes will get stored in Record Manager. If there is an existing hash, the embedding and upserting process will be skipped.
In some cases, you might want to delete existing documents that are derived from the same sources as the new documents being indexed. For that, there are 3 cleanup modes for Record Manager:
- Incremental
- Full
- None
When you are upserting multiple documents, and you want to prevent deletion of the existing documents that are not part of the current upserting process, use Incremental Cleanup mode.
- Let's have a Record Manager with
IncrementalCleanup andsourceas SourceId Key
 (1) (1) (1) (1) (1) (1) (1) (1).png)
 (1) (1) (1) (1) (1) (1) (1) (1).png)
- And have the following 2 documents:
| Text | Metadata |
|---|---|
| Cat | {source:"cat"} |
| Dog | {source:"dog"} |
 (1) (1).png)
 (1) (1) (1) (1).png)
 (1) (1) (1) (1) (2).png)
 (1) (1) (1) (1) (1) (1) (2).png)
- After an upsert, we will see 2 documents that are upserted:
 (1) (1) (1) (1) (2).png)
- Now, if we delete the Dog document, and update Cat to Cats, we will now see the following:
 (2).png)
- The original Cat document is deleted
- A new document with Cats is added
- Dog document is left untouched
- The remaining vector embeddings in Vector Store are Cats and Dog
.png)
When you are upserting multiple documents, Full Cleanup mode will automatically delete any vector embeddings that are not part of the current upserting process.
- Let's have a Record Manager with
FullCleanup. We don't need to have a SourceId Key for Full Cleanup mode.
.png)
- And have the following 2 documents:
| Text | Metadata |
|---|---|
| Cat | {source:"cat"} |
| Dog | {source:"dog"} |
- After an upsert, we will see 2 documents that are upserted:
- Now, if we delete the Dog document, and update Cat to Cats, we will now see the following:
.png)
- The original Cat document is deleted
- A new document with Cats is added
- Dog document is deleted
- The remaining vector embeddings in Vector Store is just Cats
.png)
No cleanup will be performed
Current available Record Manager nodes are:
- SQLite
- MySQL
- PostgresQL