Vector Collection Connector
Vector collection stores vectors with their related metadata. This allows entries to be found efficiently based on vector distance.
For further information on vector collections, see Vector Collection.
Installing the Connector
This connector is included in the full and slim Enterprise Edition distributions of Hazelcast.
Permissions
If security is enabled, you can set up permissions to restrict clients' access to these data structures.
To search in vector collection, you must add the create and read permissions for those collections. If you use the vector collection sink to write to vector collections, you must add the create and put permissions for those collections.
For further information on adding these permissions, see Client Authorization.
Vector Collection as a Sink
To write an entry to a vector collection, to index it for searching, create a key and VectorDocument, which consists of
additional metadata and vectors (embeddings). Embeddings can be generated earlier in the pipeline or loaded or obtained from external source.
Pipeline p = Pipeline.create();
p.readFrom(Sources.<String, String>map("idToDocumentText"))
// generate embeddings
.mapUsingService(getAllMiniLmL6V2EmbeddingModelServiceFactory(),
(service, e) -> tuple3(e.getKey(), e.getValue(), VectorValues.of(service.embed(e.getValue()).content().vector())))
// write to vector collection
.writeTo(VectorSinks.vectorCollection("indexedDocuments", Tuple3::f0, Tuple3::f1, Tuple3::f2));Searching in Vector Collection
You can search vector collections in Jet pipelines using VectorTransforms.mapUsingVectorSearch transformation.
Pipeline p = Pipeline.create();
p.readFrom(TestSources.items("text to search for"))
// generate embedding for the object for which we are finding similarities
.mapUsingService(getAllMiniLmL6V2EmbeddingModelServiceFactory(),
(service, query) -> tuple2(query, VectorValues.of(service.embed(query).content().vector())))
// find similar objects
.apply(VectorTransforms.mapUsingVectorSearch("indexedDocuments",
SearchOptions.builder().limit(10).includeValue().build(),
// query vector
Tuple3::f1,
// process the search results
(input, result) -> tuple2(input, result)))
// use the results
.writeTo(Sinks.logger());