haive.core.engine.vectorstore

Quick Links

Overview

This module provides comprehensive abstractions and implementations for working with vector stores in the Haive framework. Vector stores are specialized databases optimized for storing and retrieving high-dimensional vectors, typically used for similarity search in RAG (Retrieval-Augmented Generation) applications.

Note

Vector stores enable efficient semantic search by storing document embeddings and providing fast similarity-based retrieval. They are essential components for building RAG systems, recommendation engines, and other applications that require similarity search over large document collections.

Supported Providers

Open Source

  • Chroma - Local and server modes

  • FAISS - Facebook AI Similarity Search

  • Weaviate - Vector search engine

  • Qdrant - Similarity search engine

  • Milvus - Distributed vector database

Cloud Services

  • Pinecone - Managed vector database

  • Supabase - PostgreSQL + pgvector

  • MongoDB Atlas - Vector search

  • OpenSearch - Elasticsearch-based

  • Redis - Vector search capabilities

Specialized

  • LanceDB - Serverless vector DB

  • Marqo - Tensor search engine

  • Zilliz - Cloud-native Milvus

Quick Start

from haive.core.engine.vectorstore import VectorStoreConfig, VectorStoreProvider

# Configure a local vector store
config = VectorStoreConfig(
    provider=VectorStoreProvider.Chroma,
    collection_name="documents",
    persist_directory="./chroma_db"
)

# Create and use the vector store
vectorstore = config.instantiate()
vectorstore.add_texts(["Document content"], metadatas=[{"source": "doc1"}])
results = vectorstore.similarity_search("query text", k=5)

# Configure for production with Pinecone
config = VectorStoreConfig(
    provider=VectorStoreProvider.Pinecone,
    api_key_env_var="PINECONE_API_KEY",
    environment="us-west1-gcp",
    index_name="production-index"
)

# Create with custom embeddings
from haive.core.engine.embeddings import OpenAIEmbeddingConfig

embedding_config = OpenAIEmbeddingConfig(model="text-embedding-3-small")
config.embedding_model = embedding_config

from haive.core.engine.vectorstore import VectorStoreConfig, VectorStoreProvider
from langchain_core.documents import Document

# Create documents
docs = [
    Document(page_content="Content 1", metadata={"source": "file1.txt"}),
    Document(page_content="Content 2", metadata={"source": "file2.txt"})
]

# Create vector store from documents
vs_config = VectorStoreConfig.create_vs_config_from_documents(
    documents=docs,
    vector_store_provider=VectorStoreProvider.Chroma
)

API Reference

Configuration Classes

VectorStoreConfig(*[, id, name, ...])

Configuration model for a vector store engine.

VectorStoreProvider(*values)

Enumeration of supported vector store providers.
class haive.core.engine.vectorstore.VectorStoreConfig(*, id=<factory>, name=<factory>, engine_type=EngineType.VECTOR_STORE, description=None, input_schema=None, output_schema=None, version='1.0.0', metadata=<factory>, embedding_model=<factory>, vector_store_provider=VectorStoreProvider.FAISS, documents=<factory>, vector_store_path='vector_store', docstore_path='docstore', k=4, score_threshold=None, search_type='similarity', vector_store_kwargs=<factory>)[source]

Bases: InvokableEngine[Union[str, dict[str, Any]], list[Document]]

Configuration model for a vector store engine.

VectorStoreConfig provides a consistent interface for creating and using vector stores with embeddings. It encapsulates all the configuration needed to create and interact with various vector store backends, abstracting away provider-specific implementation details.

This class enables: 1. Creating vector stores with various providers (FAISS, Chroma, Pinecone, etc.) 2. Managing documents and embeddings for vector storage 3. Performing similarity searches with configurable parameters 4. Creating retrievers that can be used in retrieval chains

Parameters:
engine_type

The type of engine (always VECTOR_STORE).

Type:

EngineType

embedding_model

Configuration for the embedding model.

Type:

BaseEmbeddingConfig

vector_store_provider

The vector store provider to use.

Type:

VectorStoreProvider

documents

Documents to store in the vector store.

Type:

List[Document]

vector_store_path

Path for storing vector indices on disk.

Type:

str

docstore_path

Path for storing document data.

Type:

str

k

Default number of documents to retrieve in searches.

Type:

int

score_threshold

Minimum similarity score for results.

Type:

Optional[float]

search_type

Search algorithm to use (e.g., “similarity”, “mmr”).

Type:

str

vector_store_kwargs

Additional provider-specific parameters.

Type:

Dict[str, Any]

Examples

>>> from haive.core.engine.vectorstore import VectorStoreConfig, VectorStoreProvider
>>> from haive.core.models.embeddings.base import HuggingFaceEmbeddingConfig
>>> from langchain_core.documents import Document
>>>
>>> # Create configuration
>>> config = VectorStoreConfig(
...     name="product_search",
...     documents=[Document(page_content="iPhone 13: The latest smartphone from Apple")],
...     vector_store_provider=VectorStoreProvider.FAISS,
...     embedding_model=HuggingFaceEmbeddingConfig(
...         model="sentence-transformers/all-MiniLM-L6-v2"
...     ),
...     k=5
... )
>>>
>>> # Create vector store
>>> vectorstore = config.create_vectorstore()
>>>
>>> # Perform similarity search
>>> results = config.similarity_search("smartphone", k=3)
>>>
>>> # Create a retriever
>>> retriever = config.create_retriever(search_type="mmr")

Configuration Examples

Local Vector Store:

config = VectorStoreConfig(
    provider=VectorStoreProvider.Chroma,
    persist_directory="./local_db",
    collection_name="my_documents"
)

Cloud Vector Store with Authentication:

config = VectorStoreConfig(
    provider=VectorStoreProvider.Pinecone,
    api_key_env_var="PINECONE_API_KEY",
    environment="us-west1-gcp",
    index_name="production",
    vector_store_kwargs={
        "metric": "cosine",
        "dimension": 1536
    }
)
classmethod __get_pydantic_json_schema__(core_schema, handler, /)

Hook into generating the model’s JSON schema.

Parameters:

  • core_schema (CoreSchema) – A pydantic-core CoreSchema. You can ignore this argument and call the handler with a new CoreSchema, wrap this CoreSchema ({‘type’: ‘nullable’, ‘schema’: current_schema}), or just call the handler with the original schema.

  • handler (GetJsonSchemaHandler) – Call into Pydantic’s internal JSON schema generation. This will raise a pydantic.errors.PydanticInvalidForJsonSchema if JSON schema generation fails. Since this gets called by BaseModel.model_json_schema you can override the schema_generator argument to that function to change JSON schema generation globally for a type.

Returns:

A JSON schema, as a Python object.

Return type:

JsonSchemaValue

classmethod __pydantic_init_subclass__(**kwargs)

This is intended to behave just like __init_subclass__, but is called by ModelMetaclass only after basic class initialization is complete. In particular, attributes like model_fields will be present when this is called, but forward annotations are not guaranteed to be resolved yet, meaning that creating an instance of the class may fail.

This is necessary because __init_subclass__ will always be called by type.__new__, and it would require a prohibitively large refactor to the ModelMetaclass to ensure that type.__new__ was called in such a manner that the class would already be sufficiently initialized.

This will receive the same kwargs that would be passed to the standard __init_subclass__, namely, any kwargs passed to the class definition that aren’t used internally by Pydantic.

Parameters:

**kwargs (Any) – Any keyword arguments passed to the class definition that aren’t used internally by Pydantic.

Return type:

None

Note

You may want to override [__pydantic_on_complete__()][pydantic.main.BaseModel.__pydantic_on_complete__] instead, which is called once the class and its fields are fully initialized and ready for validation.

classmethod __pydantic_on_complete__()

This is called once the class and its fields are fully initialized and ready to be used.

This typically happens when the class is created (just before [__pydantic_init_subclass__()][pydantic.main.BaseModel.__pydantic_init_subclass__] is called on the superclass), except when forward annotations are used that could not immediately be resolved. In that case, it will be called later, when the model is rebuilt automatically or explicitly using [model_rebuild()][pydantic.main.BaseModel.model_rebuild].

Return type:

None

classmethod construct(_fields_set=None, **values)
Parameters:
Return type:

Self

classmethod create_vs_config_from_documents(documents, embedding_model=None, **kwargs)[source]

Create a VectorStoreConfig from a list of documents.

Parameters:

  • documents (list[Document]) – List of documents to include

  • embedding_model (BaseEmbeddingConfig | None) – Optional embedding model configuration

  • **kwargs – Additional parameters for the config

Returns:

Configured VectorStoreConfig

Return type:

VectorStoreConfig

classmethod create_vs_from_documents(documents, embedding_model=None, **kwargs)[source]

Create a VectorStore from a list of documents.

Parameters:

  • documents (list[Document]) – List of documents to include

  • embedding_model (BaseEmbeddingConfig | None) – Optional embedding model configuration

  • **kwargs – Additional parameters for the config

Returns:

Instantiated VectorStore

Return type:

VectorStore

classmethod from_dict(data)

Create an engine from a dictionary representation.

Reconstructs an engine instance from its dictionary representation, attempting to use the original engine class if available, or falling back to the base class if the specific class cannot be loaded.

Parameters:

data (Dict[str, Any]) – Dictionary representation of the engine, typically created by the to_dict method.

Returns:

The reconstructed engine instance.

Return type:

Engine

Raises:

  • Various exceptions may be raised during dynamic class loading, but these –

  • are caught and logged, with a fallback to the base class. –

Examples

>>> # Create an engine
>>> original = MyLLMEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> # Convert to dictionary
>>> data = original.to_dict()
>>> # Reconstruct from dictionary
>>> reconstructed = Engine.from_dict(data)
>>> # Verify it's the same type
>>> type(reconstructed) == type(original)
True
classmethod from_json(json_str)

Create an engine from a JSON string representation.

Deserializes an engine from its JSON string representation, reconstructing the original engine instance or a compatible instance if the exact class is not available.

Parameters:

json_str (str) – JSON string representation of the engine, typically created by the to_json method.

Returns:

The reconstructed engine instance.

Return type:

Engine

Examples

>>> # Create and serialize an engine
>>> original = MyLLMEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> json_str = original.to_json()
>>> # Reconstruct from JSON
>>> reconstructed = Engine.from_json(json_str)
>>> reconstructed.name
'gpt-4'
>>> reconstructed.engine_type
<EngineType.LLM: 'llm'>
classmethod from_orm(obj)
Parameters:

obj (Any)

Return type:

Self

classmethod model_construct(_fields_set=None, **values)

Creates a new instance of the Model class with validated data.

Creates a new model setting __dict__ and __pydantic_fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed.

!!! note

model_construct() generally respects the model_config.extra setting on the provided model. That is, if model_config.extra == ‘allow’, then all extra passed values are added to the model instance’s __dict__ and __pydantic_extra__ fields. If model_config.extra == ‘ignore’ (the default), then all extra passed values are ignored. Because no validation is performed with a call to model_construct(), having model_config.extra == ‘forbid’ does not result in an error if extra values are passed, but they will be ignored.

Parameters:

  • _fields_set (set[str] | None) – A set of field names that were originally explicitly set during instantiation. If provided, this is directly used for the [model_fields_set][pydantic.BaseModel.model_fields_set] attribute. Otherwise, the field names from the values argument will be used.

  • values (Any) – Trusted or pre-validated data dictionary.

Returns:

A new instance of the Model class with validated data.

Return type:

Self

classmethod model_json_schema(by_alias=True, ref_template='#/$defs/{model}', schema_generator=<class 'pydantic.json_schema.GenerateJsonSchema'>, mode='validation', *, union_format='any_of')

Generates a JSON schema for a model class.

Parameters:

  • by_alias (bool) – Whether to use attribute aliases or not.

  • ref_template (str) – The reference template.

  • union_format (Literal['any_of', 'primitive_type_array']) –

    The format to use when combining schemas from unions together. Can be one of:

    keyword to combine schemas (the default). - ‘primitive_type_array’: Use the [type](https://json-schema.org/understanding-json-schema/reference/type) keyword as an array of strings, containing each type of the combination. If any of the schemas is not a primitive type (string, boolean, null, integer or number) or contains constraints/metadata, falls back to any_of.

  • schema_generator (type[GenerateJsonSchema]) – To override the logic used to generate the JSON schema, as a subclass of GenerateJsonSchema with your desired modifications

  • mode (Literal['validation', 'serialization']) – The mode in which to generate the schema.

Returns:

The JSON schema for the given model class.

Return type:

dict[str, Any]

classmethod model_parametrized_name(params)

Compute the class name for parametrizations of generic classes.

This method can be overridden to achieve a custom naming scheme for generic BaseModels.

Parameters:

params (tuple[type[Any], ...]) – Tuple of types of the class. Given a generic class Model with 2 type variables and a concrete model Model[str, int], the value (str, int) would be passed to params.

Returns:

String representing the new class where params are passed to cls as type variables.

Raises:

TypeError – Raised when trying to generate concrete names for non-generic models.

Return type:

str

classmethod model_rebuild(*, force=False, raise_errors=True, _parent_namespace_depth=2, _types_namespace=None)

Try to rebuild the pydantic-core schema for the model.

This may be necessary when one of the annotations is a ForwardRef which could not be resolved during the initial attempt to build the schema, and automatic rebuilding fails.

Parameters:

  • force (bool) – Whether to force the rebuilding of the model schema, defaults to False.

  • raise_errors (bool) – Whether to raise errors, defaults to True.

  • _parent_namespace_depth (int) – The depth level of the parent namespace, defaults to 2.

  • _types_namespace (MappingNamespace | None) – The types namespace, defaults to None.

Returns:

Returns None if the schema is already “complete” and rebuilding was not required. If rebuilding _was_ required, returns True if rebuilding was successful, otherwise False.

Return type:

bool | None

classmethod model_validate(obj, *, strict=None, extra=None, from_attributes=None, context=None, by_alias=None, by_name=None)

Validate a pydantic model instance.

Parameters:

  • obj (Any) – The object to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • from_attributes (bool | None) – Whether to extract data from object attributes.

  • context (Any | None) – Additional context to pass to the validator.

  • by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Raises:

ValidationError – If the object could not be validated.

Returns:

The validated model instance.

Return type:

Self

classmethod model_validate_json(json_data, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)
!!! abstract “Usage Documentation”

[JSON Parsing](../concepts/json.md#json-parsing)

Validate the given JSON data against the Pydantic model.

Parameters:

  • json_data (str | bytes | bytearray) – The JSON data to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (Any | None) – Extra variables to pass to the validator.

  • by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Raises:

ValidationError – If json_data is not a JSON string or the object could not be validated.

Return type:

Self

classmethod model_validate_strings(obj, *, strict=None, extra=None, context=None, by_alias=None, by_name=None)

Validate the given object with string data against the Pydantic model.

Parameters:

  • obj (Any) – The object containing string data to validate.

  • strict (bool | None) – Whether to enforce types strictly.

  • extra (Literal['allow', 'ignore', 'forbid'] | None) – Whether to ignore, allow, or forbid extra data during model validation. See the [extra configuration value][pydantic.ConfigDict.extra] for details.

  • context (Any | None) – Extra variables to pass to the validator.

  • by_alias (bool | None) – Whether to use the field’s alias when validating against the provided input data.

  • by_name (bool | None) – Whether to use the field’s name when validating against the provided input data.

Returns:

The validated Pydantic model.

Return type:

Self

classmethod parse_file(path, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)
Parameters:

  • path (str | Path)

  • content_type (str | None)

  • encoding (str)

  • proto (DeprecatedParseProtocol | None)

  • allow_pickle (bool)

Return type:

Self

classmethod parse_obj(obj)
Parameters:

obj (Any)

Return type:

Self

classmethod parse_raw(b, *, content_type=None, encoding='utf8', proto=None, allow_pickle=False)
Parameters:

  • b (str | bytes)

  • content_type (str | None)

  • encoding (str)

  • proto (DeprecatedParseProtocol | None)

  • allow_pickle (bool)

Return type:

Self

classmethod schema(by_alias=True, ref_template='#/$defs/{model}')
Parameters:

  • by_alias (bool)

  • ref_template (str)

Return type:

Dict[str, Any]

classmethod schema_json(*, by_alias=True, ref_template='#/$defs/{model}', **dumps_kwargs)
Parameters:

  • by_alias (bool)

  • ref_template (str)

  • dumps_kwargs (Any)

Return type:

str

classmethod update_forward_refs(**localns)
Parameters:

localns (Any)

Return type:

None

classmethod validate(value)
Parameters:

value (Any)

Return type:

Self

classmethod validate_engine_type(v)[source]

Validate Engine Type.

Parameters:

v – [TODO: Add description]

Returns:

Add return description]

Return type:

[TODO

__copy__()

Returns a shallow copy of the model.

Return type:

Self

__deepcopy__(memo=None)

Returns a deep copy of the model.

Parameters:

memo (dict[int, Any] | None)

Return type:

Self

__init__(**data)

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Parameters:

data (Any)

Return type:

None

__iter__()

So dict(model) works.

Return type:

Generator[tuple[str, Any], None, None]

__pretty__(fmt, **kwargs)

Used by devtools (https://python-devtools.helpmanual.io/) to pretty print objects.

Parameters:
Return type:

Generator[Any]

__repr_name__()

Name of the instance’s class, used in __repr__.

Return type:

str

__repr_recursion__(object)

Returns the string representation of a recursive object.

Parameters:

object (Any)

Return type:

str

__rich_repr__()

Used by Rich (https://rich.readthedocs.io/en/stable/pretty.html) to pretty print objects.

Return type:

RichReprResult

add_document(document)[source]

Add a single document to the vector store config.

Parameters:

document (Document) – Document to add

Return type:

None

add_documents(documents)[source]

Add multiple documents to the vector store config.

Parameters:

documents (list[Document]) – List of documents to add

Return type:

None

async ainvoke(input_data, runnable_config=None)

Convenience method to create and asynchronously invoke the runnable.

Creates a runnable instance using create_runnable() and immediately invokes it asynchronously with the provided input data. If the runnable doesn’t support asynchronous invocation natively, this method will run it in a separate thread.

Parameters:

  • input_data (TIn) – Input data for the runnable to process.

  • runnable_config (Optional[RunnableConfig]) – Optional runtime configuration to apply when creating and invoking the runnable.

Returns:

Output from the runnable after processing the input data.

Return type:

TOut

Raises:

  • NotImplementedError – If the created runnable cannot be invoked asynchronously.

  • Various exceptions may be propagated from the runnable's ainvoke method. –

Examples

>>> engine = MyLLMEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> # Usage in an async context
>>> async def generate():
...     response = await engine.ainvoke("What is the capital of France?")
...     print(response)
>>> # 'The capital of France is Paris.'
apply_runnable_config(runnable_config=None)

Extract parameters from runnable_config relevant to this engine.

Processes the provided runtime configuration to extract parameters that apply to this specific engine. The method implements a hierarchical lookup strategy, prioritizing more specific configurations (by ID, then name, then type) over general ones.

Parameters:

runnable_config (Optional[RunnableConfig]) – Runtime configuration dictionary containing engine-specific parameters.

Returns:

Dictionary of configuration parameters that apply to this engine.

Return type:

Dict[str, Any]

Examples

>>> engine = MyEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> config = {
...     "configurable": {
...         "engine_configs": {
...             "gpt-4": {"temperature": 0.7},
...             "llm_config": {"max_tokens": 1000}
...         },
...         "temperature": 0.5
...     }
... }
>>> params = engine.apply_runnable_config(config)
>>> # Name-specific config takes priority over engine type config
>>> params["temperature"]
0.7
>>> # Type-specific config is included
>>> params["max_tokens"]
1000
copy(*, include=None, exclude=None, update=None, deep=False)

Returns a copy of the model.

!!! warning “Deprecated”

This method is now deprecated; use model_copy instead.

If you need include or exclude, use:

`python {test="skip" lint="skip"} data = self.model_dump(include=include, exclude=exclude, round_trip=True) data = {**data, **(update or {})} copied = self.model_validate(data) `

Parameters:

  • include (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to include in the copied model.

  • exclude (AbstractSetIntStr | MappingIntStrAny | None) – Optional set or mapping specifying which fields to exclude in the copied model.

  • update (Dict[str, Any] | None) – Optional dictionary of field-value pairs to override field values in the copied model.

  • deep (bool) – If True, the values of fields that are Pydantic models will be deep-copied.

Returns:

A copy of the model with included, excluded and updated fields as specified.

Return type:

Self

create_retriever(search_type=None, search_kwargs=None, **kwargs)[source]

Create a retriever from the vector store.

Parameters:

  • search_type (str | None) – Search type (similarity, mmr, etc.)

  • search_kwargs (dict[str, Any] | None) – Search parameters

  • **kwargs – Additional parameters for the retriever

Returns:

Configured retriever

Return type:

BaseRetriever

create_runnable(runnable_config=None)[source]

Create a vector store instance with configuration applied.

Parameters:

runnable_config (RunnableConfig | None) – Optional runtime configuration

Returns:

Instantiated vector store

Return type:

VectorStore

create_vectorstore(async_mode=False)[source]

Create a vector store instance from this configuration.

Instantiates a vector store of the configured provider type, using the documents and embedding model specified in the configuration. This method handles the details of creating the appropriate vector store class, initializing it with the correct parameters, and populating it with documents.

The method supports both synchronous and asynchronous initialization paths, and includes special handling for empty document collections.

Parameters:

async_mode (bool) – Whether to use async methods for vector store creation. Default is False. If True, the method will use asynchronous variants of the vector store creation methods if available.

Returns:

An instantiated vector store of the configured provider type,

populated with the configured documents and using the specified embedding model.

Return type:

VectorStore

Raises:

ValueError – If an empty vector store cannot be created with the specified provider.

Examples

>>> config = VectorStoreConfig(
...     name="product_catalog",
...     vector_store_provider=VectorStoreProvider.FAISS,
...     documents=[Document(page_content="Product description...")]
... )
>>> vectorstore = config.create_vectorstore()
>>>
>>> # With async mode
>>> async def create_async():
...     return await config.create_vectorstore(async_mode=True)
derive_input_schema()

Derive input schema for this engine as a Pydantic model.

Generates a Pydantic model representing the input schema for this engine, based on the fields returned by get_input_fields() or using the explicitly provided input_schema if available.

Returns:

A Pydantic model class representing the input schema.

Return type:

Type[BaseModel]

Examples

>>> engine = MyEngine(name="test_engine", engine_type=EngineType.LLM)
>>> InputSchema = engine.derive_input_schema()
>>> input_data = InputSchema(prompt="Hello, world!", temperature=0.7)
>>> print(input_data.model_dump())
{'prompt': 'Hello, world!', 'temperature': 0.7}
derive_output_schema()

Derive output schema for this engine as a Pydantic model.

Generates a Pydantic model representing the output schema for this engine, based on the fields returned by get_output_fields() or using the explicitly provided output_schema if available.

Returns:

A Pydantic model class representing the output schema.

Return type:

Type[BaseModel]

Examples

>>> engine = MyEngine(name="test_engine", engine_type=EngineType.LLM)
>>> OutputSchema = engine.derive_output_schema()
>>> output_data = OutputSchema(completion="Generated text", tokens_used=15)
>>> print(output_data.model_dump())
{'completion': 'Generated text', 'tokens_used': 15}
dict(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False)
Parameters:
Return type:

Dict[str, Any]

extract_params()[source]

Extract parameters from this engine for serialization.

Returns:

Dictionary of engine parameters

Return type:

dict[str, Any]

get_input_fields()[source]

Return input field definitions as field_name -> (type, default) pairs.

Returns:

Dictionary mapping field names to (type, default) tuples

Return type:

dict[str, tuple[type, Any]]

get_output_fields()[source]

Return output field definitions as field_name -> (type, default) pairs.

Returns:

Dictionary mapping field names to (type, default) tuples

Return type:

dict[str, tuple[type, Any]]

get_schema_fields()

Get combined schema fields for this engine.

Combines the input and output fields from get_input_fields() and get_output_fields() into a single dictionary. This is useful for generating a comprehensive schema for the engine.

Returns:

Dictionary mapping field names to

(type, default) tuples.

Return type:

Dict[str, Tuple[Type, Any]]

Examples

>>> engine = MyEngine(name="test_engine", engine_type=EngineType.LLM)
>>> fields = engine.get_schema_fields()
>>> print(fields.keys())
dict_keys(['prompt', 'temperature', 'completion', 'tokens_used'])
get_vectorstore(embedding=None, async_mode=False)[source]

Get the vector store with optional embedding override.

Parameters:

  • embedding – Optional embedding model override

  • async_mode (bool) – Whether to use async methods

Returns:

Instantiated vector store

Return type:

VectorStore

instantiate(runnable_config=None)

Instantiate the engine (alias for create_runnable).

This is a convenience method that provides a more intuitive name for the create_runnable method, particularly in contexts where “instantiate” is more semantically appropriate than “create_runnable”.

Parameters:

runnable_config (Optional[RunnableConfig]) – Optional runtime configuration to apply when creating the runnable.

Returns:

The instantiated runtime object.

Return type:

Any

Examples

>>> engine = MyEngine(name="test_engine", engine_type=EngineType.LLM)
>>> llm = engine.instantiate({"temperature": 0.5})
>>> response = llm.generate("Tell me a joke")
invoke(input_data, runnable_config=None)[source]

Invoke the vector store with input data.

Parameters:

  • input_data (str | dict[str, Any]) – Query string or dictionary with search parameters

  • runnable_config (RunnableConfig | None) – Optional runtime configuration

Returns:

List of retrieved documents

Return type:

list[Document]

json(*, include=None, exclude=None, by_alias=False, exclude_unset=False, exclude_defaults=False, exclude_none=False, encoder=PydanticUndefined, models_as_dict=PydanticUndefined, **dumps_kwargs)
Parameters:
Return type:

str

model_copy(*, update=None, deep=False)
!!! abstract “Usage Documentation”

[model_copy](../concepts/models.md#model-copy)

Returns a copy of the model.

!!! note

The underlying instance’s [__dict__][object.__dict__] attribute is copied. This might have unexpected side effects if you store anything in it, on top of the model fields (e.g. the value of [cached properties][functools.cached_property]).

Parameters:

  • update (Mapping[str, Any] | None) – Values to change/add in the new model. Note: the data is not validated before creating the new model. You should trust this data.

  • deep (bool) – Set to True to make a deep copy of the model.

Returns:

New model instance.

Return type:

Self

model_dump(*, mode='python', include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)
!!! abstract “Usage Documentation”

[model_dump](../concepts/serialization.md#python-mode)

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

Parameters:

  • mode (Literal['json', 'python'] | str) – The mode in which to_python should run. If mode is ‘json’, the output will only contain JSON serializable types. If mode is ‘python’, the output may contain non-JSON-serializable Python objects.

  • include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to include in the output.

  • exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – A set of fields to exclude from the output.

  • context (Any | None) – Additional context to pass to the serializer.

  • by_alias (bool | None) – Whether to use the field’s alias in the dictionary key if defined.

  • exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (bool) – Whether to exclude fields that are set to their default value.

  • exclude_none (bool) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A dictionary representation of the model.

Return type:

dict[str, Any]

model_dump_json(*, indent=None, ensure_ascii=False, include=None, exclude=None, context=None, by_alias=None, exclude_unset=False, exclude_defaults=False, exclude_none=False, exclude_computed_fields=False, round_trip=False, warnings=True, fallback=None, serialize_as_any=False)
!!! abstract “Usage Documentation”

[model_dump_json](../concepts/serialization.md#json-mode)

Generates a JSON representation of the model using Pydantic’s to_json method.

Parameters:

  • indent (int | None) – Indentation to use in the JSON output. If None is passed, the output will be compact.

  • ensure_ascii (bool) – If True, the output is guaranteed to have all incoming non-ASCII characters escaped. If False (the default), these characters will be output as-is.

  • include (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to include in the JSON output.

  • exclude (set[int] | set[str] | Mapping[int, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | Mapping[str, set[int] | set[str] | Mapping[int, IncEx | bool] | Mapping[str, IncEx | bool] | bool] | None) – Field(s) to exclude from the JSON output.

  • context (Any | None) – Additional context to pass to the serializer.

  • by_alias (bool | None) – Whether to serialize using field aliases.

  • exclude_unset (bool) – Whether to exclude fields that have not been explicitly set.

  • exclude_defaults (bool) – Whether to exclude fields that are set to their default value.

  • exclude_none (bool) – Whether to exclude fields that have a value of None.

  • exclude_computed_fields (bool) – Whether to exclude computed fields. While this can be useful for round-tripping, it is usually recommended to use the dedicated round_trip parameter instead.

  • round_trip (bool) – If True, dumped values should be valid as input for non-idempotent types such as Json[T].

  • warnings (bool | Literal['none', 'warn', 'error']) – How to handle serialization errors. False/”none” ignores them, True/”warn” logs errors, “error” raises a [PydanticSerializationError][pydantic_core.PydanticSerializationError].

  • fallback (Callable[[Any], Any] | None) – A function to call when an unknown value is encountered. If not provided, a [PydanticSerializationError][pydantic_core.PydanticSerializationError] error is raised.

  • serialize_as_any (bool) – Whether to serialize fields with duck-typing serialization behavior.

Returns:

A JSON string representation of the model.

Return type:

str

model_post_init(context, /)

Override this method to perform additional initialization after __init__ and model_construct. This is useful if you want to do some validation that requires the entire model to be initialized.

Parameters:

context (Any)

Return type:

None

register()

Register this engine in the global registry.

Adds this engine instance to the global EngineRegistry, making it available for lookup by name, ID, or type. This method is typically called after creating an engine to make it available throughout the application.

Returns:

Self for method chaining.

Return type:

Engine

Examples

>>> engine = (
...     MyEngine(name="gpt-4", engine_type=EngineType.LLM)
...     .register()
... )
>>> # The engine is now available in the registry
>>> from haive.core.engine.base.registry import EngineRegistry
>>> registry = EngineRegistry.get_instance()
>>> same_engine = registry.get(EngineType.LLM, "gpt-4")
>>> engine is same_engine
True
serialize_engine_type(engine_type)

Ensure engine_type is serialized as its value, not string representation.

Parameters:

engine_type (EngineType)

Return type:

str

Perform similarity search with configurable parameters.

Parameters:

  • query (str) – Query string

  • k (int | None) – Number of documents to retrieve (overrides default)

  • score_threshold (float | None) – Score threshold for filtering results

  • filter (dict[str, Any] | None) – Optional filter for the search

  • search_type (str | None) – Search type (similarity, mmr, etc.)

  • runnable_config (RunnableConfig | None) – Optional runtime configuration

Returns:

List of retrieved documents

Return type:

list[Document]

to_dict()

Convert engine to a dictionary suitable for serialization.

Creates a complete dictionary representation of this engine that can be used for serialization, persistence, or reconstruction. Includes class information to allow proper reconstruction of the specific engine type.

Returns:

Dictionary representation of the engine with all necessary

information for reconstruction.

Return type:

Dict[str, Any]

Examples

>>> engine = MyLLMEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> data = engine.to_dict()
>>> # Contains the class information for reconstruction
>>> data["engine_class"]
'haive.core.engine.llm.my_llm_engine.MyLLMEngine'
>>> # Serializable to JSON
>>> import json
>>> json_str = json.dumps(data)
to_json()

Convert engine to a JSON string representation.

Serializes this engine to a JSON string that can be stored, transmitted, or persisted. The serialization process ensures that all values are JSON-compatible.

Returns:

JSON string representation of the engine.

Return type:

str

Examples

>>> engine = MyLLMEngine(name="gpt-4", engine_type=EngineType.LLM)
>>> json_str = engine.to_json()
>>> # Can be saved to file
>>> with open("engine_config.json", "w") as f:
...     f.write(json_str)
with_config_overrides(overrides)

Create a new engine with configuration overrides applied.

Creates a new instance of this engine with the specified configuration overrides applied. This is useful for creating variations of an engine without modifying the original instance.

Parameters:

overrides (Dict[str, Any]) – Dictionary of configuration overrides to apply. Only keys that exist in the engine’s configuration will be applied.

Returns:

A new engine instance with the overrides applied.

Return type:

Engine

Examples

>>> original = MyLLMEngine(
...     name="gpt-4",
...     engine_type=EngineType.LLM,
...     temperature=0.7
... )
>>> # Create a variation with different temperature
>>> variation = original.with_config_overrides({"temperature": 0.3})
>>> # Original is unchanged
>>> original.temperature
0.7
>>> # New instance has override applied
>>> variation.temperature
0.3
>>> # Other properties remain the same
>>> variation.name
'gpt-4'
description: str | None
docstore_path: str
documents: list[Document]
embedding_model: BaseEmbeddingConfig
engine_type: EngineType
id: str
input_schema: type[BaseModel] | None
k: int
metadata: dict[str, Any]
model_computed_fields = {}
model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property model_extra: dict[str, Any] | None

Get extra fields set during validation.

Returns:

A dictionary of extra fields, or None if config.extra is not set to “allow”.

model_fields = {'description': FieldInfo(annotation=Union[str, NoneType], required=False, default=None, description='Optional description of this engine'), 'docstore_path': FieldInfo(annotation=str, required=False, default='docstore', description='Where to store raw and processed documents'), 'documents': FieldInfo(annotation=list[Document], required=False, default_factory=list, description='The raw documents to store'), 'embedding_model': FieldInfo(annotation=BaseEmbeddingConfig, required=False, default_factory=<lambda>, description='The embedding model to use for the vector store'), 'engine_type': FieldInfo(annotation=EngineType, required=False, default=<EngineType.VECTOR_STORE: 'vector_store'>), 'id': FieldInfo(annotation=str, required=False, default_factory=<lambda>, description='Unique identifier for this engine instance'), 'input_schema': FieldInfo(annotation=Union[type[BaseModel], NoneType], required=False, default=None, description='Input schema for this engine', exclude=True), 'k': FieldInfo(annotation=int, required=False, default=4, description='Default number of documents to retrieve'), 'metadata': FieldInfo(annotation=dict[str, Any], required=False, default_factory=dict, description='Additional metadata for this engine'), 'name': FieldInfo(annotation=str, required=False, default_factory=<lambda>, description='Name of this engine instance'), 'output_schema': FieldInfo(annotation=Union[type[BaseModel], NoneType], required=False, default=None, description='Output schema for this engine', exclude=True), 'score_threshold': FieldInfo(annotation=Union[float, NoneType], required=False, default=None, description='Score threshold for similarity search'), 'search_type': FieldInfo(annotation=str, required=False, default='similarity', description='Default search type (similarity, mmr, etc.)'), 'vector_store_kwargs': FieldInfo(annotation=dict[str, Any], required=False, default_factory=dict, description='Optional kwargs for the vector store'), 'vector_store_path': FieldInfo(annotation=str, required=False, default='vector_store', description='The path to the vector store'), 'vector_store_provider': FieldInfo(annotation=VectorStoreProvider, required=False, default=<VectorStoreProvider.FAISS: 'FAISS'>, description='The type of vector store to use'), 'version': FieldInfo(annotation=str, required=False, default='1.0.0', description='Version of this engine configuration')}
property model_fields_set: set[str]

Returns the set of fields that have been explicitly set on this model instance.

Returns:

A set of strings representing the fields that have been set,

i.e. that were not filled from defaults.

name: str
output_schema: type[BaseModel] | None
score_threshold: float | None
search_type: str
vector_store_kwargs: dict[str, Any]
vector_store_path: str
vector_store_provider: VectorStoreProvider
version: str
class haive.core.engine.vectorstore.VectorStoreProvider(*values)[source]

Bases: str, Enum

Enumeration of supported vector store providers.

This enum defines the built-in vector store providers supported by the Haive framework. Each provider corresponds to a specific vector database implementation with its own features, capabilities, and requirements.

The enum can be dynamically extended at runtime using the extend() method or through the VectorStoreProviderRegistry, allowing for custom providers without modifying the core code.

CHROMA

Chroma vector database

Type:

str

FAISS

Facebook AI Similarity Search

Type:

str

PINECONE

Pinecone managed vector database

Type:

str

WEAVIATE

Weaviate vector database

Type:

str

ZILLIZ

Zilliz cloud vector database

Type:

str

MILVUS

Milvus vector database

Type:

str

QDRANT

Qdrant vector database

Type:

str

IN_MEMORY

In-memory vector store (for testing/development)

Type:

str

PGVECTOR

PostgreSQL with pgvector extension

Type:

str

ELASTICSEARCH

Elasticsearch vector search

Type:

str

REDIS

Redis vector database

Type:

str

SUPABASE

Supabase vector store

Type:

str

MONGODB_ATLAS

MongoDB Atlas vector search

Type:

str

Azure Cognitive Search

Type:

str

OPENSEARCH

OpenSearch vector search

Type:

str

CASSANDRA

Apache Cassandra vector store

Type:

str

CLICKHOUSE

ClickHouse vector database

Type:

str

TYPESENSE

Typesense vector search

Type:

str

LANCEDB

LanceDB vector database

Type:

str

NEO4J

Neo4j vector search

Type:

str

Examples

>>> from haive.core.engine.vectorstore import VectorStoreProvider
>>> # Using an enum value
>>> provider = VectorStoreProvider.FAISS
>>> str(provider)
'FAISS'
>>> # Checking if a value is in the enum
>>> "Chroma" in [p.value for p in VectorStoreProvider]
True

Available Providers

Provider

Type

Best For

Chroma

Open Source

Local development, prototyping

Pinecone

Cloud Service

Production, managed infrastructure

FAISS

Local/Memory

High-performance similarity search

Weaviate

Open Source

GraphQL queries, hybrid search
classmethod extend(name, value)[source]

Extend the enum with a new provider value.

Parameters:

  • name (str) – The enum member name (e.g., ‘MY_PROVIDER’)

  • value (str) – The string value (e.g., ‘MyProvider’)

Return type:

None

Note

This is a workaround to extend an Enum at runtime. For proper type hinting, use the VectorStoreProviderRegistry instead.

AZURE_SEARCH = 'AzureSearch'
CASSANDRA = 'Cassandra'
CHROMA = 'Chroma'
CLICKHOUSE = 'ClickHouse'
ELASTICSEARCH = 'Elasticsearch'
FAISS = 'FAISS'
IN_MEMORY = 'InMemory'
LANCEDB = 'LanceDB'
MILVUS = 'Milvus'
MONGODB_ATLAS = 'MongoDBAtlas'
NEO4J = 'Neo4j'
OPENSEARCH = 'OpenSearch'
PGVECTOR = 'PGVector'
PINECONE = 'Pinecone'
QDRANT = 'Qdrant'
REDIS = 'Redis'
SUPABASE = 'Supabase'
TYPESENSE = 'Typesense'
WEAVIATE = 'Weaviate'
ZILLIZ = 'Zilliz'

Functions

Architecture

        graph LR
    A[Documents] --> B[Embeddings]
    B --> C[Vector Store]
    C --> D[Similarity Search]
    D --> E[Retrieved Documents]

    subgraph "Vector Store Types"
        F[Local/File-based]
        G[Cloud Services]
        H[In-Memory]
    end

Performance Considerations

Optimization Tips

  • Index Type: Different providers support different index types (HNSW, IVF, etc.)

  • Batch Operations: Use batch operations for better performance when adding many documents

  • Connection Pooling: Configured automatically for cloud providers

  • Caching: In-memory caching for frequently accessed embeddings

Warning

Large-scale deployments should consider:

  • Index size limitations

  • Query latency requirements

  • Cost per query/storage

  • Data persistence needs

Extended Examples

RAG Pipeline Example

Migration Between Providers

# Migrate from one provider to another
def migrate_vectorstore(source_config, target_config):
    """Migrate documents between vector stores."""
    # Extract from source
    source_vs = source_config.instantiate()
    docs = source_vs.similarity_search("", k=1000)  # Get all

    # Load into target
    target_vs = target_config.instantiate()
    target_vs.add_documents(docs)

    return target_vs

See Also