Client

A module for writing and querying vectors to Postgres 
_ = load_dotenv(find_dotenv(), override=True)
service_url = os.environ['TIMESCALE_SERVICE_URL']

source

uuid_from_time

 uuid_from_time (time_arg=None, node=None, clock_seq=None)

Converts a datetime or timestamp to a type 1 uuid.UUID.

Type Default Details
time_arg NoneType None The time to use for the timestamp portion of the UUID.
This can either be a datetime object or a timestamp in seconds
(as returned from time.time()).
node NoneType None Bytes for the UUID (up to 48 bits). If not specified, this
field is randomized.
clock_seq NoneType None Clock sequence field for the UUID (up to 14 bits). If not specified,
a random sequence is generated.
Returns uuid.UUID: For the given time, node, and clock sequence

Index Definitions

source

DiskAnnIndex

 DiskAnnIndex (search_list_size:Optional[int]=None,
               num_neighbors:Optional[int]=None,
               max_alpha:Optional[float]=None,
               storage_layout:Optional[str]=None,
               num_dimensions:Optional[int]=None,
               num_bits_per_dimension:Optional[int]=None)

Timescale’s vector index.

source

HNSWIndex

 HNSWIndex (m:Optional[int]=None, ef_construction:Optional[int]=None)

Pgvector’s hnsw index.

source

IvfflatIndex

 IvfflatIndex (num_records:Optional[int]=None,
               num_lists:Optional[int]=None)

Pgvector’s ivfflat index.

source

BaseIndex

 BaseIndex ()

Initialize self. See help(type(self)) for accurate signature.

Query Params

source

HNSWIndexParams

 HNSWIndexParams (ef_search:int)

Initialize self. See help(type(self)) for accurate signature.

source

IvfflatIndexParams

 IvfflatIndexParams (probes:int)

Initialize self. See help(type(self)) for accurate signature.

source

DiskAnnIndexParams

 DiskAnnIndexParams (search_list_size:Optional[int]=None,
                     rescore:Optional[int]=None)

Initialize self. See help(type(self)) for accurate signature.

source

QueryParams

 QueryParams (params:dict[str,typing.Any])

Initialize self. See help(type(self)) for accurate signature.

Query Builder

source

UUIDTimeRange

 UUIDTimeRange (start_date:Union[datetime.datetime,str,NoneType]=None,
                end_date:Union[datetime.datetime,str,NoneType]=None,
                time_delta:Optional[datetime.timedelta]=None,
                start_inclusive=True, end_inclusive=False)

*A UUIDTimeRange is a time range predicate on the UUID Version 1 timestamps.

Note that naive datetime objects are interpreted as local time on the python client side and converted to UTC before being sent to the database.*

source

Predicates

 Predicates (*clauses:Union[ForwardRef('Predicates'),Tuple[str,Union[str,i
             nt,float,datetime.datetime,list,tuple]],Tuple[str,str,Union[s
             tr,int,float,datetime.datetime,list,tuple]],str,int,float,dat
             etime.datetime,list,tuple], operator:str='AND')

Predicates class defines predicates on the object metadata. Predicates can be combined using logical operators (&, |, and ~).

Type Default Details
clauses Union Predicate clauses. Can be either another Predicates object or a tuple of the form (field, operator, value) or (field, value).
operator str AND

source

QueryBuilder

 QueryBuilder (table_name:str, num_dimensions:int, distance_type:str,
               id_type:str,
               time_partition_interval:Optional[datetime.timedelta],
               infer_filters:bool, schema_name:Optional[str])

Initializes a base Vector object to generate queries for vector clients.

Type Details
table_name str The name of the table.
num_dimensions int The number of dimensions for the embedding vector.
distance_type str The distance type for indexing.
id_type str The type of the id column. Can be either ‘UUID’ or ‘TEXT’.
time_partition_interval Optional The time interval for partitioning the table (optional).
infer_filters bool Whether to infer start and end times from the special __start_date and __end_date filters.
schema_name Optional The schema name for the table (optional, uses the database’s default schema if not specified).
Returns None

source

QueryBuilder.get_create_query

 QueryBuilder.get_create_query ()

Generates a query to create the tables, indexes, and extensions needed to store the vector data.

Async Client

source

Async

 Async (service_url:str, table_name:str, num_dimensions:int,
        distance_type:str='cosine', id_type='UUID',
        time_partition_interval:Optional[datetime.timedelta]=None,
        max_db_connections:Optional[int]=None, infer_filters:bool=True,
        schema_name:Optional[str]=None)

Initializes a async client for storing vector data.

Type Default Details
service_url str The connection string for the database.
table_name str The name of the table.
num_dimensions int The number of dimensions for the embedding vector.
distance_type str cosine The distance type for indexing.
id_type str UUID The type of the id column. Can be either ‘UUID’ or ‘TEXT’.
time_partition_interval Optional None The time interval for partitioning the table (optional).
max_db_connections Optional None
infer_filters bool True Whether to infer start and end times from the special __start_date and __end_date filters.
schema_name Optional None The schema name for the table (optional, uses the database’s default schema if not specified).
Returns None

source

Async.create_tables

 Async.create_tables ()

Creates necessary tables.

source

Async.create_tables

 Async.create_tables ()

Creates necessary tables.

source

Async.search

 Async.search (query_embedding:Optional[List[float]]=None, limit:int=10,
               filter:Union[Dict[str,str],List[Dict[str,str]],NoneType]=No
               ne, predicates:Optional[__main__.Predicates]=None,
               uuid_time_filter:Optional[__main__.UUIDTimeRange]=None,
               query_params:Optional[__main__.QueryParams]=None)

Retrieves similar records using a similarity query.

Type Default Details
query_embedding Optional None The query embedding vector.
limit int 10 The number of nearest neighbors to retrieve.
filter Union None A filter for metadata. Should be specified as a key-value object or a list of key-value objects (where any objects in the list are matched).
predicates Optional None A Predicates object to filter the results. Predicates support more complex queries than the filter parameter. Predicates can be combined using logical operators (&, |, and ~).
uuid_time_filter Optional None A UUIDTimeRange object to filter the results by time using the id column.
query_params Optional None
Returns List: List of similar records.

Usage Example

for schema in ["tschema", None]:
    vec = Async(service_url, "data_table", 2, schema_name=schema)
    await vec.create_tables()
    empty = await vec.table_is_empty()
    assert empty
    await vec.upsert([(uuid.uuid4(), {"key": "val"}, "the brown fox", [1.0, 1.2])])
    empty = await vec.table_is_empty()
    assert not empty

    await vec.upsert([
        (uuid.uuid4(), '''{"key":"val"}''', "the brown fox", [1.0, 1.3]),
        (uuid.uuid4(), '''{"key":"val2", "key_10": "10", "key_11": "11.3"}''', "the brown fox", [1.0, 1.4]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.5]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.6]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.6]),
        (uuid.uuid4(), '''{"key2":"val2"}''', "the brown fox", [1.0, 1.7]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.9]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 100.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 101.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key_1":"val_1", "key_2":"val_2"}''',
        "the brown fox", [1.0, 1.8]),

        (uuid.uuid4(), '''{"key0": [1,2,3,4]}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key0": [8,9,"A"]}''', "the brown fox", [1.0, 1.8]), # mixed types
        (uuid.uuid4(), '''{"key0": [5,6,7], "key3": 3}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key0": ["B", "C"]}''', "the brown fox", [1.0, 1.8]),
    
    ])

    await vec.create_embedding_index(IvfflatIndex())
    await vec.drop_embedding_index()
    await vec.create_embedding_index(IvfflatIndex(100))
    await vec.drop_embedding_index()
    await vec.create_embedding_index(HNSWIndex())
    await vec.drop_embedding_index()
    await vec.create_embedding_index(HNSWIndex(20, 125))
    await vec.drop_embedding_index()
    await vec.create_embedding_index(DiskAnnIndex())
    await vec.drop_embedding_index()
    await vec.create_embedding_index(DiskAnnIndex(50, 50, 1.5, "memory_optimized", 2, 1))

    rec = await vec.search([1.0, 2.0])
    assert len(rec) == 10
    rec = await vec.search([1.0, 2.0], limit=4)
    assert len(rec) == 4
    rec = await vec.search(limit=4)
    assert len(rec) == 4
    rec = await vec.search([1.0, 2.0], limit=4, filter={"key2": "val2"})
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, filter={"key2": "does not exist"})
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, filter={"key_1": "val_1"})
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], filter={"key_1": "val_1", "key_2": "val_2"})
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, filter={"key_1": "val_1", "key_2": "val_3"})
    assert len(rec) == 0
    rec = await vec.search(limit=4, filter={"key_1": "val_1", "key_2": "val_3"})
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 2
    rec = await vec.search(limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 2

    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}, {"no such key": "no such val"}])
    assert len(rec) == 2

    assert isinstance(rec[0][SEARCH_RESULT_METADATA_IDX], dict)
    assert isinstance(rec[0]["metadata"], dict)
    assert rec[0]["contents"] == "the brown fox"


    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(("key", "val2")))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(("key", "==", "val2")))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key", "==", "val2"))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<", 100))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<", 10))
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<=", 10))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<=", 10.0))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_11", "<=", 11.3))
    assert len(rec) == 1
    rec = await vec.search(limit=4, predicates=Predicates("key_11", ">=", 11.29999))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_11", "<", 11.299999))
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [1, 2]))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [3, 7]))
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [42]))
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [4]))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [9, "A"]))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", ["A"]))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", ("C", "B")))
    assert len(rec) == 1

    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(*[("key", "val2"), ("key_10", "<", 100)]))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(("key", "val2"), ("key_10", "<", 100), operator='AND'))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(("key", "val2"), ("key_2", "val_2"), operator='OR'))
    assert len(rec) == 2
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<", 100) & (Predicates("key","==", "val2",) | Predicates("key_2", "==", "val_2"))) 
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key_10", "<", 100) and (Predicates("key","==", "val2") or Predicates("key_2","==", "val_2"))) 
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [6,7]) and Predicates("key3","==", 3))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates("key0", "@>", [6,7]) and Predicates("key3","==", 6))
    assert len(rec) == 0
    rec = await vec.search(limit=4, predicates=~Predicates(("key", "val2"), ("key_10", "<", 100)))
    assert len(rec) == 4

    raised = False
    try:
        # can't upsert using both keys and dictionaries
        await vec.upsert([
            (uuid.uuid4(), {"key": "val"}, "the brown fox", [1.0, 1.2]),
            (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.2])
        ])
    except ValueError as e:
        raised = True
    assert raised

    raised = False
    try:
        # can't upsert using both keys and dictionaries opposite order
        await vec.upsert([
            (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.2]),
            (uuid.uuid4(),  {"key": "val"}, "the brown fox", [1.0, 1.2])
        ])
    except BaseException as e:
        raised = True
    assert raised

    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 2
    await vec.delete_by_ids([rec[0]["id"]])
    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 1
    await vec.delete_by_metadata([{"key_1": "val_1"}, {"key2": "val2"}])
    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key2": "val"}])
    assert len(rec) == 4
    await vec.delete_by_metadata([{"key2": "val"}])
    rec = await vec.search([1.0, 2.0], limit=4, filter=[{"key2": "val"}])
    assert len(rec) == 0

    assert not await vec.table_is_empty()
    await vec.delete_all()
    assert await vec.table_is_empty()

    await vec.drop_table()
    await vec.close()

    vec = Async(service_url, "data_table", 2, id_type="TEXT")
    await vec.create_tables()
    empty = await vec.table_is_empty()
    assert empty
    await vec.upsert([("Not a valid UUID", {"key": "val"}, "the brown fox", [1.0, 1.2])])
    empty = await vec.table_is_empty()
    assert not empty
    await vec.delete_by_ids(["Not a valid UUID"])
    empty = await vec.table_is_empty()
    assert empty
    await vec.drop_table()
    await vec.close()

    vec = Async(service_url, "data_table", 2, time_partition_interval=timedelta(seconds=60))
    await vec.create_tables()
    empty = await vec.table_is_empty()
    assert empty
    id = uuid.uuid1()
    await vec.upsert([(id, {"key": "val"}, "the brown fox", [1.0, 1.2])])
    empty = await vec.table_is_empty()
    assert not empty
    await vec.delete_by_ids([id])
    empty = await vec.table_is_empty()
    assert empty

    raised = False
    try:
        # can't upsert with uuid type 4 in time partitioned table
        await vec.upsert([
            (uuid.uuid4(),  {"key": "val"}, "the brown fox", [1.0, 1.2])
        ])
    except BaseException as e:
        raised = True
    assert raised

    specific_datetime = datetime(2018, 8, 10, 15, 30, 0)
    await vec.upsert([
            # current time
            (uuid.uuid1(),  {"key": "val"}, "the brown fox", [1.0, 1.2]),
            #time in 2018
            (uuid_from_time(specific_datetime),  {"key": "val"}, "the brown fox", [1.0, 1.2])
    ])
    assert not await vec.table_is_empty()

    #check all the possible ways to specify a date range
    async def search_date(start_date, end_date, expected):
        #using uuid_time_filter
        rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(start_date, end_date))
        assert len(rec) == expected
        rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(str(start_date), str(end_date)))
        assert len(rec) == expected
    
        #using filters
        filter = {}
        if start_date is not None:
            filter["__start_date"] = start_date
        if end_date is not None:
            filter["__end_date"] = end_date
        rec = await vec.search([1.0, 2.0], limit=4, filter=filter)
        assert len(rec) == expected
        #using filters with string dates
        filter = {}
        if start_date is not None:
            filter["__start_date"] = str(start_date)
        if end_date is not None:
            filter["__end_date"] = str(end_date)
        rec = await vec.search([1.0, 2.0], limit=4, filter=filter)
        assert len(rec) == expected
        #using predicates
        predicates = []
        if start_date is not None:
            predicates.append(("__uuid_timestamp", ">=", start_date))
        if end_date is not None:
            predicates.append(("__uuid_timestamp", "<", end_date))
        rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(*predicates))
        assert len(rec) == expected
        #using predicates with string dates
        predicates = []
        if start_date is not None:
            predicates.append(("__uuid_timestamp", ">=", str(start_date)))
        if end_date is not None:
            predicates.append(("__uuid_timestamp", "<", str(end_date)))
        rec = await vec.search([1.0, 2.0], limit=4, predicates=Predicates(*predicates))
        assert len(rec) == expected

    await search_date(specific_datetime-timedelta(days=7), specific_datetime+timedelta(days=7), 1)
    await search_date(specific_datetime-timedelta(days=7), None, 2)
    await search_date(None, specific_datetime+timedelta(days=7), 1)
    await search_date(specific_datetime-timedelta(days=7), specific_datetime-timedelta(days=2), 0)

    #check timedelta handling
    rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(start_date=specific_datetime, time_delta=timedelta(days=7)))
    assert len(rec) == 1
    #end is exclusive
    rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(end_date=specific_datetime, time_delta=timedelta(days=7)))
    assert len(rec) == 0
    rec = await vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(end_date=specific_datetime+timedelta(seconds=1), time_delta=timedelta(days=7)))
    assert len(rec) == 1
    rec = await vec.search([1.0, 2.0], limit=4, query_params=DiskAnnIndexParams(10, 5))
    assert len(rec) == 2
    rec = await vec.search([1.0, 2.0], limit=4, query_params=DiskAnnIndexParams(100))
    assert len(rec) == 2
    await vec.drop_table()
    await vec.close()

Sync Client

source

Sync

 Sync (service_url:str, table_name:str, num_dimensions:int,
       distance_type:str='cosine', id_type='UUID',
       time_partition_interval:Optional[datetime.timedelta]=None,
       max_db_connections:Optional[int]=None, infer_filters:bool=True,
       schema_name:Optional[str]=None)

Initializes a sync client for storing vector data.

Type Default Details
service_url str The connection string for the database.
table_name str The name of the table.
num_dimensions int The number of dimensions for the embedding vector.
distance_type str cosine The distance type for indexing.
id_type str UUID The type of the primary id column. Can be either ‘UUID’ or ‘TEXT’.
time_partition_interval Optional None The time interval for partitioning the table (optional).
max_db_connections Optional None
infer_filters bool True Whether to infer start and end times from the special __start_date and __end_date filters.
schema_name Optional None The schema name for the table (optional, uses the database’s default schema if not specified).
Returns None

source

Sync.create_tables

 Sync.create_tables ()

Creates necessary tables.

source

Sync.upsert

 Sync.upsert (records)

Performs upsert operation for multiple records.

Type Details
records Records to upsert.
Returns None 
/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/fastcore/docscrape.py:230: UserWarning: potentially wrong underline length... 
Returns 
-------- in 
Retrieves similar records using a similarity query.
...
  else: warn(msg)

source

Sync.search

 Sync.search (query_embedding:Optional[List[float]]=None, limit:int=10,
              filter:Union[Dict[str,str],List[Dict[str,str]],NoneType]=Non
              e, predicates:Optional[__main__.Predicates]=None,
              uuid_time_filter:Optional[__main__.UUIDTimeRange]=None,
              query_params:Optional[__main__.QueryParams]=None)

Retrieves similar records using a similarity query.

Type Default Details
query_embedding Optional None The query embedding vector.
limit int 10 The number of nearest neighbors to retrieve.
filter Union None A filter for metadata. Should be specified as a key-value object or a list of key-value objects (where any objects in the list are matched).
predicates Optional None A Predicates object to filter the results. Predicates support more complex queries than the filter parameter. Predicates can be combined using logical operators (&, |, and ~).
uuid_time_filter Optional None
query_params Optional None
Returns List: List of similar records.

Usage Example:

for schema in [None, "tschema"]:  
    vec = Sync(service_url, "data_table", 2, schema_name=schema)
    vec.create_tables()
    empty = vec.table_is_empty()

    assert empty
    vec.upsert([(uuid.uuid4(), {"key": "val"}, "the brown fox", [1.0, 1.2])])
    empty = vec.table_is_empty()
    assert not empty

    vec.upsert([
        (uuid.uuid4(), '''{"key":"val"}''', "the brown fox", [1.0, 1.3]),
        (uuid.uuid4(), '''{"key":"val2"}''', "the brown fox", [1.0, 1.4]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.5]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.6]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.6]),
        (uuid.uuid4(), '''{"key2":"val2"}''', "the brown fox", [1.0, 1.7]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.9]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 100.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 101.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key_1":"val_1", "key_2":"val_2"}''',
        "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key0": [1,2,3,4]}''', "the brown fox", [1.0, 1.8]),
        (uuid.uuid4(), '''{"key0": [5,6,7], "key3": 3}''', "the brown fox", [1.0, 1.8]),
    ])

    vec.create_embedding_index(IvfflatIndex())
    vec.drop_embedding_index()
    vec.create_embedding_index(IvfflatIndex(100))
    vec.drop_embedding_index()
    vec.create_embedding_index(HNSWIndex())
    vec.drop_embedding_index()
    vec.create_embedding_index(HNSWIndex(20, 125))
    vec.drop_embedding_index()
    vec.create_embedding_index(DiskAnnIndex())
    vec.drop_embedding_index()
    vec.create_embedding_index(DiskAnnIndex(50, 50, 1.5))

    rec = vec.search([1.0, 2.0])
    assert len(rec) == 10
    rec = vec.search(np.array([1.0, 2.0]))
    assert len(rec) == 10
    rec = vec.search([1.0, 2.0], limit=4)
    assert len(rec) == 4
    rec = vec.search(limit=4)
    assert len(rec) == 4
    rec = vec.search([1.0, 2.0], limit=4, filter={"key2": "val2"})
    assert len(rec) == 1
    rec = vec.search([1.0, 2.0], limit=4, filter={"key2": "does not exist"})
    assert len(rec) == 0
    rec = vec.search(limit=4, filter={"key2": "does not exist"})
    assert len(rec) == 0
    rec = vec.search([1.0, 2.0], limit=4, filter={"key_1": "val_1"})
    assert len(rec) == 1
    rec = vec.search([1.0, 2.0], filter={"key_1": "val_1", "key_2": "val_2"})
    assert len(rec) == 1
    rec = vec.search([1.0, 2.0], limit=4, filter={
                    "key_1": "val_1", "key_2": "val_3"})
    assert len(rec) == 0

    rec = vec.search([1.0, 2.0], limit=4, filter=[
                    {"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 2

    rec = vec.search([1.0, 2.0], limit=4, filter=[{"key_1": "val_1"}, {
                    "key2": "val2"}, {"no such key": "no such val"}])
    assert len(rec) == 2

    raised = False
    try:
        # can't upsert using both keys and dictionaries
        await vec.upsert([
            (uuid.uuid4(), {"key": "val"}, "the brown fox", [1.0, 1.2]),
            (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.2])
        ])
    except ValueError as e:
        raised = True
    assert raised

    raised = False
    try:
        # can't upsert using both keys and dictionaries opposite order
        await vec.upsert([
            (uuid.uuid4(), '''{"key2":"val"}''', "the brown fox", [1.0, 1.2]),
            (uuid.uuid4(),  {"key": "val"}, "the brown fox", [1.0, 1.2])
        ])
    except BaseException as e:
        raised = True
    assert raised

    rec = vec.search([1.0, 2.0], filter={"key_1": "val_1", "key_2": "val_2"})
    assert rec[0][SEARCH_RESULT_CONTENTS_IDX] == 'the brown fox'
    assert rec[0]["contents"] == 'the brown fox'
    assert rec[0][SEARCH_RESULT_METADATA_IDX] == {
        'key_1': 'val_1', 'key_2': 'val_2'}
    assert rec[0]["metadata"] == {
        'key_1': 'val_1', 'key_2': 'val_2'}
    assert isinstance(rec[0][SEARCH_RESULT_METADATA_IDX], dict)
    assert rec[0][SEARCH_RESULT_DISTANCE_IDX] == 0.0009438353921149556
    assert rec[0]["distance"] == 0.0009438353921149556

    rec = vec.search([1.0, 2.0], limit=4, predicates=Predicates("key","==", "val2"))
    assert len(rec) == 1

    rec = vec.search([1.0, 2.0], limit=4, filter=[
                    {"key_1": "val_1"}, {"key2": "val2"}])
    len(rec) == 2
    vec.delete_by_ids([rec[0][SEARCH_RESULT_ID_IDX]])
    rec = vec.search([1.0, 2.0], limit=4, filter=[
                    {"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 1
    vec.delete_by_metadata([{"key_1": "val_1"}, {"key2": "val2"}])
    rec = vec.search([1.0, 2.0], limit=4, filter=[
                    {"key_1": "val_1"}, {"key2": "val2"}])
    assert len(rec) == 0
    rec = vec.search([1.0, 2.0], limit=4, filter=[{"key2": "val"}])
    assert len(rec) == 4
    vec.delete_by_metadata([{"key2": "val"}])
    rec = vec.search([1.0, 2.0], limit=4, filter=[{"key2": "val"}])
    len(rec) == 0

    assert not vec.table_is_empty()
    vec.delete_all()
    assert vec.table_is_empty()

    vec.drop_table()
    vec.close()

    vec = Sync(service_url, "data_table", 2, id_type="TEXT", schema_name=schema)
    vec.create_tables()
    assert vec.table_is_empty()
    vec.upsert([("Not a valid UUID", {"key": "val"}, "the brown fox", [1.0, 1.2])])
    assert not vec.table_is_empty()
    vec.delete_by_ids(["Not a valid UUID"])
    assert vec.table_is_empty()
    vec.drop_table()
    vec.close()

    vec = Sync(service_url, "data_table", 2, time_partition_interval=timedelta(seconds=60), schema_name=schema)
    vec.create_tables()
    assert vec.table_is_empty()
    id = uuid.uuid1()
    vec.upsert([(id, {"key": "val"}, "the brown fox", [1.0, 1.2])])
    assert not vec.table_is_empty()
    vec.delete_by_ids([id])
    assert vec.table_is_empty()
    raised = False
    try:
        # can't upsert with uuid type 4 in time partitioned table
        vec.upsert([
            (uuid.uuid4(),  {"key": "val"}, "the brown fox", [1.0, 1.2])
        ])
        #pass
    except BaseException as e:
        raised = True
    assert raised

    specific_datetime = datetime(2018, 8, 10, 15, 30, 0)
    vec.upsert([
            # current time
            (uuid.uuid1(),  {"key": "val"}, "the brown fox", [1.0, 1.2]),
            #time in 2018
            (uuid_from_time(specific_datetime),  {"key": "val"}, "the brown fox", [1.0, 1.2])
    ])

    def search_date(start_date, end_date, expected):
        #using uuid_time_filter
        rec = vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(start_date, end_date))
        assert len(rec) == expected
        rec = vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(str(start_date), str(end_date)))
        assert len(rec) == expected
        
        #using filters
        filter = {}
        if start_date is not None:
            filter["__start_date"] = start_date
        if end_date is not None:
            filter["__end_date"] = end_date
        rec = vec.search([1.0, 2.0], limit=4, filter=filter)
        assert len(rec) == expected
        #using filters with string dates
        filter = {}
        if start_date is not None:
            filter["__start_date"] = str(start_date)
        if end_date is not None:
            filter["__end_date"] = str(end_date)
        rec = vec.search([1.0, 2.0], limit=4, filter=filter)
        assert len(rec) == expected
        #using predicates
        predicates = []
        if start_date is not None:
            predicates.append(("__uuid_timestamp", ">=", start_date))
        if end_date is not None:
            predicates.append(("__uuid_timestamp", "<", end_date))
        rec = vec.search([1.0, 2.0], limit=4, predicates=Predicates(*predicates))
        assert len(rec) == expected
        #using predicates with string dates
        predicates = []
        if start_date is not None:
            predicates.append(("__uuid_timestamp", ">=", str(start_date)))
        if end_date is not None:
            predicates.append(("__uuid_timestamp", "<", str(end_date)))
        rec = vec.search([1.0, 2.0], limit=4, predicates=Predicates(*predicates))
        assert len(rec) == expected

    assert not vec.table_is_empty()

    search_date(specific_datetime-timedelta(days=7), specific_datetime+timedelta(days=7), 1)
    search_date(specific_datetime-timedelta(days=7), None, 2)
    search_date(None, specific_datetime+timedelta(days=7), 1)
    search_date(specific_datetime-timedelta(days=7), specific_datetime-timedelta(days=2), 0)

    #check timedelta handling
    rec = vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(start_date=specific_datetime, time_delta=timedelta(days=7)))
    assert len(rec) == 1
    #end is exclusive
    rec =  vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(end_date=specific_datetime, time_delta=timedelta(days=7)))
    assert len(rec) == 0
    rec = vec.search([1.0, 2.0], limit=4, uuid_time_filter=UUIDTimeRange(end_date=specific_datetime+timedelta(seconds=1), time_delta=timedelta(days=7)))
    assert len(rec) == 1
    rec = vec.search([1.0, 2.0], limit=4, query_params=DiskAnnIndexParams(10, 5))
    assert len(rec) == 2
    rec = vec.search([1.0, 2.0], limit=4, query_params=DiskAnnIndexParams(100, rescore=2))
    assert len(rec) == 2
    vec.drop_table()
    vec.close()