Skip to content

Library API

This page documents the arcticdb.version_store.library module. This module is the main interface exposing read/write functionality within a given Arctic instance.

The key functionality is exposed through arcticdb.version_store.library.Library instances. See the Arctic API section for notes on how to create these. The other types exposed in this module are less important and are used as part of the signature of arcticdb.version_store.library.Library instance methods.

arcticdb.version_store.library.Library

The main interface exposing read/write functionality within a given Arctic instance.

Arctic libraries contain named symbols which are the atomic unit of data storage within Arctic. Symbols contain data that in most cases resembles a DataFrame and are versioned such that all modifying operations can be tracked and reverted.

Instances of this class provide a number of primitives to write, modify and remove symbols, as well as also providing methods to manage library snapshots. For more information on snapshots please see the snapshot method.

Arctic libraries support concurrent writes and reads to multiple symbols as well as concurrent reads to a single symbol. However, concurrent writers to a single symbol are not supported other than for primitives that explicitly state support for single-symbol concurrent writes.

name property

name

The name of this library.

append

append(
    symbol: str,
    data: NormalizableType,
    metadata: Any = None,
    prune_previous_versions: bool = False,
    validate_index: bool = True,
) -> Optional[VersionedItem]

Appends the given data to the existing, stored data. Append always appends along the index. A new version will be created to reference the newly-appended data. Append only accepts data for which the index of the first row is equal to or greater than the index of the last row in the existing data.

Appends containing differing column sets to the existing data are only possible if the library has been configured to support dynamic schemas.

Note that append is not designed for multiple concurrent writers over a single symbol.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

data

Data to be written.

TYPE: NormalizableType

metadata

Optional metadata to persist along with the new symbol version. Note that the metadata is not combined in any way with the metadata stored in the previous version.

TYPE: Any DEFAULT: None

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

default

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

validate_index

If True, verify that the index of data supports date range searches and update operations. This tests that the data is sorted in ascending order, using Pandas DataFrame.index.is_monotonic_increasing.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the written symbol in the store.

RAISES DESCRIPTION
UnsortedDataException

If data is unsorted, when validate_index is set to True.

Examples:

>>> df = pd.DataFrame(
...    {'column': [1,2,3]},
...    index=pd.date_range(start='1/1/2018', end='1/03/2018')
... )
>>> df
            column
2018-01-01       1
2018-01-02       2
2018-01-03       3
>>> lib.write("symbol", df)
>>> to_append_df = pd.DataFrame(
...    {'column': [4,5,6]},
...    index=pd.date_range(start='1/4/2018', end='1/06/2018')
... )
>>> to_append_df
            column
2018-01-04       4
2018-01-05       5
2018-01-06       6
>>> lib.append("symbol", to_append_df)
>>> lib.read("symbol").data
            column
2018-01-01       1
2018-01-02       2
2018-01-03       3
2018-01-04       4
2018-01-05       5
2018-01-06       6

append_batch

append_batch(
    append_payloads: List[WritePayload],
    prune_previous_versions: bool = False,
    validate_index=True,
) -> List[Union[VersionedItem, DataError]]

Append data to multiple symbols in a batch fashion. This is more efficient than making multiple append calls in succession as some constant-time operations can be executed only once rather than once for each element of append_payloads. Note that this isn't an atomic operation - it's possible for one symbol to be fully written and readable before another symbol.

PARAMETER DESCRIPTION
append_payloads

Symbols and their corresponding data. There must not be any duplicate symbols in append_payloads.

TYPE: `List[WritePayload]`

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

validate_index

Verify that each entry in the batch has an index that supports date range searches and update operations. This tests that the data is sorted in ascending order, using Pandas DataFrame.index.is_monotonic_increasing.

DEFAULT: True

RETURNS DESCRIPTION
List[Union[VersionedItem, DataError]]

List of versioned items. i-th entry corresponds to i-th element of append_payloads. Each result correspond to a structure containing metadata and version number of the affected symbol in the store. If a key error or any other internal exception is raised, a DataError object is returned, with symbol, error_code, error_category, and exception_string properties.

RAISES DESCRIPTION
ArcticDuplicateSymbolsInBatchException

When duplicate symbols appear in payload.

ArcticUnsupportedDataTypeException

If data that is not of NormalizableType appears in any of the payloads.

compact_symbol_list

compact_symbol_list() -> None

Compact the symbol list cache into a single key in the storage

RETURNS DESCRIPTION
The number of symbol list keys prior to compaction
RAISES DESCRIPTION
PermissionException

Library has been opened in read-only mode

InternalException

Storage lock required to compact the symbol list could not be acquired

defragment_symbol_data

defragment_symbol_data(
    symbol: str,
    segment_size: Optional[int] = None,
    prune_previous_versions: bool = False,
) -> VersionedItem

Compacts fragmented segments by merging row-sliced segments (https://docs.arcticdb.io/technical/on_disk_storage/#data-layer). This method calls is_symbol_fragmented to determine whether to proceed with the defragmentation operation.

CAUTION - Please note that a major restriction of this method at present is that any column slicing present on the data will be removed in the new version created as a result of this method. As a result, if the impacted symbol has more than 127 columns (default value), the performance of selecting individual columns of the symbol (by using the columns parameter) may be negatively impacted in the defragmented version. If your symbol has less than 127 columns this caveat does not apply. For more information, please see columns_per_segment here:

https://docs.arcticdb.io/api/arcticdb/arcticdb.LibraryOptions

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

segment_size

Target for maximum no. of rows per segment, after compaction. If parameter is not provided, library option - "segment_row_size" will be used Note that no. of rows per segment, after compaction, may exceed the target. It is for achieving smallest no. of segment after compaction. Please refer to below example for further explanation.

TYPE: Optional[int] DEFAULT: None

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the defragmented symbol in the store.

RAISES DESCRIPTION
1002 ErrorCategory.INTERNAL:E_ASSERTION_FAILURE

If is_symbol_fragmented returns false.

2001 ErrorCategory.NORMALIZATION:E_UNIMPLEMENTED_INPUT_TYPE

If library option - "bucketize_dynamic" is ON

Examples:

>>> lib.write("symbol", pd.DataFrame({"A": [0]}, index=[pd.Timestamp(0)]))
>>> lib.append("symbol", pd.DataFrame({"A": [1, 2]}, index=[pd.Timestamp(1), pd.Timestamp(2)]))
>>> lib.append("symbol", pd.DataFrame({"A": [3]}, index=[pd.Timestamp(3)]))
>>> lib_tool = lib._dev_tools.library_tool()
>>> lib_tool.read_index(sym)
                    start_index                     end_index  version_id stream_id          creation_ts          content_hash  index_type  key_type  start_col  end_col  start_row  end_row
1970-01-01 00:00:00.000000000 1970-01-01 00:00:00.000000001          20    b'sym'  1678974096622685727   6872717287607530038          84         2          1        2          0        1
1970-01-01 00:00:00.000000001 1970-01-01 00:00:00.000000003          21    b'sym'  1678974096931527858  12345256156783683504          84         2          1        2          1        3
1970-01-01 00:00:00.000000003 1970-01-01 00:00:00.000000004          22    b'sym'  1678974096970045987   7952936283266921920          84         2          1        2          3        4
>>> lib.version_store.defragment_symbol_data("symbol", 2)
>>> lib_tool.read_index(sym)  # Returns two segments rather than three as a result of the defragmentation operation
                    start_index                     end_index  version_id stream_id          creation_ts         content_hash  index_type  key_type  start_col  end_col  start_row  end_row
1970-01-01 00:00:00.000000000 1970-01-01 00:00:00.000000003          23    b'sym'  1678974097067271451  5576804837479525884          84         2          1        2          0        3
1970-01-01 00:00:00.000000003 1970-01-01 00:00:00.000000004          23    b'sym'  1678974097067427062  7952936283266921920          84         2          1        2          3        4
Notes

Config map setting - SymbolDataCompact.SegmentCount will be replaced by a library setting in the future. This API will allow overriding the setting as well.

delete

delete(
    symbol: str,
    versions: Optional[Union[int, Iterable[int]]] = None,
)

Delete all versions of the symbol from the library, unless version is specified, in which case only those versions are deleted.

This may not actually delete the underlying data if a snapshot still references the version. See snapshot for more detail.

Note that this may require data to be removed from the underlying storage which can be slow.

This method does not remove any staged data, use delete_staged_data for that.

If no symbol called symbol exists then this is a no-op. In particular this method does not raise in this case.

PARAMETER DESCRIPTION
symbol

Symbol to delete.

TYPE: str

versions

Version or versions of symbol to delete. If None then all versions will be deleted.

TYPE: Optional[Union[int, Iterable[int]]] DEFAULT: None

delete_data_in_range

delete_data_in_range(
    symbol: str,
    date_range: Tuple[
        Optional[Timestamp], Optional[Timestamp]
    ],
    prune_previous_versions: bool = False,
)

Delete data within the given date range, creating a new version of symbol.

The existing symbol version must be timeseries-indexed.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

date_range

The date range in which to delete data. Leaving any part of the tuple as None leaves that part of the range open ended.

TYPE: Tuple[Optional[Timestamp], Optional[Timestamp]]

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

Examples:

>>> df = pd.DataFrame({"column": [5, 6, 7, 8]}, index=pd.date_range(start="1/1/2018", end="1/4/2018"))
>>> lib.write("symbol", df)
>>> lib.delete_data_in_range("symbol", date_range=(datetime.datetime(2018, 1, 1), datetime.datetime(2018, 1, 2)))
>>> lib["symbol"].version
1
>>> lib["symbol"].data
                column
2018-01-03       7
2018-01-04       8

delete_snapshot

delete_snapshot(snapshot_name: str) -> None

Delete a named snapshot. This may take time if the given snapshot is the last reference to the underlying symbol(s) as the underlying data will be removed as well.

PARAMETER DESCRIPTION
snapshot_name

The snapshot name to delete.

TYPE: str

RAISES DESCRIPTION
Exception

If the named snapshot does not exist.

delete_staged_data

delete_staged_data(symbol: str)

Removes staged data.

PARAMETER DESCRIPTION
symbol

Symbol to remove staged data for.

TYPE: `str`

See Also

write Documentation on the staged parameter explains the concept of staged data in more detail.

enterprise_options

enterprise_options() -> EnterpriseLibraryOptions

Enterprise library options set on this library. See also options for non-enterprise options.

finalize_staged_data

finalize_staged_data(
    symbol: str,
    mode: Optional[
        StagedDataFinalizeMethod
    ] = StagedDataFinalizeMethod.WRITE,
    prune_previous_versions: bool = False,
    metadata: Any = None,
    validate_index=True,
)

Finalizes staged data, making it available for reads.

PARAMETER DESCRIPTION
symbol

Symbol to finalize data for.

TYPE: `str`

mode

Finalize mode. Valid options are WRITE or APPEND. Write collects the staged data and writes them to a new version. Append collects the staged data and appends them to the latest version.

TYPE: `StagedDataFinalizeMethod` DEFAULT: StagedDataFinalizeMethod.WRITE

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

metadata

Optional metadata to persist along with the symbol.

TYPE: Any DEFAULT: None

validate_index

If True, and staged segments are timeseries, will verify that the index of the symbol after this operation supports date range searches and update operations. This requires that the indexes of the staged segments are non-overlapping with each other, and, in the case of StagedDataFinalizeMethod.APPEND, fall after the last index value in the previous version.

DEFAULT: True

See Also

write Documentation on the staged parameter explains the concept of staged data in more detail.

get_description

get_description(
    symbol: str, as_of: Optional[AsOf] = None
) -> SymbolDescription

Returns descriptive data for symbol.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

as_of

See documentation on read.

TYPE: AsOf DEFAULT: None

RETURNS DESCRIPTION
SymbolDescription

Named tuple containing the descriptive data.

See Also

SymbolDescription For documentation on each field.

get_description_batch

get_description_batch(
    symbols: List[Union[str, ReadInfoRequest]]
) -> List[Union[SymbolDescription, DataError]]

Returns descriptive data for a list of symbols.

PARAMETER DESCRIPTION
symbols

List of symbols to read.

TYPE: List[Union[str, ReadInfoRequest]]

RETURNS DESCRIPTION
List[Union[SymbolDescription, DataError]]

A list of the descriptive data, whose i-th element corresponds to the i-th element of the symbols parameter. If the specified version does not exist, a DataError object is returned, with symbol, version_request_type, version_request_data properties, error_code, error_category, and exception_string properties. If a key error or any other internal exception occurs, the same DataError object is also returned.

See Also

SymbolDescription For documentation on each field.

get_staged_symbols

get_staged_symbols() -> List[str]

Returns all symbols with staged, unfinalized data.

RETURNS DESCRIPTION
List[str]

Symbol names.

See Also

write Documentation on the staged parameter explains the concept of staged data in more detail.

has_symbol

has_symbol(
    symbol: str, as_of: Optional[AsOf] = None
) -> bool

Whether this library contains the given symbol.

PARAMETER DESCRIPTION
symbol

Symbol name for the item

TYPE: str

as_of

Return the data as it was as_of the point in time. See read for more documentation. If absent then considers symbols that are live in the library as of the current time.

TYPE: AsOf DEFAULT: None

RETURNS DESCRIPTION
bool

True if the symbol is in the library, False otherwise.

Examples:

>>> lib.write("symbol", pd.DataFrame())
>>> lib.has_symbol("symbol")
True
>>> lib.has_symbol("another_symbol")
False

The contains operator also checks whether a symbol exists in this library as of now:

>>> "symbol" in lib
True
>>> "another_symbol" in lib
False

head

head(
    symbol: str,
    n: int = 5,
    as_of: Optional[AsOf] = None,
    columns: List[str] = None,
) -> VersionedItem

Read the first n rows of data for the named symbol. If n is negative, return all rows except the last n rows.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

n

Number of rows to select if non-negative, otherwise number of rows to exclude.

TYPE: int DEFAULT: 5

as_of

See documentation on read.

TYPE: AsOf DEFAULT: None

columns

See documentation on read.

TYPE: List[str] DEFAULT: None

RETURNS DESCRIPTION
VersionedItem object that contains a .data and .metadata element.

is_symbol_fragmented

is_symbol_fragmented(
    symbol: str, segment_size: Optional[int] = None
) -> bool

Check whether the number of segments that would be reduced by compaction is more than or equal to the value specified by the configuration option "SymbolDataCompact.SegmentCount" (defaults to 100).

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

segment_size

Target for maximum no. of rows per segment, after compaction. If parameter is not provided, library option for segments's maximum row size will be used

TYPE: Optional[int] DEFAULT: None

Notes

Config map setting - SymbolDataCompact.SegmentCount will be replaced by a library setting in the future. This API will allow overriding the setting as well.

RETURNS DESCRIPTION
bool

list_snapshots

list_snapshots(
    load_metadata: Optional[bool] = True,
) -> Union[List[str], Dict[str, Any]]

List the snapshots in the library.

PARAMETER DESCRIPTION
load_metadata

Load the snapshot metadata. May be slow so opt for false if you don't need it.

TYPE: `Optional[bool]` DEFAULT: True

RETURNS DESCRIPTION
Union[List[str], Dict[str, Any]]

Snapshots in the library. Returns a list of snapshot names if load_metadata is False, otherwise returns a dictionary where keys are snapshot names and values are metadata associated with that snapshot.

list_symbols

list_symbols(
    snapshot_name: Optional[str] = None,
    regex: Optional[str] = None,
) -> List[str]

Return the symbols in this library.

PARAMETER DESCRIPTION
regex

If passed, returns only the symbols which match the regex.

TYPE: Optional[str] DEFAULT: None

snapshot_name

Return the symbols available under the snapshot. If None then considers symbols that are live in the library as of the current time.

TYPE: Optional[str] DEFAULT: None

RETURNS DESCRIPTION
List[str]

Symbols in the library.

list_versions

list_versions(
    symbol: Optional[str] = None,
    snapshot: Optional[str] = None,
    latest_only: bool = False,
    skip_snapshots: bool = False,
) -> Dict[SymbolVersion, VersionInfo]

Get the versions in this library, filtered by the passed in parameters.

PARAMETER DESCRIPTION
symbol

Symbol to return versions for. If None returns versions across all symbols in the library.

TYPE: Optional[str] DEFAULT: None

snapshot

Only return the versions contained in the named snapshot.

TYPE: Optional[str] DEFAULT: None

latest_only

Only include the latest version for each returned symbol.

TYPE: bool DEFAULT: False

skip_snapshots

Don't populate version list with snapshot information. Can improve performance significantly if there are many snapshots.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
Dict[SymbolVersion, VersionInfo]

Dictionary describing the version for each symbol-version pair in the library. Since symbol version is a (named) tuple you can index in to the dictionary simply as shown in the examples below.

Examples:

>>> df = pd.DataFrame()
>>> lib.write("symbol", df, metadata=10)
>>> lib.write("symbol", df, metadata=11, prune_previous_versions=False)
>>> lib.snapshot("snapshot")
>>> lib.write("symbol", df, metadata=12, prune_previous_versions=False)
>>> lib.delete("symbol", versions=(1, 2))
>>> versions = lib.list_versions("symbol")
>>> versions["symbol", 1].deleted
True
>>> versions["symbol", 1].snapshots
["my_snap"]

options

options() -> LibraryOptions

Library options set on this library. See also enterprise_options.

prune_previous_versions

prune_previous_versions(symbol)

Removes all (non-snapshotted) versions from the database for the given symbol, except the latest.

PARAMETER DESCRIPTION
symbol

Symbol name to prune.

TYPE: `str`

read

read(
    symbol: str,
    as_of: Optional[AsOf] = None,
    date_range: Optional[
        Tuple[Optional[Timestamp], Optional[Timestamp]]
    ] = None,
    row_range: Optional[Tuple[int, int]] = None,
    columns: Optional[List[str]] = None,
    query_builder: Optional[QueryBuilder] = None,
) -> VersionedItem

Read data for the named symbol. Returns a VersionedItem object with a data and metadata element (as passed into write).

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

as_of

Return the data as it was as of the point in time. None means that the latest version should be read. The various types of this parameter mean: - int: specific version number. Negative indexing is supported, with -1 representing the latest version, -2 the version before that, etc. - str: snapshot name which contains the version - datetime.datetime : the version of the data that existed as_of the requested point in time

TYPE: AsOf DEFAULT: None

date_range

DateRange to restrict read data to.

Applicable only for time-indexed Pandas dataframes or series. Returns only the part of the data that falls withing the given range (inclusive). None on either end leaves that part of the range open-ended. Hence specifying (None, datetime(2025, 1, 1) declares that you wish to read all data up to and including 20250101. The same effect can be achieved by using the date_range clause of the QueryBuilder class, which will be slower, but return data with a smaller memory footprint. See the QueryBuilder.date_range docstring for more details.

Only one of date_range or row_range can be provided.

TYPE: Optional[Tuple[Optional[Timestamp], Optional[Timestamp]]] DEFAULT: None

row_range

Row range to read data for. Inclusive of the lower bound, exclusive of the upper bound lib.read(symbol, row_range=(start, end)).data should behave the same as df.iloc[start:end], including in the handling of negative start/end values.

Only one of date_range or row_range can be provided.

TYPE: Optional[Tuple[int, int]] DEFAULT: None

columns

Applicable only for Pandas data. Determines which columns to return data for. Special values: - None: Return a dataframe containing all columns - []: Return a dataframe containing only the index columns

TYPE: Optional[List[str]] DEFAULT: None

query_builder

A QueryBuilder object to apply to the dataframe before it is returned. For more information see the documentation for the QueryBuilder class (from arcticdb import QueryBuilder; help(QueryBuilder)).

TYPE: Optional[QueryBuilder] DEFAULT: None

RETURNS DESCRIPTION
VersionedItem object that contains a .data and .metadata element

Examples:

>>> df = pd.DataFrame({'column': [5,6,7]})
>>> lib.write("symbol", df, metadata={'my_dictionary': 'is_great'})
>>> lib.read("symbol").data
   column
0       5
1       6
2       7

The default read behaviour is also available through subscripting:

>>> lib["symbol"].data
   column
0       5
1       6
2       7

read_batch

read_batch(
    symbols: List[Union[str, ReadRequest]],
    query_builder: Optional[QueryBuilder] = None,
) -> List[Union[VersionedItem, DataError]]

Reads multiple symbols.

PARAMETER DESCRIPTION
symbols

List of symbols to read.

TYPE: List[Union[str, ReadRequest]]

query_builder

A single QueryBuilder to apply to all the dataframes before they are returned. If this argument is passed then none of the symbols may have their own query_builder specified in their request.

TYPE: Optional[QueryBuilder] DEFAULT: None

RETURNS DESCRIPTION
List[Union[VersionedItem, DataError]]

A list of the read results, whose i-th element corresponds to the i-th element of the symbols parameter. If the specified version does not exist, a DataError object is returned, with symbol, version_request_type, version_request_data properties, error_code, error_category, and exception_string properties. If a key error or any other internal exception occurs, the same DataError object is also returned.

RAISES DESCRIPTION
ArcticInvalidApiUsageException

If kwarg query_builder and per-symbol query builders both used.

Examples:

>>> lib.write("s1", pd.DataFrame())
>>> lib.write("s2", pd.DataFrame({"col": [1, 2, 3]}))
>>> lib.write("s2", pd.DataFrame(), prune_previous_versions=False)
>>> lib.write("s3", pd.DataFrame())
>>> batch = lib.read_batch(["s1", adb.ReadRequest("s2", as_of=0), "s3", adb.ReadRequest("s2", as_of=1000)])
>>> batch[0].data.empty
True
>>> batch[1].data.empty
False
>>> batch[2].data.empty
True
>>> batch[3].symbol
"s2"
>>> isinstance(batch[3], adb.DataError)
True
>>> batch[3].version_request_type
VersionRequestType.SPECIFIC
>>> batch[3].version_request_data
1000
>>> batch[3].error_code
ErrorCode.E_NO_SUCH_VERSION
>>> batch[3].error_category
ErrorCategory.MISSING_DATA
See Also

read

read_metadata

read_metadata(
    symbol: str, as_of: Optional[AsOf] = None
) -> VersionedItem

Return the metadata saved for a symbol. This method is faster than read as it only loads the metadata, not the data itself.

PARAMETER DESCRIPTION
symbol

Symbol name

TYPE: str

as_of

Return the metadata as it was as of the point in time. See documentation on read for documentation on the different forms this parameter can take.

TYPE: AsOf DEFAULT: None

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the affected symbol in the store. The data attribute will be None.

read_metadata_batch

read_metadata_batch(
    symbols: List[Union[str, ReadInfoRequest]]
) -> List[Union[VersionedItem, DataError]]

Reads the metadata of multiple symbols.

PARAMETER DESCRIPTION
symbols

List of symbols to read metadata.

TYPE: List[Union[str, ReadInfoRequest]]

RETURNS DESCRIPTION
List[Union[VersionedItem, DataError]]

A list of the read metadata results, whose i-th element corresponds to the i-th element of the symbols parameter. A VersionedItem object with the metadata field set as None will be returned if the requested version of the symbol exists but there is no metadata If the specified version does not exist, a DataError object is returned, with symbol, version_request_type, version_request_data properties, error_code, error_category, and exception_string properties. If a key error or any other internal exception occurs, the same DataError object is also returned.

See Also

read_metadata

reload_symbol_list

reload_symbol_list()

Forces the symbol list cache to be reloaded.

This can take a long time on large libraries or certain S3 implementations, and once started, it cannot be safely interrupted. If the call is interrupted somehow (exception/process killed), please call this again ASAP.

snapshot

snapshot(
    snapshot_name: str,
    metadata: Any = None,
    skip_symbols: Optional[List[str]] = None,
    versions: Optional[Dict[str, int]] = None,
) -> None

Creates a named snapshot of the data within a library.

By default, the latest version of every symbol that has not been deleted will be contained within the snapshot. You can change this behaviour with either versions (an allow-list) or with skip_symbols (a deny-list). Concurrent writes with prune previous versions set while the snapshot is being taken can potentially lead to corruption of the affected symbols in the snapshot.

The symbols and versions contained within the snapshot will persist regardless of new symbols and versions being written to the library afterwards. If a version or symbol referenced in a snapshot is deleted then the underlying data will be preserved to ensure the snapshot is still accessible. Only once all referencing snapshots have been removed will the underlying data be removed as well.

At most one of skip_symbols and versions may be truthy.

PARAMETER DESCRIPTION
snapshot_name

Name of the snapshot.

TYPE: str

metadata

Optional metadata to persist along with the snapshot.

TYPE: Any DEFAULT: None

skip_symbols

Optional symbols to be excluded from the snapshot.

TYPE: List[str] DEFAULT: None

versions

Optional dictionary of versions of symbols to snapshot. For example versions={"a": 2, "b": 3} will snapshot version 2 of symbol "a" and version 3 of symbol "b".

TYPE: Optional[Dict[str, int]] DEFAULT: None

RAISES DESCRIPTION
InternalException

If a snapshot already exists with snapshot_name. You must explicitly delete the pre-existing snapshot.

MissingDataException

If a symbol or the version of symbol specified in versions does not exist or has been deleted in the library, or, the library has no symbol.

sort_and_finalize_staged_data

sort_and_finalize_staged_data(
    symbol: str,
    mode: Optional[
        StagedDataFinalizeMethod
    ] = StagedDataFinalizeMethod.WRITE,
    prune_previous_versions: bool = False,
)

sort_merge will sort and finalize staged data. This differs from finalize_staged_data in that it can support staged segments with interleaved time periods - the end result will be ordered. This requires performing a full sort in memory so can be time consuming.

PARAMETER DESCRIPTION
symbol

Symbol to finalize data for.

TYPE: `str`

mode

Finalize mode. Valid options are WRITE or APPEND. Write collects the staged data and writes them to a new timeseries. Append collects the staged data and appends them to the latest version.

TYPE: `StagedDataFinalizeMethod` DEFAULT: StagedDataFinalizeMethod.WRITE

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

See Also

write Documentation on the staged parameter explains the concept of staged data in more detail.

tail

tail(
    symbol: str,
    n: int = 5,
    as_of: Optional[Union[int, str]] = None,
    columns: List[str] = None,
) -> VersionedItem

Read the last n rows of data for the named symbol. If n is negative, return all rows except the first n rows.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

n

Number of rows to select if non-negative, otherwise number of rows to exclude.

TYPE: int DEFAULT: 5

as_of

See documentation on read.

TYPE: AsOf DEFAULT: None

columns

See documentation on read.

TYPE: List[str] DEFAULT: None

RETURNS DESCRIPTION
VersionedItem object that contains a .data and .metadata element.

update

update(
    symbol: str,
    data: Union[DataFrame, Series],
    metadata: Any = None,
    upsert: bool = False,
    date_range: Optional[
        Tuple[Optional[Timestamp], Optional[Timestamp]]
    ] = None,
    prune_previous_versions: bool = False,
) -> VersionedItem

Overwrites existing symbol data with the contents of data. The entire range between the first and last index entry in data is replaced in its entirety with the contents of data, adding additional index entries if required. update only operates over the outermost index level - this means secondary index rows will be removed if not contained in data.

Both the existing symbol version and data must be timeseries-indexed.

In the case where data has zero rows, nothing will be done and no new version will be created. This means that update cannot be used with date_range to just delete a subset of the data. We have delete_data_in_range for exactly this purpose and to make it very clear when deletion is intended.

Note that update is not designed for multiple concurrent writers over a single symbol.

PARAMETER DESCRIPTION
symbol

Symbol name.

TYPE: str

data

Timeseries indexed data to use for the update.

TYPE: Union[DataFrame, Series]

metadata

Metadata to persist along with the new symbol version.

TYPE: Any DEFAULT: None

upsert

If True, will write the data even if the symbol does not exist.

TYPE: bool DEFAULT: False

date_range

If a range is specified, it will delete the stored value within the range and overwrite it with the data in data. This allows the user to update with data that might only be a subset of the stored value. Leaving any part of the tuple as None leaves that part of the range open ended. Only data with date_range will be modified, even if data covers a wider date range.

TYPE: Optional[Tuple[Optional[Timestamp], Optional[Timestamp]]] DEFAULT: None

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the written symbol in the store.

Examples:

>>> df = pd.DataFrame(
...    {'column': [1,2,3,4]},
...    index=pd.date_range(start='1/1/2018', end='1/4/2018')
... )
>>> df
            column
2018-01-01       1
2018-01-02       2
2018-01-03       3
2018-01-04       4
>>> lib.write("symbol", df)
>>> update_df = pd.DataFrame(
...    {'column': [400, 40]},
...    index=pd.date_range(start='1/1/2018', end='1/3/2018', freq='2D')
... )
>>> update_df
            column
2018-01-01     400
2018-01-03      40
>>> lib.update("symbol", update_df)
>>> # Note that 2018-01-02 is gone despite not being in update_df
>>> lib.read("symbol").data
            column
2018-01-01     400
2018-01-03      40
2018-01-04       4

write

write(
    symbol: str,
    data: NormalizableType,
    metadata: Any = None,
    prune_previous_versions: bool = False,
    staged=False,
    validate_index=True,
) -> VersionedItem

Write data to the specified symbol. If symbol already exists then a new version will be created to reference the newly written data. For more information on versions see the documentation for the read primitive.

data must be of a format that can be normalised into Arctic's internal storage structure. Pandas DataFrames, Pandas Series and Numpy NDArrays can all be normalised. Normalised data will be split along both the columns and rows into segments. By default, a segment will contain 100,000 rows and 127 columns.

If this library has write_deduplication enabled then segments will be deduplicated against storage prior to write to reduce required IO operations and storage requirements. Data will be effectively deduplicated for all segments up until the first differing row when compared to storage. As a result, modifying the beginning of data with respect to previously written versions may significantly reduce the effectiveness of deduplication.

Note that write is not designed for multiple concurrent writers over a single symbol unless the staged keyword argument is set to True. If staged is True, written segments will be staged and left in an "incomplete" stage, unable to be read until they are finalized. This enables multiple writers to a single symbol - all writing staged data at the same time - with one process able to later finalize all staged data rendering the data readable by clients. To finalize staged data, see finalize_staged_data.

Note: ArcticDB will use the 0-th level index of the Pandas DataFrame for its on-disk index.

Any non-DatetimeIndex will converted into an internal RowCount index. That is, ArcticDB will assign each row a monotonically increasing integer identifier and that will be used for the index.

PARAMETER DESCRIPTION
symbol

Symbol name. Limited to 255 characters. The following characters are not supported in symbols: "*", "&", "<", ">"

TYPE: str

data

Data to be written. To write non-normalizable data, use write_pickle.

TYPE: NormalizableType

metadata

Optional metadata to persist along with the symbol.

TYPE: Any DEFAULT: None

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

staged

Whether to write to a staging area rather than immediately to the library. See documentation on finalize_staged_data for more information.

TYPE: bool DEFAULT: False

validate_index

If True, verify that the index of data supports date range searches and update operations. This tests that the data is sorted in ascending order, using Pandas DataFrame.index.is_monotonic_increasing.

DEFAULT: True

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the written symbol in the store.

RAISES DESCRIPTION
ArcticUnsupportedDataTypeException

If data is not of NormalizableType.

UnsortedDataException

If data is unsorted and validate_index is set to True.

Examples:

>>> df = pd.DataFrame({'column': [5,6,7]})
>>> lib.write("symbol", df, metadata={'my_dictionary': 'is_great'})
>>> lib.read("symbol").data
   column
0       5
1       6
2       7

Staging data for later finalisation (enables concurrent writes):

>>> df = pd.DataFrame({'column': [5,6,7]}, index=pd.date_range(start='1/1/2000', periods=3))
>>> lib.write("staged", df, staged=True)  # Multiple staged writes can occur in parallel
>>> lib.finalize_staged_data("staged", StagedDataFinalizeMethod.WRITE)  # Must be run after all staged writes have completed
>>> lib.read("staged").data  # Would return error if run before finalization
            column
2000-01-01       5
2000-01-02       6
2000-01-03       7

WritePayload objects can be unpacked and used as parameters:

>>> w = adb.WritePayload("symbol", df, metadata={'the': 'metadata'})
>>> lib.write(*w, staged=True)

write_batch

write_batch(
    payloads: List[WritePayload],
    prune_previous_versions: bool = False,
    validate_index=True,
) -> List[Union[VersionedItem, DataError]]

Write a batch of multiple symbols.

PARAMETER DESCRIPTION
payloads

Symbols and their corresponding data. There must not be any duplicate symbols in payload.

TYPE: `List[WritePayload]`

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

validate_index

Verify that each entry in the batch has an index that supports date range searches and update operations. This tests that the data is sorted in ascending order, using Pandas DataFrame.index.is_monotonic_increasing.

DEFAULT: True

RETURNS DESCRIPTION
List[Union[VersionedItem, DataError]]

List of versioned items. The data attribute will be None for each versioned item. i-th entry corresponds to i-th element of payloads. Each result correspond to a structure containing metadata and version number of the written symbols in the store, in the same order as payload. If a key error or any other internal exception is raised, a DataError object is returned, with symbol, error_code, error_category, and exception_string properties.

RAISES DESCRIPTION
ArcticDuplicateSymbolsInBatchException

When duplicate symbols appear in payload.

ArcticUnsupportedDataTypeException

If data that is not of NormalizableType appears in any of the payloads.

See Also

write: For more detailed documentation.

Examples:

Writing a simple batch:

>>> df_1 = pd.DataFrame({'column': [1,2,3]})
>>> df_2 = pd.DataFrame({'column': [4,5,6]})
>>> payload_1 = adb.WritePayload("symbol_1", df_1, metadata={'the': 'metadata'})
>>> payload_2 = adb.WritePayload("symbol_2", df_2)
>>> items = lib.write_batch([payload_1, payload_2])
>>> lib.read("symbol_1").data
   column
0       1
1       2
2       3
>>> lib.read("symbol_2").data
   column
0       4
1       5
2       6
>>> items[0].symbol, items[1].symbol
('symbol_1', 'symbol_2')

write_metadata

write_metadata(
    symbol: str,
    metadata: Any,
    prune_previous_versions: bool = False,
) -> VersionedItem

Write metadata under the specified symbol name to this library. The data will remain unchanged. A new version will be created.

If the symbol is missing, it causes a write with empty data (None, pickled, can't append) and the supplied metadata.

This method should be faster than write as it involves no data segment read/write operations.

PARAMETER DESCRIPTION
symbol

Symbol name for the item

TYPE: str

metadata

Metadata to persist along with the symbol

TYPE: Any

prune_previous_versions

Removes previous (non-snapshotted) versions from the database. Note that metadata is versioned alongside the data it is referring to, and so this operation removes old versions of data as well as metadata.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
VersionedItem

Structure containing metadata and version number of the affected symbol in the store.

write_metadata_batch

write_metadata_batch(
    write_metadata_payloads: List[WriteMetadataPayload],
    prune_previous_versions: bool = False,
) -> List[Union[VersionedItem, DataError]]

Write metadata to multiple symbols in a batch fashion. This is more efficient than making multiple write_metadata calls in succession as some constant-time operations can be executed only once rather than once for each element of write_metadata_payloads. Note that this isn't an atomic operation - it's possible for the metadata for one symbol to be fully written and readable before another symbol.

PARAMETER DESCRIPTION
write_metadata_payloads

Symbols and their corresponding metadata. There must not be any duplicate symbols in payload.

TYPE: `List[WriteMetadataPayload]`

prune_previous_versions

Removes previous (non-snapshotted) versions from the database. Note that metadata is versioned alongside the data it is referring to, and so this operation removes old versions of data as well as metadata.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
List[Union[VersionedItem, DataError]]

List of versioned items. The data attribute will be None for each versioned item. i-th entry corresponds to i-th element of write_metadata_payloads. Each result correspond to a structure containing metadata and version number of the affected symbol in the store. If any internal exception is raised, a DataError object is returned, with symbol, error_code, error_category, and exception_string properties.

RAISES DESCRIPTION
ArcticDuplicateSymbolsInBatchException

When duplicate symbols appear in write_metadata_payloads.

Examples:

Writing a simple batch:

>>> payload_1 = adb.WriteMetadataPayload("symbol_1", {'the': 'metadata_1'})
>>> payload_2 = adb.WriteMetadataPayload("symbol_2", {'the': 'metadata_2'})
>>> items = lib.write_metadata_batch([payload_1, payload_2])
>>> lib.read_metadata("symbol_1")
{'the': 'metadata_1'}
>>> lib.read_metadata("symbol_2")
{'the': 'metadata_2'}

write_pickle

write_pickle(
    symbol: str,
    data: Any,
    metadata: Any = None,
    prune_previous_versions: bool = False,
    staged=False,
) -> VersionedItem

See write. This method differs from write only in that data can be of any type that is serialisable via the Pickle library. There are significant downsides to storing data in this way:

  • Retrieval can only be done in bulk. Calls to read will not support date_range, query_builder or columns.
  • The data cannot be updated or appended to via the update and append methods.
  • Writes cannot be deduplicated in any way.
PARAMETER DESCRIPTION
symbol

See documentation on write.

TYPE: str

data

Data to be written.

TYPE: `Any`

metadata

See documentation on write.

TYPE: Any DEFAULT: None

prune_previous_versions

See documentation on write.

TYPE: bool DEFAULT: False

staged

See documentation on write.

DEFAULT: False

RETURNS DESCRIPTION
VersionedItem

See documentation on write.

Examples:

>>> lib.write_pickle("symbol", [1,2,3])
>>> lib.read("symbol").data
[1, 2, 3]
See Also

write: For more detailed documentation.

write_pickle_batch

write_pickle_batch(
    payloads: List[WritePayload],
    prune_previous_versions: bool = False,
    staged=False,
) -> List[VersionedItem]

Write a batch of multiple symbols, pickling their data if necessary.

PARAMETER DESCRIPTION
payloads

Symbols and their corresponding data. There must not be any duplicate symbols in payload.

TYPE: `List[WritePayload]`

prune_previous_versions

Removes previous (non-snapshotted) versions from the database.

TYPE: bool DEFAULT: False

staged

See documentation on write.

DEFAULT: False

RETURNS DESCRIPTION
List[VersionedItem]

Structures containing metadata and version number of the written symbols in the store, in the same order as payload.

RAISES DESCRIPTION
ArcticDuplicateSymbolsInBatchException

When duplicate symbols appear in payload.

See Also

write: For more detailed documentation. write_pickle: For information on the implications of providing data that needs to be pickled.