Skip to content

Release Notes

Release notes for every stable ArcticDB release, from GitHub Releases.

v6.28.0

Released 2026-10-14 · GitHub release

There are no user facing changes in this version

v6.27.1

Released 2026-10-06 · GitHub release

🚀 Features

  • Revert defaults for new S3 configurations (#3443)

🐛 Fixes

  • Add deprecation warnings for the staged parameter of write and write_pickle methods of the Library class (#3420)

v6.26.0

Released 2026-09-14 · GitHub release

🐛 Fixes

  • Performance Improvement on VAST storage: Set CURLOPT_DNS_SHUFFLE_ADDRESSES in our S3 client (#3345)

v6.25.0

Released 2026-09-09 · GitHub release

🚀 Features

  • Read and write support for arrow-backed string dtype (#3242)
  • Make prune_previous a library option (#3340)
  • Enhancement 18053144964: fix get description output with arrow written data (#3337)
  • Enhancement 11864701485: arrow string formatting norm metadata (#3333)

🐛 Fixes

  • Bound processing pipeline memory use (#3191)

v6.24.0

Released 2026-09-02 · GitHub release

🐛 Fixes

  • Log a warning in a process using arcticdb that forks, from Python 3.12+ on Linux (#3295)

v6.23.0

Released 2026-08-17 · GitHub release

🚀 Features

  • Add compact_data_batch method (#3291)
    This is multi-symbol equivalent of compact_data for defragmenting data chunks on disk.

🐛 Fixes

  • Add MD5 checksum variant of DeleteObjectsRequest for S3 storage (#3259)

v6.22.0

Released 2026-08-10 · GitHub release

🚀 Features

  • Support compact_data arg in append_batch (#3270)

This allows you to "compact", i.e. defragment, the files containing a symbol's data during append_batch, which can improve subsequent read performance. See the docstring on append_batch for more information.

  • Add progress logging to finalize_staged_data's do_compact loop (#3250)

v6.21.0

Released 2026-08-04 · GitHub release

🐛 Fixes

  • Bug fix for query builder filtering with ISIN filters over sets with duplicate str representations (#3201)
  • Fix macOS 15 ImportError: Symbol not found: __hash_memory on mac back-deployed wheels (#3243)
  • Correctly filter against NaT as a timestamp rather than an int64 with the query builder, so q["col"] > pd.NaT will now return nothing (like Pandas). This is because we are now tracking that pd.NaT is a time type, whereas before we treated it as int64. (#3266)

🏎️ Performance

  • Merge filter clauses in to one expression during query builder evaluation (#3204).

This improves performance and memory use for query builders with long chains of filters like:

        q = QueryBuilder()
        q = q[q["v3"] > 0.0]  # keeps ~all rows
        q = q[q["v3"] < 99.0]  # keeps ~99%
        q = q[q["v2"] > 1]  # keeps ~93%
        q = q[q["v2"] < 15]  # keeps ~93%
        q = q[q["v1"] > 1]  # keeps ~80%
        q = q[q["v1"] < 5]  # keeps ~80%
        q = q[q["v3"] < 50.0]  # narrows to ~50%

v6.20.0

Released 2026-07-29 · GitHub release

🚀 Features

  • Enhancement 9929831600: arrow timezone support (#3225)
  • Enhancement 12036593840: append inline defrag post refactors (#3208)

🐛 Fixes

  • Add NaN and NULL count to MIN/MAX stats (#3139)
  • Performance 12498089633: Make more index key reading async (#3237)

v6.19.0

Released 2026-07-21 · GitHub release

🚀 Features

  • Upgrade LMDB 0.9.22 → 0.9.35 and patch for CVE-2019-16224..16228 (#3163)

🐛 Fixes

  • Persist metadata when appending or updating empty frames, and skip writing unnecessary symbol list keys (#3214)
    • Note that this is a behaviour change. Previously, when an empty dataframe was provided to append or update (or batch variations thereof), a warning would be logged and no new version was created. In particular, this meant that the provided metadata was not persisted
  • Read Index Key Only Once When Creating Column Stats (#3222)
  • Reduce symbol list peak memory usage (#3165)

v6.18.7

Released 2026-07-13 · GitHub release

🐛 Fixes

v6.18.6

Released 2026-07-13 · GitHub release

🐛 Fixes

v6.18.5

Released 2026-07-06 · GitHub release

🚀 Features

  • Propagate SortedValue to Polars set_sorted() on index column (#3049)

🐛 Fixes

  • Bugfix 12337436632: do not orphan data keys in append/update when schemas mismatch (#3187)
  • Allow compacting numpy arrays (#3189)
  • Add aws_profile support when aws_auth is set default/true in connection string (#3169)
  • Make index key reading async in append (#3188)
  • Fix copying index keys to preserve their start/end index (#3192)

v6.18.4

Released 2026-06-29 · GitHub release

🏎 Performance

  • Deletion is now faster and uses less memory for libraries with many snapshots (#3181)

v6.18.3

Released 2026-06-22 · GitHub release

🐛 Fixes

  • Banning backslash in azure storage (#3160)

v6.18.2

Released 2026-06-17 · GitHub release

🏎️ Performance

  • Use atom key packed in version map cache to decrease cache memory footpring (#3116)
  • Resampling performance improvement and sparse aggregation columns support (#3062)
  • Submit delete batches in parallel (#3123)

🐛 Fixes

  • Route AWS SDK S3 logs to stderr by default (#3149)

v6.18.1

Released 2026-06-08 · GitHub release

🐛 Fixes

  • [Bugfix] Fix wrong column naming after append and update when the DataFrame has non-string column name (#3090)

v6.18.0

Released 2026-06-01 · GitHub release

🚀 Features

  • Fully support compact_data and deprecate old methods (#3119)
  • Implement POW operator in the QueryBuilder(#3045)

🐛 Fixes

  • add_to_snapshot does not work with negative version (#3056)
  • Be consistent with NaN when filtering on NaT values (#3085)
  • Reject adding deleted versions to snapshot (#3113)

🏎 Performance

  • Processing pipeline performance improvements (#3086)
  • Defrag profiling and optimisation (#3080)

⚠ Technical API Break ⚠

#3085 Changes the way query builder handles NaT in filtering. From now on it will use NaN-like semantics - NaT OP ANYTHING is false for any OP other than !=, for which NaT != ANYTHING is true. For code samples and detailed info see  #3085's PR description.

v6.17.0

Released 2026-05-18 · GitHub release

🚀 Features

  • Support sparse data in compact_data_experimental (#3059)
  • Adds method to check what the effect of calling compact_data_experimental would be (#3063)

🐛 Fixes

  • Add Arrow NaT handling (#3032)
  • Add warning message to various v1 read and write APIs (#3089)
  • Processing pipeline perf improvements (#3086)
  • Change add_to_snapshot so it throws when called with negative versions (#3056)

v6.16.0

Released 2026-05-11 · GitHub release

🚀 Features

  • Improve version map cache validation for as_of queries by version and timestamp (#2894)

🐛 Fixes

  • Improve batch reads performance on Windows (#3048)
  • Fix small performance regression reading None / NaN columns after Folly upgrade (#3055)
  • Upgrade sparrow to 2.4.0 and use memcpy for bool truncation (#3052)
  • Reduce malloc trim frequency to improve batch read performance (#3060)

v6.15.0

Released 2026-05-04 · GitHub release

🚀 Features

  • Enhancement 11460943163: defrag dynamic schema (#3029)

v6.14.0

Released 2026-04-29 · GitHub release

🚀 Features

  • Validate kwargs provided to NativeVersionStore APIs and raise if invalid kwargs are provided (#3035). Permit allow_secondary. The exception can be flipped to a warning log by setting the env var ARCTICDB_DISABLE_KWARG_VALIDATION=1.

🐛 Fixes

  • ⚠ Minor API change - Use UnsortedDataException not SortingException in update (#3043). Note that UnsortedDataException is a sub-type of SortingException so we don't expect breaks from this. The only case affected would be if you have code that wanted to handle all sorting exceptions other than those from update, which seems very improbable. We now consistently use UnsortedDataException throughout our APIs.
  • Document our symbol name validation rules in faq.md (#3022)

v6.13.0

Released 2026-04-21 · GitHub release

🏎️ Performance

  • Improve performance of delete_batch (#2999)

🐛 Fixes

  • Fix seg fault in lib.read due to race condition in exception handling after network error (#3004)

v6.12.1

Released 2026-04-15 · GitHub release

🐛 Fixes

v6.11.1

Released 2026-03-30 · GitHub release

🐛 Fixes

  • Fixes for number of IO operations performed by index only reads (#2942)

v6.11.0

Released 2026-03-23 · GitHub release

🚀 Features

  • Validate kwargs passed to NativeVersionStore methods (#2936)

🐛 Fixes

  • Fix reading of sparse string columns as Arrow (#2932)
  • Match on multiple columns in merge updates (#2926)
  • Fix RangeIndex returned by index-only reads to match that returned by reads that return data (#2955)

v6.10.2

Released 2026-03-16 · GitHub release

🐛 Fixes

  • Include malloc.h to have malloc_trim's declaration (#2946)
  • Remove redundant move in initialization (#2885)
  • Improve logging for symbol list compaction (#2948)
  • Remove unused WriteOptions from WriteClause (#2944)

v6.10.1

Released 2026-03-12 · GitHub release

🐛 Fixes

  • Do not log a warning when reading a non-existent storage lock (#2937)

v6.10.0

Released 2026-03-02 · GitHub release

🚀 Features

  • Experimental lazy dataframe collect schema (#2906)

🏎️ Performance

  • Improve read(output_format="pyarrow") performance for bool columns (#2883)

🐛 Fixes

  • Fix import alongside matplotlib (#2918)

v6.9.2

Released 2026-02-23 · GitHub release

🚀 Features

- Implementation of experimental merge capability in ArcticDB (#2890)
⚠️ The API is under development. It is not subject to semver and can change even in minor or patch versions. The function is named merge_experimental, but will be renamed to merge when the API becomes stable.

The merge API offers a straightforward way for users to modify or insert specific rows in their data.

import arcticdb
import pandas as pd

ac = arcticdb.Arctic("lmdb://demo")
lib = ac.get_library("prices", create_if_missing=True)
sym = "AAPL"

prices = pd.DataFrame(
    {"price": [100.5, 102.3, 99.8, 101.0, 103.5]},
    index=pd.date_range("2023-01-01", periods=5),
)
lib.write(sym, prices)
print(lib.read(sym).data)

Original data:

price
2023-01-01 100.5
2023-01-02 102.3
2023-01-03 99.8
2023-01-04 101.0
2023-01-05 103.5

Dataframe with corrections:

corrections = pd.DataFrame(
    {"price": [999.0, 888.0]},
    index=pd.DatetimeIndex([pd.Timestamp("2023-01-02"), pd.Timestamp("2023-01-04")]),
)
print(corrections)
price
2023-01-02 999.0
2023-01-04 888.0

Apply the corrections to the original data:

lib.merge_experimental(sym, corrections)
print(lib.read(sym).data)
price
2023-01-01 100.5
2023-01-02 999.0
2023-01-03 99.8
2023-01-04 888.0
2023-01-05 103.5

Demo notebook with more example usage and current limitations

🐛 Fixes

  • Fix list_versions bug - include versions of symbols that have been deleted but are kept alive by a snapshot (#2864)
  • Fix read(output_format=Arrow) when a column contains only empty strings (#2871)
  • Claude documentation and skills for navigating and enhancing the codebase (#2902)
  • Fix s3 sts windows storage test (#2892)
  • Implement query stats for in-memory storage, use them to fix flaky ASV benchmark (#2899)
  • Arrow read fix truncation with dynamic schema (#2872)
  • Refactor Python bindings for improved cross-ABI compatibility with enterprise downstream. (#2884)

v6.9.0

Released 2026-02-02 · GitHub release

🔨 Maintenance

  • Remove python 3.8 build (#2845)
  • Pin to Pandas<3 for now (#2865)
  • Upgrade curl to >=8.18 (#2853)

🐛 Fixes

  • conda-build: Relax libprotobuf's specification (#2835)
  • docs: Update prebuilt binary availability (#2813)
  • Improve parallel writes error message to include symbol name (#2859)
  • Fix real azure tests issue (#2762)
  • Consolidate bindings for Enterprise ABI compatibility (#2821)
  • List versions with skip_snapshots and a snapshot specified should not read all snapshots in the library (#2863)

v6.8.0

Released 2026-01-28 · GitHub release

🚀 Features

  • Python 3.14 build for pypi (#2828)

🐛 Fixes

  • Bugfix/10722938876 fix listing operations on s3 express (#2841)
  • Pin to Pandas<3 (#2880)

v6.7.0

Released 2026-01-12 · GitHub release

🚀 Features

  • No pickle V2 meta structure normalization for recursive normalizer (#2778)
    Introduce new normalization (V2) for recursive normalizer metastructure as the existing one requires pickle.
    Please note that data with V2 normalized metastructure is not readable by <v6.7.0 releases.
    Plan of introduction:
Version Change
>= v6.7.0 Deprecation warning when writing V1 meta structure; V2 meta structure can be enabled optionally
>= v7.0.0 V2 meta structure will be enabled by default

Please consider enabling V2 as V1 will be deprecated in future v7.0.0 release. To enabld it, you need to set environment variable ARCTICDB_VersionStore_RecursiveNormalizerMetastructure_int to 2

🐛 Fixes

  • Testing improvments (#2804, #2801, #2808, #2807, #2815)
  • Properly specify LMDB's include directory (#2805)
  • conda-build: Add workflow for linux_aarch64 (#2806)
  • build: Include all headers uniformly (#2817)
  • Dropping support for SecureTransport on Mac using only OpenSSL (#2816)

v6.6.0

Released 2026-01-06 · GitHub release

🚀 Features

  • Log string pool decode timings in release builds (#2802)
  • Upgrade sparrow to 1.4.0 (#2789)
  • conda-build: Build for Windows (#2252)
  • Allow Fortran styled columns starting with None and NaN (#2797)
    For instance,
    np.array([[None, "string"], ["aaa", "bbb"], ["ccc", "ddd"]]) and np.array([[np.nan, "string"], ["aaa", "bbb"], ["ccc", "ddd"]]) can now be written to ArcticDB libraries

🐛 Fixes

  • Suppress mongo endSession warning (#2777)
  • Restrict custom CA certificate configurations for the Azure SDK to Linux only (#2282)

v6.5.0

Released 2025-12-12 · GitHub release

✨ Features

🚀 Performance

  • Decode only required columns in processing pipeline with dynamic schema (#2766)
  • Improve hash grouping aggregation parallelism (#2729)
  • Improve recursive normalized data read performance (#2787)

🐛 Fixes

  • Improve StageResult printout (#2760)
  • Allow colon characters in Arctic URI query parameters (#2783)
  • Fix ASV benchmarks LMDB map full error (#2786)

v6.3.2

Released 2025-11-20 · GitHub release

🏎️ Performance

  • Improve list versions performance when there are lots of snapshots (#2732)
  • Improve peak memory use when writing multi-index dataframes (#2749)

🐛 Fixes

  • conda-build: Pin libcurl <8.17 (#2746)
  • Add rollback support for recursively normalized data (#2738)

v6.3.1

Released 2025-11-03 · GitHub release

🐛 Fixes

  • Reduce number of requests when getting a library (#2722)
  • Delete written data keys when storage quota is reached. (#2707)
  • Fix staged data corner case when using finalize with delete_staged_data_on_failure and update docs (#2714)
  • Allow grouping on empty columns (#2727)
  • Fix reads of series with multiindex (#2726)
  • Update sparrow to 1.3.0 (#2725)
  • Register cleanup function reliably (#2730)

v6.2.6

Released 2025-10-20 · GitHub release

🐛 Fixes

  • conda build now depends on sparrow-devel (#2665)
  • Fix flaky filter test (#2694)
  • Performance 18146257858: speedup reading as_of tombstoned versions alive in snapshots (#2699)

v6.2.5

Released 2025-10-13 · GitHub release

🐛 Fixes

  • Increase the version when batch_append and batch_update are called with upsert and the symbol is tombstoned (#2661)
  • Fix changed message for GCP (#2676)
  • Raise exception if filtering recursively normalized data or numpy array (#2641)
  • Fix azure missing keys exception on mac (#2637)
  • Downgrade Windows distribution to windows-2022 in GitHub CI (#2672)
  • Increase testing coverage of the v1 API update_time, name, batch_restore_version (#2668)
  • Fix flaky test test_library_get_key_path (#2648)
  • Update installation tests with 6.1.1 version (#2682)

v6.2.4

Released 2025-10-08 · GitHub release

🐛 Fixes

  • Add UpdatePayload and DeleteRequest to arcticdb. import path (#2664)
# Now you can
import arcticdb.WritePayload
import arcticdb.UpdatePayload
  • v1 API - batch_read_an_join specific tests (#2659)
  • Add exception when trying to write an index or a column with a boolean name (#2667)
df = pd.DataFrame({"col": [1, 2, 3]}, index=idx)
df.index.name = True
# Below line now raises ArcticNativeException
lmdb_version_store.write(symbol, df)

df = pd.DataFrame({True: [1, 2, 3]}, index=pd.date_range(ts, periods=3))
# Below line now raises ArcticNativeException
lmdb_version_store.write(symbol, df)
  • Adds http response code to S3 error message (#2656)
  • Downgrade Windows distribution to windows-2022 (#2672) (#2675)
  • Fix asv test disk space issue (#2692)

v6.2.3

Released 2025-09-30 · GitHub release

🐛 Fixes

  • Fix range intersection calculation in query.cpp (#2632)
  • additional check for batch_metadata_multi (#2653)

v6.2.2

Released 2025-09-23 · GitHub release

  • Upgrade node version in CI to 24 (#2636)

https://github.com/man-group/ArcticDB/compare/v6.2.1...v6.2.2

v6.2.1

Released 2025-09-15 · GitHub release

🐛 Fixes

  • Update vcpkg baseline and sparrow to 1.1.0 (#2634)

v6.2.0

Released 2025-09-08 · GitHub release

🚀 Features

  • Add docstring and warning for disparity between will_item_be_pickled and is_symbol_pickle (#2548)
  • Remove feature toggle and update docstrings for staged data with tokens (#2614)

🐛 Fixes

  • Retry install Azurite (#2606)
  • delete_container() operation for Azurite should not be reason tests to fail (#2603)
  • asv flakiness fixes (#2590)
  • test: ARM64-check test_type_promotion_int64_and_float64_up_to_float64 (#2613)
  • Retry on OOM during tests execution (#2617)
  • xfail for now noisy test (#2621)
  • Increase coverage - append,update and stage v1 API (#2470)

  • Make copy_index_key_recursively parallelizable (#2622)

  • Remove dependency on numpy.typing (#2626) (#2628)

v6.1.1

Released 2025-08-26 · GitHub release

🐛 Fixes

  • Mem leaks - increase mem limits so that we have time to create ASV tests (#2558)
  • [Bugfix 9754509632] Fix use-after-stack-free (#2569)
  • Add missing type hints to Library and fix append return type (#2574)
  • Azure tests added (#2498)
  • Disable operation on objects of mismatched types leading to corruption [Bugfix 9754433454] (#2572)
  • Fix ASV test failures and Installation Test Failures on Master (#2578)
  • Fix test_type_promotion_int64_and_float64_up_to_float64 (#2557)
  • Fix workflow problems for Persistеnce test executions and ASV AWS S3 tests execution (#2585)
  • Remove signaled threads on TaskScheduler destruction on Windows with … (#2588)
  • Continue on error if macos wheel removal fails (#2599)

v6.1.0

Released 2025-08-20 · GitHub release

This is the first externally published release from the major v6 releases. As such it includes some breaking changes to the type system

⚠️ Breaking Changes

  • Groupby using dynamic schema now produces a stable dtype. Previously, the dtype depended on the segments being processed; now the dtype will always be the same dtype able to represent the column across all segments on disk.
  • Return type for min/max aggregations in groupby for libraries using dynamic schema now is the type able to represent the column across all segments, while in earlier versions it was float64 regardless of the type of the column across different segments.
  • Using sum aggregation in groupby on a bool column now returns the count of rows containing True as a uint64.
  • Using mean aggregation in groupby on timestamp columns now returns a timestamp (instead of float64). It is computed by taking the integer part of (1/n) * Σ ts[i], for i = 1 to n
  • The return type of a projection operation involving a floating point number will always be of type float64 regardless of the types involved in the computation.
  • Performing a filter or projection on an empty DataFrame might throw an exception depending on the dtype of the columns, while older versions always returned an empty DataFrame. The dtype of the columns of an empty DataFrame depends on the version of Pandas that is used to write them. Some versions use float64 by default; others use object. Filtering like query_builder = query_builder[query_builder["col"] < 5] will throw an exception if the DataFrame is empty and the type of an empty column is object.

🚀 Features

  • Resampling for libraries that use dynamic schema
  • Add batch_delete APIs and functionality (#2463)
  • Support open ended row_range on QueryBuilder and read methods (#2550)

🐛 Fixes

  • Fix handling of segments contained only of None values in sort merge (#2536)
  • Fix OverflowError from to_json() call (#2556)
  • Storage lock increase wait time and add artificial slow writes (#2497)
  • Remove signaled threads on TaskScheduler destruction on Windows (#2544)
  • Add utilities to let the storage failure simulator simulate high latency conditions (#2407)
  • Extend testing for the index names returned by get_info (#2448)
  • Fix flaky segfault in concat testing (#2458)
  • conda-build: Use clang and clang++ 18 for osx (#2476)
  • fix: Remove use after free (#2459)
  • Fix segfault during encoding sparse data (#2475)
  • maint: Replace Folly's ranges with the standard library's (#2479)
  • conda-build: Use macos-14 (#2482)
  • Bugfix 8083916814: Respect pickle_on_failure kwarg (#2474)
  • Remove numpy pin (#2487)
  • Update_batch additional tests (#2437)
  • Make append_batch and update_batch noop with empty dataframes when there's an existing version (#2507)
  • Release the GIL when logging from the Python API (#2486)

v5.10.0

Released 2025-07-29 · GitHub release

Performance

Fixes

v5.9.3

Released 2025-07-28 · GitHub release

🐛 Fixes

  • Remove numpy pin (#2494)

v5.9.2

Released 2025-07-23 · GitHub release

🐛 Fixes

  • Load only required data keys on edge case values of date_range (#2387)
  • Fix the __str__ for a query builder with row_range (#2418)
  • Validate snapshot names (#2415). This applies our normal symbol name validation to snapshots. Snapshot names must be at most 254 chars long, in ASCII range [32, 126] and must not include * < or >

🏎️ Performance

  • Improve performance of reads after a write_and_prune_previous call (#2460)

v5.9.0

Released 2025-07-07 · GitHub release

🚀 Features

  • Enhancement 9351235967: Support projecting a column of constant values (#2405)

🐛 Fixes

  • Remove arcticdb package dependencies on tests and do checks during build (#2403)
  • Fix Windows 3.13 CI (#2409)
  • Bugfix/9256801170/memory over allocation when reading shorts dfs (#2404)
  • fixed asv benchmark.json file (#2410)
  • Reimplement version map reload delete fix (#2395)
  • Apply perf improvements from pydata talk (#2408)
  • Validate staged data symbol names (#2414)
  • Test for test_admin_tools.py::test_get_sizes_for_symbol flakyness (#2398)
  • [Cherry-pick] Fix V2 API created from V1 API when use_norm_failure_handler_known_ty… (#2423)

v5.8.0

Released 2025-06-09 · GitHub release

🚀 Features

  • Query Stats s3 v3 (#2309)
  • Implement delete versions funcationality (#2352)

🐛 Fixes

  • Bugfix 9209057536: Allow concatenation of uint64 columns with int* columns (#2365)
  • Try to fix flaky compat tests (#2364)
  • Fix for scheduled workflow build failures to appear in slack (#2371)
  • New versions 5.5 and 5.6 added to installation tests monitoring (#2373)
  • Extend test coverage for recursive normalizers (#2366)
  • Respect legacy env vars (#2376)
  • Query Stats s3 v3 benchmark (#2346)
  • Update recursive normalizers layout assertions (#2377)

v5.7.0

Released 2025-06-02 · GitHub release

🚀 Features

  • Introduce a ternary operator for filters and projections (#2103)

The new ternary operator is designed to work like numpy.where, allowing operations like:

q = QueryBuilder()
q = q.apply("new_col", where(q["col1"] == 0, q["col2"], q["col3"]))

For more details, see the docstring and associated tests.

🐛 Fixes

  • Bug fixes for resampling (#2351)
    • Fix crash when resampling data that was updated with some particular old ArcticDB client versions
    • Fix resampling over a date_range not covered by the symbol's data
    • Better error message when resampling a non-timeseries indexed symbol
  • Fix for reading multiindex series with timestamps as the first level and strings as the second level (#2363)
    • This regressed in 5.6.0

📖 Docs

  • Add concat demo notebook (#2361). This is available on our docs site here.

v5.6.0

Released 2025-05-26 · GitHub release

🚀 Features

  • Disable s3 checksumming (#2337)
  • Enhancement 8277989680: symbol concatenation (#2142)

🐛 Fixes

  • Asv v2 s3 tests (Refactored) (#2249)
  • Installation tests v4 (#2339)
  • Installation Tests Workflow Fixes (#2354)
  • Add resolve_defaults back in as a static method of NativeVersionStore (#2358)

  • Patch for Real S3 library names (#2353)

  • File operations (#2345)

v5.5.1

Released 2025-05-19 · GitHub release

🐛 Fixes

  • Fix permissions for the automatic docs building (#2347)
  • Delete STS roles regularly (#2344)

v5.5.0

Released 2025-05-13 · GitHub release

🚀 Features

  • Query Stat framework v3 (#2304)

🐛 Fixes

  • Skip azurite api check (#2288)
  • Update docs for sts ca issue (#2265)
  • Hold the GIL when incrementing None's refcount to prevent race conditions when there are multiple Python threads (#2334)

v5.4.1

Released 2025-05-05 · GitHub release

🐛 Fixes

  • Timeseries defrag function (#2319)
  • initial version fix for GCP (#2326)

v5.4.0

Released 2025-04-28 · GitHub release

🚀 Features

  • Symbol sizes API (#2266)

🐛 Fixes

  • Refactor to_atom and to_ref to properly use forwarding references (#2321)
  • ci: Benchmark workflow adaptations (#2327)
  • Fix pr getting action (#2323)
  • conda-build: Extend development environment for Windows (#2328)
  • Prevent exceptions in ReliableStorageLockGuard destructor (#2331)
  • Increase tolerance to arithmetic mismatches with Pandas with floats (#2333)
  • Support MSVC 19.29 (#2332)
  • Fix race conditions on the None object refcount during a multithreaded read (#2320)

v5.3.4

Released 2025-04-22 · GitHub release

🐛 Fixes

  • More uniqueness in library names (#2305)
  • Remove python conda pin (#2287)
  • Replace moto import with one compatible with moto 5.0 (#2312)
  • Fix compatibility test cleanup (#2308)
  • Allow trimming ChunkedBuffers to zero bytes (#2313)
  • Add warning when integer types are promoted to float during aggregations with dynamic schema. (#2275)
  • Fix appending with compact incomplete to an empty dataframe (#2268)
  • Fix nfs storage tests and bugs in nfs storage (#2298)
  • Change CMake version so that coverity static analysis builds don't crash (#2299)
  • Update s3.py moto*.create_fixture - add retry attempts (#2311)

v5.3.2

Released 2025-04-07 · GitHub release

🐛 Fixes

v5.3.1

Released 2025-04-01 · GitHub release

🐛 Fixes

  • Precompute output schema from processing (#2233)
  • Fix convert_int_to_float in compact_incomplete V1 library API method (#2235)
  • Slightly wider tolerance to arithmetic mismatches with Pandas with floats (#2244)
  • Get object sizes based on S3's ListObjects output (just with scan_object_sizes() to start with) (#2248)
  • Add bounds check for empty tensors and enhance tests for numpy slicing (#2222)
  • Improve CI run time (#2215)
  • Add more logging to compat tests (#2250)
  • Accept strings for the mode when finalizing staged data (#2232)
  • Use is for None (#2256)
  • Correctly handle zoneinfo timezones (#2253)
  • Fix for STS support (#2254)
  • Backport fmt pin removal on conda to v5.3.1 (#2276)

v5.3.0

Released 2025-03-24 · GitHub release

🚀 Features

  • Implement batch_update (#2100)
  • GCP Support over the S3 XML API (#2176)
  • Expose set_config_int, and get_config_int on arcticdb.config, add config docs, documentation fixes (#2225)

🐛 Fixes

  • Include wheels in releases (#2180)
  • Fix s3 https connection on windows (#2182)
  • Fix off by one errors in staged segment validation (#2191)
  • Fix asv execution in PRs and run asv with chosen tests (#2188)
  • Add debug logging for finalize_staged_data (#2189)
  • Update supported Python versions docs (#2193)
  • Developmental releases on demand (#2190)
  • Skip the compat tests in Conda (#2194)
  • Code formatting (#1769)
  • Dev build naming and tag action fix (#2195)
  • Update utils.py to overcome usage of test package for asv tests (#2198)
  • Update test_mem_leaks.py not to fail on MacOS (#2197)
  • Remove cloudflare steps from docs build (#2200)
  • Remove Change Type template and change label names (#2196)
  • when workflow executed on schedule run all tests (#2205)
  • Fix job to work properly when called from other workflow (#2211)
  • Improve segment serialization for v1 segments (#2183)
  • Type promotion between integral and floating types (#2173)
  • Fix size issue (#2213)
  • Add pin for fmt in conda (#2223)
  • Memray tests for memory leaks for head() and tail() (#2199)
  • Release the GIL during parallel writes to avoid deadlock (#2220)
  • Fix and test arcticdb reading streaming data (#2218)
  • Return correct timestamp in the VersionItem returned by write when recursive normalizers are used (#2209)
  • Tolerate keys being deleted while scanning object sizes (#2206)
  • Fix regression when writing combination of np.NaN and None (#2212)
  • Fix test_chunks_the_same flakiness (#2239)
  • Testing for unicode handling across our API (#2224)
  • Automatically build docs on release (#2230)
  • Read and write snapshot metadata from the correct place (#2161)
  • Only publish benchmark results for master builds (#2157)
  • Fix issue with running only selected tests (#2155)
  • Multiple fixes for STS token refresh (#2115)
  • Extend update docs to mention possible data fragmentation (#2164)
  • Update Arrow so that ArcticDB can be compiled with Clang 19 (#2134)
  • Read arrow data (#2141)
  • Small improvements to test_compatibility.py (#2163)
  • DBA tools for ReliableStorageLock (#2152)
  • Bound the number of data segments kept live in memory during compaction and sort_merge (#2167)
  • Key segment pair refactor - do not move the segment out of the KeySegmentPair as it is a shared resource (#2166)
  • Remove unnecessary clone following the key segment pair refactor (#2169)
  • Enhance MockS3Client to support real client delegation and configurable failures (#2158)
  • Simplify our unpickling logic and add tests for Python 2 pickles (#2160)
  • Enhancement: Implement Library.get_key_path for NFS and Azure storages (#2172)
  • Automated Release scripts (#2174)
  • Real Storage asv perf POC (#2165)
  • Migrate Coverity (#2179)

v5.2.6

Released 2025-03-10 · GitHub release

  • Release the GIL during staged writes (writes with staged=True or parallel=True). This fixes a deadlock when staging a dataframe containing unicode strings.

v5.2.5

Released 2025-02-26 · GitHub release

What's Changed

v5.2.3

Released 2025-02-07 · GitHub release

🐛 Fixes

v5.2.2

Released 2025-02-03 · GitHub release

Fixes

Read and write snapshot metadata from the correct place PR #2161 .

Snapshot metadata is an optional structure of extra information saved with a snapshot, created when you call,

from arcticdb import Library
lib: Library

lib.snapshot("snap", metadata=["example", "metadata"])

lib.list_snapshots()
# {"snap": ["example", "metadata"]}

If you never call snapshot with metadata= then you are not affected by this issue. Metadata associated with a symbol is not affected by this issue.

Between v4.5.0 and v5.2.1 we mistakenly changed to save snapshot metadata in a new data format. This affects the snapshot metadata in the dict values returned by list_snapshots. Those versions incorrectly return None for snapshot metadata written by earlier versions, even if that metadata does exist.

This fix has us now check both data formats for snapshot metadata, so ArcticDB v5.2.2 onwards is compatible with snapshot metadata written by any version. We also revert to writing the metadata in its old location from v5.2.2 onwards. This means that ArcticDB clients between v4.5.0 and v5.2.1 will need to upgrade to be able to read snapshot metadata written by clients outside of those versions (otherwise they will see the metadata as None). ArcticDB clients older than v4.5.0 need to upgrade to v5.2.2+ to be able to read snapshot metadata written by versions between v4.5.0 and v5.2.1.

v5.2.1

Released 2025-01-29 · GitHub release

🐛 Fixes

  • Use Pandas unpickling to handle Pandas 1 vs Pandas 2 API differences better . This only affects reading Pandas structures saved as metadata with an ArcticDB symbol. (#2151)
  • Don't warn about missing keys when reading symbol ref as it is common to try to read a non-existent symbol (#2153).
    • This fixes a minor regression in v5.2.0 that caused noisy logging output like Failed to find segment for key 'r:aaa' : No response body..

v5.2.0

Released 2025-01-27 · GitHub release

🚀 Features

  • Introduce the block version ref key (#1969)
  • Add AWS STS authentication support (#1884)
  • Python 3.12 and Python 3.13 support (#1945) (#2016)
  • Numpy 2 support (#2050)
  • Reliable storage lock (#2014)
  • Storage mover (#2039)
  • Add S3 STS proxy support (#2072)
  • Implement origin for pandas resampling (#1962)
  • Refactor to storages to support async reads (#2012)
  • Chunk up incomplete segments by rows when they are staged (#2117)
  • read_batch performance improvements - now up to 10 times faster

🐛 Fixes

  • Fix decoding of fields with >2^16 blocks (#2089)
  • Handle very old normalization metadata RangeIndexes (#2118)
  • Finalize staged data memory use improvements (#2013)
  • Fix handling of empty DF in pandas 1.0 (#2010)
  • Fix string reference count leak (#1998)
  • Fix version release not attaching symbols for debug (#2018)
  • Delete staged segments after writing vref key (#2037)
  • Improve the performance of update by parallelising reads. Implement internal async update method. (#2087)
  • Performance regression when requesting a timestamp before the earliest version (#2076)
  • Using compact incomplete on a library with dynamic schema with a named index can result in an unreadable index (#2116)
  • Notimplemented handling (#2108)
  • Fix race between list_versions and delete_snapshot on NFS (#2092)

v5.1.3

Released 2025-01-21 · GitHub release

What's Changed

v5.1.2

Released 2024-12-10 · GitHub release

What's Changed

v5.1.1

Released 2024-12-05 · GitHub release

🐛 Fixes

v5.1.0

Released 2024-11-15 · GitHub release

🚀 Features

  • Enhancement 1895: Fully parallelise processing in read_batch (#1950)

🐛 Bugfixes:

  • Fix reference counting on python strings (#1999)
  • Make snapshot names in run_scenario more unique (#1961)
  • Fix Mac ARM Build (#1960)
  • Fix delete snapshot so that it doesn't orphan data keys or delete the wrong key (#1973)
  • Bugfix 1970: Give helpful error message if and/or/not operators are provided in QueryBuilder operations (#1976)

  • Bugfix 127: Improve error message when recursively normalized metastruct is too big (#1981)
    Fix publish result curl fail (#1974)

  • Bugfix/1841/maintain empty series names (#1983)

  • Bugfix/1937/do not write append ref keys when staging incompletes (#1985)
  • Bugfix 1306: Add nonreg test for update failing with 1-nanosecond different timestamps (#1978)
  • Test meaningful error message when reading incompletes from non-existent symbol (#1991)
  • Bugfix 1655: Disallow / character in mongo lib names (#1992)

v5.0.0

Released 2024-10-31 · GitHub release

⚠ API Changes

  • date_range returned by get_info[_batch] was a datetime.datetime now returns a pandas.Timestamp with nanosecond precision (#1461)
  • staged argument removed from write_pickle_batch (#1642)
  • Set row_count/rows to None in get_description/get_infoand batch versions thereof if symbol is pickled (#1664)
  • Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
  • Use float64 as the result type for all division operations in the processing pipeline (#1794)
  • Make SymbolDescription return type match type hints (#1877)

🚀 Features

  • Lazy dataframe implementation (#1703)
  • Arbitrary clause ordering (#1860)
  • Sort staged data before writing (#1869)
  • Make library manager a LRU cache (#1930)
  • Preserve nanoseconds and timezones in date range from get info (#1461)
  • Do not fallback to iterating snapshots if if version as_of int or timestamp not found in version chain in V2 API (#1672)
  • Use float64 as the result type for all division operations in the processing pipeline (#1794)

🐛 Fixes

  • Improve full build time by 30% by adding PCH (#1764)
  • Fix typo in docs flow (#1952)
  • Fix entt dep management for conda builds (#1955)
  • Run the pull request on the PR sha with the pull_request_target event (#1954)
  • Fix the timers macro, configure it with a cmake option (#1914)
  • Small fixes for mac arm compilation and fix some tests (#1905)
  • Fixes for sort and finalize (#1763)
  • Bugfix 1552: Set row_count/rows to None in get_description/get_info and batch versions thereof if symbol is pickled (#1664)
  • Bugfix 1641: Remove unused staged arg from write_pickle_batch (#1642)
  • Refactor 1749: make processing tests great again (#1758)
  • Always use strict weak ordering in std::sort comparators (#1747)
  • Use EncodedFieldImpl.blocks() rather than casting member variable _blocks (#1745)
  • Fix batch_restore_version failing to use correct next_version_id (#1823)
  • Bugfix arcticdb-man 96: Improve error message with pd.Timedelta columns (#1874)
  • Bugfix 1865: Make SymbolDescription return type match type hints (#1877)
  • Bugfix 1652: Improve error message when object dtype columns contains Timestamps with mixed timezones (#1880)
  • Bugfix 1841: Correctly roundtrip None and empty string pd.Series names (#1878)
  • Fix library options compatibility across ArcticDB versions (#1862)
  • Bugfix 1830: fix resampling with multiindex (#1873)
Uncategorized
  • Extend tests for merge sort (#1708)
  • Check if object is an instance of QueryBuilder when comparing for equality in QueryBuilder.eq (#1754)
  • Add resampling offset (#1743)
  • Refactor/1722/remove composite from processing pipeline (#1741)
  • Fix Segment use-after-move when replicating to NFS (#1756)
  • Throw exception if appending using sort_and_finalize will create unordered index (#1760)
  • Improve logging for stress tests (#1759)
  • Remove rocksdb (#1761)
  • Update BSL table for v4.5.0 (#1768)
  • Handle KeyNotFoundException in recurse_index_key (#1766)
  • Add Storage API to check for existence of a key matching a predicate (#1762)
  • Revert "Handle KeyNotFoundException in recurse_index_key" (#1776)
  • Add 1 Billion Row Challenge Notebook (#1774)
  • Remove KeyNotFoundException catch in CopyCompressedInterStoreTask (#1778)
  • Limit 3 item ref key bypass to only when loading undeleted versions (#1775)
  • append docstring improvement: added that append on a new symbol will create it (#1787)
  • Parallelize string handling, enable python type handlers, prepare for Arrow (#1698)
  • Add interface to pass a cached entry to tombstone all (#1793)
  • Fix sort index (#1796)
  • Fix windpws CI by adding the string header in decimal.cpp (#1801)
  • Remove unused RuntimeConfig feature (#1803)
  • Add unimplemented increment counter (#1804)
  • Pre arrow refactor (#1805)
  • Fix regression in bool pickling (#1810)
  • Add API to remove Prometheus metrics (#1806)
  • Fix symbol list bug on compaction (#1324) (#1798)
  • Allow metrics to be registered twice (#1816)
  • Docs 1809: lazy dataframe documentation and example notebook (#1815)
  • Bugfix 1818: Fix QueryBuilder equality checks (#1819)
  • Support reading from prefixes that include a dot (#1820)
  • Fix compilation errors for clang 18 (#1813)
  • Add StagedDataFinalizeMethod to docs (#1744)
  • Add demo video link to README.md (#1837)
  • More sort and finalize fixes (#1799)
  • Fix conda build (#1854)
  • Fix row slicing with sort_and_finalize throw on column slicing when the column group is larger than the segment column size (#1838)
  • Don't override the https value for backwards compatibility (#1840)
  • Support the ignore missing key read option in recurse_index_keys (#1844)
  • Add flag to clear staged data on failure to both finalize functions (#1856)
  • Change test for library emptiness to allow specifying key types to exclude (#1861)
  • Fixing clean builds (#1868)
  • Extend LMDB instance lifetime (#1879)
  • Allow customizing the most useful AWS SDK settings. (#1875)
  • Refactor 1833:use entt as ecs (#1834)
  • Fix mono CI issue (#1917)
  • Fix lib_tool.read_to_keys (#1832)
  • Skip lmdb compat tests for versions <= 4.5.0 (#1923)
  • Use relative paths for compat tests (#1924)
  • ci: Use mamba 2.0 via mamba-org/setup-micromamba@v2.0.0 (#1855)
  • Remove unneeded step (#1929)
  • Adds a type change utility for append data keys in lib_tool (#1932)
  • fix introduction help (#1933)
  • Rewrite the mkdocs template so the links point to the correct version (#1942)
  • Change the pull_request trigger to pull_request_target (#1947)
  • Revert pull_request_target back to pull_request (#1949)

New Contributors

v4.5.1

Released 2024-10-18 · GitHub release

🐛 Fixes

Uncategorized

v4.5.0

Released 2024-08-16 · GitHub release

🚀 Resampling (#1495)

We are pleased to introduce a first version of resampling into ArcticDB.
Using the QueryBuilder you can easily resample timseries data as part of the read. The following code will resample a symbol into hourly buckets, for example. Please see the QueryBuilder docs for more info.

q = adb.QueryBuilder()
q = q.resample("h").agg({"to_sum": "sum"})
lib.read("symbol", query_builder=q).data

🚀 Features

Remove Composite from processing pipeline (#1722)
Refactor VersionMap to include all combinations of LoadTypes and whether to include_deleted (#1443)
Multiple aggregators per column are now possible in QueryBuilder (#1468)
Add s3 custom ca support (#1338)
Add version_chain ASV benchmarks (#1508)
Support mongodb+srv connection strings (#1513)
Enhancement/515/support metadata in finalize staged data (#1525)
list_snapshots: Add option to skip loading metadata (#1531)
LibraryTool add read_index (#1533)
Make service alive checking more universal (#1607)
Enhancement/1610/method to force compact symbol list (#1624)

🐛 Fixes

Add maybe_unused to fix MSVC compilation error by @vasil-pashov in #1739
Properly hash descriptor in v1 (master) by @willdealtry in #1682
Bugfix 1692: fix batch read with row range no query builder (#1710)
Bugfix 1720: Speedup processing pipeline with lots of columns (#1723)
Fix use after move in op_log.cpp (#1732)
Implement support for very old style symbol list entries. (#1630)
Fix ArcticDB reading streaming data (#1647)
Add validate_index arg to staged data finalization in both V1 and V2 APIs (#1694)
Fix sort_and_finalize_staged_data (#1617)
Bugfix 1609: Allow parallel appends when incomplete index values match last existing index value in the symbol (#1619)
Fix SYMBOL_LIST Segment index out of bounds bug (#1473)
Fix overflow bug on append and update (#1585)
Bugfix/1250/compact incompletes rationalisation (#1467)
Guard against numpy v2 to prevent breakage when it gets released (#1465)
Consistently handle timezones in QueryBuilder filtering (#1483)
Align azure timeout with s3 (#1487)
bugfix/1247: Don't create an lmdb library if the name is invalid. (#1481)
Cleanup arctic client after usage in local_quary_builder in ASV (#1522)
Bugfix/1518/get description date range incorrect after delete data in range (#1523)
Bugfix 1507+1509: dynamic schema different index append fixes (#1529)
Only write ref key once when writing with prune_previous_versions (#1560)
Fix #1148: Show clearer error message for mismatched types in QueryBuilder (#1557)
Bugfix 1615: Fix segfault in benchmark_clause.cpp (#1616)
Bugfix 1612: Raise a meaningful exception type/message when an empty string is provided as a symbol/snapshot name (#1618)
Adapt Raise to use format string as message directly if no args are given (#1606)
Restore mistake in #1315 (#1482)
Update docstring: added section to clarify that an empty df does not delete (#1592)

New Contributors

v4.4.7

Released 2025-01-08 · GitHub release

What's Changed

v4.4.3

Released 2024-06-26 · GitHub release

🐛 Fixes

  • Release GIL on update and append (#1559)
  • Only write ref key once when writing with prune_previous_versions (#1571)
  • Fix overflow bug on append and update (#1585)
  • Require numpy < 2 (#1465)
Uncategorized

v4.4.2

Released 2024-05-08 · GitHub release

🐛 Fixes

  • Fix segmentation fault related to Ray package (by reverting to C++ 17 build) #1542
  • Fix get_description date range after using delete_data_in_range #1526
  • Fix tail segfault with empty data #1543

Please see v4.4.1 release notes for the previous changes

v4.4.1

Released 2024-04-16 · GitHub release

🐛 Fixes

  • Bug fix in Azure Storage, clear to_delete list to allow deletion of more than 256 keys in a batch (#1482)

Please see v4.4.0 release notes for the main changes from the last release.

v4.4.0

Released 2024-04-05 · GitHub release

🚀 Features

You can now modify library options on an existing Arctic library:

from arcticdb.options import LibraryOptions

ac: Arctic = ...
lib = ac.create_library("lib")
ac.modify_library_option(lib, ModifiableLibraryOption.ROWS_PER_SEGMENT, int(1e6))

See arcticdb/options.py for a description of the modifiable options.

🐛 Fixes

💾 Storage Exception Normalization

We have made storage-related exceptions uniform across different backend storage platforms, despite the fact that the underlying behaviour varies.

Uncategorized

v4.3.1

Released 2024-02-09 · GitHub release

🐛 Fixes

  • Fix regression in round-tripping empty type for dynamic schema (#1313)

v4.3.0

Released 2024-02-07 · GitHub release

Version 4.3.0 was pulled from PyPi and Conda Forge due to a regression. We no longer provide builds for 4.3.0.
Regression is fixed in 4.3.1 release. Please use 4.3.1 instead.

🚀 Features

  • Exposes existing regex filter in lib.list_symbols (#1123)
>>> from arcticdb import Arctic
>>> import pandas as pd
>>> ac.create_library("test")
>>> lib = ac["test"]
>>> lib.write("sym0", pd.DataFrame())
>>> lib.write("sym1", pd.DataFrame())
>>> lib.list_symbols()
['sym0', 'sym1']
>>> lib.list_symbols(regex="1$")
['sym1']
  • Introduce jitter in symbol list compaction threshold (#1174)
  • Sorting speed improvements in SegmentInMemory (#1181)
  • Reduce log level from warn to debug for "Failed to find segment for key" message where appropriate (#1130)
  • Speed up writes by parellising aggregator_set_data over data segments (#1065)
  • Support sortedness checks and maintenance with parallel writes and appends (#1251)
  • #1014 Introduce storage fixtures to easily test ArcticDB against various storage backends. See arcticdb.storage_fixtures package. (#1054)

🐛 Fixes

  • Release the symbol list's storage lock if it has existed for longer than its TTL (#1134)
  • Ensure that the version chain is always updated atomically (#1104)
  • Return empty pd.DataFrame with MultiIndex if originally provided (#1126)
  • conda-build: Explicitly depend on openssl and libcurl (#1244)
  • Reintroduce attrs as a runtime dependency (#1272)
  • Speedup reading wide dataframes that have no empty columns (#1225)
  • Bugfix 1046: Prevent appending/updating numeric columns with non-identical types with static schema (#1205)
  • Bugfix 1173: Correctly apply sortedness checks when calling update with date_range argument (#1238)
  • Fix non-deterministic hashing in Linux conda builds (#1261)
  • Improve date range returned by get info for unordered and range indexed dataframes (#1241)
  • Bugfix 1248 and 1249: compact_incomplete reject incomplete segments that overlap each other, or existing segments in the case of append (#1255)
  • Detailed error in case of S3's libcurl network failure (#1265)
Uncategorized
  • [Aggregation tests] Replace non_zero_numeric_type_strategies with numeric_type_strategies (#968)
  • Fixes reuse_name for azure storage #1061 (#1115)
  • small getting-started-docs tweaks (#1103)
  • Improve fixture reliability (#1116)
  • maint: Define arcticdb::proto::logger in log.hpp (#1117)
  • maint: Remove unneeded includes (#1113)
  • [Column] Move some definitions to cpp file (#1100)
  • maint: Move implementations in memory_segment_impl.hpp to memory_segment_impl.cpp (#1092)
  • Update git blame file (#1118)
  • Flaky test hypothesis mean agg (#496) (#1125)
  • Use same region for S3 and EC2 to avoid data transfer costs (#1128)
  • build: Remove attrs from the dependencies (#1135)
  • Only build on pull request events (#1127)
  • More fixture robustness improvements (#1132)
  • Remove releasing docs as they are now in GitHub wiki (#1136)
  • Update README.md (#1141)
  • Remove test parellism, and speed up test bottleneck (#1143)
  • Fix support for shared/unique S3 prefixes (#1140)
  • maint: Remove headers in types.hpp (#1121)
  • Skip flaky pytests which check log messages (#1161)
  • Update README.md (#1156)
  • Refactor: Move DataError method implementaitons into cpp (#1155)
  • Update .git-blame-ignore-revs for DataError implementation move (#1165)
  • Add MSVC 2022 preset. Tweak MSVC CMake settings. (#1133)
  • Build-time improvments: allocator.hpp, log.hpp, buffer.hpp (#1152)
  • Fix publish.yml workflow (#1167)
  • README - put third party tools in alphabetical order (#1172)
  • Fix persistent tests (#1147)
  • Introduce sorting and merging google benchmarks (#1138)
  • Skip array type tests due to occsional segfaults (#1187)
  • build: Remove some adherence to folly (#1144)
  • Add equity options notebook + data (#1178)
  • maint: Ignore some references (#1190)
  • Added equity options notebook to index (#1193)
  • Use vcpkg for gbench (#1189)
  • Forward port internal PR #1082 (#1180)
  • Bugfix 1191: Propagate storage failures in version map batch methods to calling code (#1194)
  • Link against python explicitly in order to make MSVS builds work (#1192)
  • Final version of equity opts notebook (#1196)
  • Bugfix 1182: Unskip test that is no longer flaky (#1197)
  • Docs that StorageFailureSimulator is not used in all stores (#1203)
  • Clean and reorganize OffsetString and StringPool (#1137)
  • build: Do not depend on protobuf-lite (#1212)
  • docs: Fix documentation links (#1038)
  • Fix recurse_segment forward declaration to match the signature of its implementation (#1217)
  • Update git blame file for OffsetString and StringPool implementation move (#1211)
  • Add frequently used items at the top level of arcticdb (#1219)
  • Switch from arcticdb to adb in the demo Notebooks (#1228)
  • Add a way to handle non-string values for index names (#1170)
  • Pass unmodified argument by const& to FieldCollection::add_field (#1234)
  • Switch from arcticdb to adb in python docstrings (#1236)
  • Remove obsolete test log level environment variable (#1231)
  • Update incorrect docs for validate_index (#1233)
  • Bugfix 1207: Use pandas.Timestamp.max - 1 day in test_read_ts. Remove pointless snapshot. Improve error message when index key reading fails. (#1235)
  • Bugfix invalid library name (#1206)
  • Enhancement/1253/skip temporary allocation when decoding dynamic schema columns (#1259)
  • Expose headers for consumers via arcticdb_core_static (#1257)
  • Update WarnVersionTypeNotHandled::warn() warning message (#1273)
  • Update README correcting spelling (#1275)
  • build: Adapt protobuf compilation (#1199)
  • Enable skipped test_partial_write_hashed (#1215)

v4.2.1

Released 2023-12-15 · GitHub release

This is a patch release to version 4.2.0 which fixes Issue #1157 regarding the defragment_symbol_data method.

🐛 Fixes

v4.2.0

Released 2023-12-12 · GitHub release

🚀 Features

  • Remove python deps that are no longer needed (#1005)
  • New row_range argument on read and ReadRequest (#864)
>>> from arcticdb import Arctic
>>> import pandas as pd
>>> df = pd.DataFrame({"col1": np.arange(10), "col2": np.arange(100, 110)}, index=np.arange(10))
>>> ac = Arctic("lmdb://test")
>>> lib = ac.get_library("test_lib", create_if_missing=True)
>>> lib.write("test_symbol", df)
>>> lib.read("test_symbol", row_range=(3,7)).data
   col1  col2
3     3   103
4     4   104
5     5   105
6     6   106

🐛 Fixes

  • Symbol list refactor (#796)
  • Fixed aggregation on sparse grouping columns (#1068).
    Depending on timestamps being accurate in the symbol list has proved to be troublesome. Instead, we should use the most recent version id known to a client as an indication of the client's view of the world at the time as symbol list entry is written. That way, we can identify and correct symbol list entries that refer to conflicting writes.

Notebooks

  • Added AWS blockchain notebook (#1040)
  • Added AWS blockchain to docs index (#1043)
  • Add Snapshot + Equity Notebooks (#1071)
Uncategorized
  • 744 extend real storage tests to run with large lifelike data and all api methods (#989)
  • Update BSL table for 4.1 (#1023)
  • Centralise the pytest marks (#1024)
  • Document the S3 backends that we have tested against and "un-beta" LMDB on Windows (#1016)
  • Sparse aggregation (#1007)
  • Docs versioning (#1008)
  • set-default after deploy so that 'latest' alias can be created first (#1029)
  • build: Remove old Cython configuration and adaptation (#1028)
  • Docs workflow fixes. (#1030)
  • build: Replace emilib with robin_hood (#995)
  • hot-fix: Use previous build of libmongocxx to avoid missing symbols (#1050)
  • Remove unused C++ Wangle dep (#1047)
  • Bugfix 1055: Unflake test_read_batch_time_stamp (#1058)
  • Change tag format in docs build (#1062)
  • Snapshot notebook typos (#1088)
  • Update vcpkg dep (#1091)
  • Enhancement/732/processing unit ecs model (#960)
  • Add xfail to flaky tests (#1087)
  • Add a mechanism to extend storage transaction lifetime to lifetime of… (#975)
  • Tweak release docs (#1019)
  • Change tmpdir to tmp_path (#1093)
  • Remove unnecessary xfails (#1097)
  • Add checks to see whether we should be validating version entries during compaction (#1099)
  • 941 self hosted runners for ci (#997)
  • Make dependency of pymongo optional in running (#1027)
  • Add preliminary change for slowdown error test (#1064)
  • Add mutex to ensure only single thread at pybind->c++ layer (#973)
  • Issue #1017 Only warn if the "base" LMDB env is opened twice (#1022)
  • Fix run-cmake action (#1034)
  • Add a fallback to free GH runners, when there is a problem with the self-hosted ones (#1063)
  • fix: Empty column handling improvements (#1049)
  • Pin all our Github actions deps (#1090)

v4.1.0

Released 2023-11-01 · GitHub release

⭐ New APIs

In-memory Backend

You can now open ArcticDB with an in-memory backend,

from arcticdb import Arctic
ac = Arctic("mem://")
ac.create_library("test")
assert ac.list_libraries() == ["test"]
# Create libraries as normal. Each `Arctic` object manages its own in-memory storage, so the lifetime
# of your libraries and data is the same as the lifetime of the `Arctic` instance that owns them.

ac2 = Arctic("mem://")
assert ac2.list_libraries() == []  # ac2 is backed by different memory to ac so the "test" library is not returned
Query Builder

We now support a new "count" aggregator. You can invoke it with:

q = QueryBuilder()
q = q.groupby("grouping_column").agg({"a": "count"})

⚠️ Breaking and API Changes

LMDB Backend

This release includes a fix for issue #850 : Ensure that LMDB libraries are readable after being moved to a different location. The fix means that LMDB libraries created with arcticdb>=4.1.0 will not be readable by older clients and those clients must update.

This is because the fix stops us from serializing the LMDB library path (instead we always prefer the one in the Arctic URI), but older clients still expect to see the LMDB path serialized. Older clients reading a new LMDB library will in fact ignore the path passed in to the Arctic constructor and instead read the current working directory.

When you exceed the LMDB map size, we now raise a custom exception arcticdb.exceptions.LmdbMapFullError that explains how to re-open LMDB with a larger map size, whereas previously we raised a less helpful arcticdb.exceptions.InternalException.

🚀 Features

  • Support count aggregator with groupby (#948)
  • Warning for LMDB when two Arctic instances open over the same storage (#1000)
  • Small LMDB Fixes: 2GiB map size for Windows, Validation before delete (#918)
  • Custom exception when LMDB map is full (#1006)
  • Memory backed API (#860)
  • Allow ampersand in symbol names (#900)
  • Add querybuilder notebook demo into the docs (#875)
  • Extended testing against real cloud storages (#789)
  • ASV Benchmarking published here (#962 #970)
  • Preparatory work for RocksDB backend (#945)

🐛 Fixes

  • Fix LZ4 decoding error issues that occurred with a mix of empty and non-empty columns (#964)
  • Performance improvement for read_batch when called with many symbols (4-5x improvement) (#870)
  • Convert semimap to switch which has resolved some segmentation fault issues (#912)
  • Upgrade cUrl to 8.4.0 (#977)
  • Cache open libraries in the LibraryManager (#990)
  • Enhancement 914: Improve error messaging when string column encoding fails due to the presence of a non-string object (#933)
  • Fix storage lock mutex implementation (#966)
  • Make azure sdk stick to winhttp if possible (#851)
  • Extra update checks (#539)
  • maint: Indicate the non-support of PyArrow (#882)
  • Add pymongo to the list of install dependencies (#891)
  • conda-build: Run python tests for macos-latest (#873)
  • fix: Change comparison in test_hypothesis_{sum,mean}_agg (#931)
Uncategorized
  • Fixes IFNDR issue due to mismatching inline/non-inline functions depending on the translation unit (fixes #943) (#949)
  • docs: Post 4.0.0 release documentation (#967)
  • Remove Black and pre-commit setup (#972)
  • Improve string writing performance (#969)
  • Skip docs in build.yml (#984)
  • Move api docs from sphinx to mkdocs (#897)
  • Document support for Mac on intel (#982)
  • Update PAT for publish to master (#988)
  • Fix ccache's non-existence exiting the workflow (#996)
  • Refactor get_descriptions lib methods to be more consistent (#994)
  • Fail docs build on sphix failure (#883)
  • docs: Better document publishing release candidates on conda-forge (#901)
  • conda-build: Unpin some dependencies (#888)
  • docs: Reword conda-forge section mentioning libevent-2.1.10 (#905)
  • Update readme to reflect supported/beta status of Windows PyPi/MacOS conda-forge builds (#907)
  • Skip flaky Mac test (#924)
  • docs: Improve section "Building using mamba and conda-forge" (#917)
  • fix interface in ManualClockVersionStore getter (#925)
  • docs: Add high-level documentation of abstractions (#628)
  • Update storage compatibility (#916)
  • Update copyright notice (#939)
  • Use new get_library argument in ArcticDB_demo_lmdb.ipynb (#930)

v4.0.4

Released 2024-03-14 · GitHub release

🚀 Features

  • Better backend storage retryable error handling and error message printout #1365

🐛 Fixes

  • Stop allowing snapshotting tombstoned versions #1280
  • Add a way to enable/disable silencing of errors when deleting a library #1271

v4.0.3

Released 2023-11-23 · GitHub release

This is a patch release to version 4.0 that backports some changes from master.

🚀 Features

  • Add preliminary change for slowdown error test (#1070)

🐛 Fixes

  • Empty column handling improvements (#1079)
  • Use previous build of libmongocxx to avoid missing symbols (#1083)
  • Remove docs publish step so we don't overwrite the docs (#1025)
  • Remove Black and pre-commit setup (#1085)

v4.0.2

Released 2023-11-02 · GitHub release

This is a patch release to version 4.0 that backports some changes from master.

🐛 Fixes

Bugfix for a deadlock issue when using Python multithreading and batch_read (https://github.com/man-group/ArcticDB/pull/1021)

v4.0.1

Released 2023-11-02 · GitHub release

This is a patch release to version 4.0 that backports some fixes from master.

🚀 Features

  • Allow ampersand in symbol names (#952)

🐛 Fixes

  • Backport minor fixes to 4.0.x (#954), not user facing

v4.0.0

Released 2023-09-27 · GitHub release

⚠️ API changes

For Library.get_description_batch, Library.read_metadata_batch and Library.write_batch, a DataError object will now be returned in the position in the list returned corresponding to the symbol/version pair there was an issue reading/writing. Note this may require code changes to support the new error handling behaviour - as a result it is being considered a breaking change as described above.

  • get description batch method: method rationalisation (#814)
  • read metadata batch method: method rationalisation (#814)
  • Write batch method: method rationalisation (#814)

🚀 Features

  • Pandas 2.0 support (#343) (#540) (#804) (#846)
    • Modifications have been made to the normalisation and denormalisation processes for pandas.Series and pandas.DataFrame to match the new defaults in pandas 2.0.
    • Handling of 0-row DataFrames for improved correctness and usability.
    • Empty Column are now properly handled, especially regarding the change of defaults for empty collections for Pandas 1.X and Pandas 2.X.
    • Extended the tests to reflect changes in behaviour due to pandas 2.0's new defaults.
    • Please note, PyArrow remains unsupported in this integration.
  • conda-build: Bring support for Azure Blob Storage (#840) (#854) (#853) (#857)
  • Add uri support for mongodb (#761)
  • Code coverage analysis and report workflow (#783) (#784)
  • Add documentation with doxygen (#736)

🐛 Fixes

  • Update support status: Pandas DataFrame and Series backed by PyArrow are not supported (#882)
  • Added pymongo to the list of installation dependencies (#891)
  • Resolved dependency issues for the mergeability check step (#822)
  • Fixed issue where AWS authentication wasn't used, even though the option was enabled (#843)
  • Resolved issue of early read termination in 'has_symbol' (#836)
  • Test: Ensured that QueryBuilder is pickleable with all possible clauses (#861)
  • Fixed issue with the 'latest_only' option for the 'list_versions' method (#839)
  • Added the ability for users to specify LMDB map size in the Arctic URI (#811)
  • Fixed issue 767: Segfault in batch write with string columns has been resolved (#827)(874)
  • Renamed ArcticNativeNotYetImplemented in a way that maintains backward compatibility, to fix issue #774 (#821)
  • Modified Azure SDK to favour winhttp over libcurl on Windows for improved SSL verification (#851)
  • Updated the maximum batch size for Azure operations (#878)
Uncategorized
  • Maintenance: Added a minimal Security Policy (#823)
  • Fixed documentation following an exception renaming (#824)
  • Resolved issues in the publish step (#825)
  • Added documentation for setting LMDB map size (#826)
  • Incorporated notebooks into the documentation (#844)
  • Maintenance: Removed unused definitions from protocol buffers (#856)
  • Enhanced error handling to fail document build on Sphinx errors (#883)
  • Maintenance: Replaced deprecated ZSTD_getDecompressedSize function (#855)
  • Refactored non-functional library manager, addressing Issue #812 (#828)
  • Made minor improvements to the documentation (#841)
  • Improved handling of the deprecated S3 option "force_uri_lib_config" (#833)
  • Corrected the release date of version 3.0.0 in README.md (#858)

v3.0.0

Released 2023-09-13 · GitHub release

🔒 Security + Forwards Incompatible Change

This fixes a security issue with ArcticDB where creds were kept in storage for:

  • Azure
  • AWS if the access keys are supplied in the URI instead of aws_auth=True.

These instructions explain how to upgrade your storage to remove the credentials. See also issue https://github.com/man-group/ArcticDB/issues/802 .

Compatibility matrix

Storage Library created with < v3.
Library accessed with >= v3.		 Library created with or upgraded to >= v3.
Library accessed with < v3.
S3 with aws_auth=True Continues to work Raises InternalException: E_INVALID_ARGUMENT S3 Endpoint must be specified.
Will work again if access=_RBAC_&secret=_RBAC_&force_uri_lib_config=true is in the URI passed to Arctic()
S3 with access and secret.

Will now use the creds passed to Arctic(), but should continue to work if the creds are sufficient.

A future release might print a warning with instructions to upgrade.

 Raises InternalException: E_INVALID_ARGUMENT S3 Endpoint must be specified.
Will work if force_uri_lib_config=true is in the URI passed to Arctic()
Azure Operations on the library will fail with various internal error messages

Full details:

What's happened?

Whilst reviewing our codebase we discovered a way that access-keys for ArcticDB storage backends could be saved into the storage in clear text.

This behavior was by design, but there is a chance that this has happened for some third-party users without being obvious.
This depends on the backend used and how you connect to the storage.

What is the exact scope of the issue?

If you created an ArcticDB library, either with an S3 bucket and passed the access-keys as part of the URI, or with Azure Blob Storage with the access-keys as part of the connection-string, then the credentials were saved into the storage account as part of the ArcticDB library config.
If you then shared that storage account with others using different roles or access-keys, then those users would in theory have been able to access the credentials used to create the library.

What have you done to address this?

We've updated ArcticDB so that all new libraries do not do this, even if the credentials are passed in with the URI/connection-string.
We've prepared a storage-update script which you can run to see if the credentials are there, and then remove them if they are.

What is the impact if I am affected?

If you have shared that storage account with anyone else using different roles/credentials, then your original credentials have also been accessible to those users.
It's possible those users recorded the credentials, and because those credentials must have had write-access to create the library, they could have made changes to the data or otherwise used those credentials.

What can I do to check if I'm affected?

See these instructions.

If needed you can check on previous versions of ArcticDB using the code referenced on github:
https://github.com/man-group/ArcticDB/issues/802#issuecomment-1697814768

What should I do if I am affected?

Follow these instructions.

This change is not forwards compatible, so users on earlier clients may need to upgrade:

  • S3 libraries created with 3.0.0 will not be readable by earlier ArcticDB versions unless force_uri_lib_config=True in their connection string.
  • Azure libraries created with 3.0.0 will not be readable by earlier ArcticDB versions.

Then,

  • Rotate your credentials.
  • If you've shared access to that storage account then please also check the integrity of your data and anything else accessible via those credentials.
What was the cause?

Previous use cases of ArcticDB had split storage accounts. One account was used to configure libraries and other accounts held the data for those libraries. Credentials to read those data-libraries were then stored into the configuration account and passed to users as needed for access to the data. This code was not caught during our review, and so was not disabled or removed when we made ArcticDB available to others. When we added Azure Blob storage support subsequently, the side-effect of saving anything in the connection-string to storage was not anticipated.

Having reviewed the codebase again we are confident that this was the only way that credentials could be saved into storage using our public API.

We plan to continue supporting our split storage solution for some users, but it should always be very clear when access-keys are being stored and what the risks are for that.

🚀 Features

🐛 Fixes

Uncategorized

v2.0.0

Released 2023-08-29 · GitHub release

This version contains breaking changes to the ArcticDB API. As per the SemVer versioning scheme, we have bumped the major version.

⚠️ API changes

  • Write batch metadata method: method rationalisation (#476)

  • Append batch metadata method: method rationalisation (#548)

For Library.write_metadata_batch and Library.append_batch, a DataError object will now be returned in the position in the list returned corresponding to the symbol/version pair there was an issue writing. Note this may require code changes to support the new error handling behaviour - as a result it is being considered a breaking change as described above.

See the docs for read_batch, which uses the same exception return mechanism.

  • (Minor) The internal protobuf field arcticc.pb2.descriptors_pb2.TypeDescriptor.MICROS_UTC has changed name to NANOSECONDS_UTC. This is only visible via the Arctic API as a string via get_description & get_description_batch on dtype attributes, so external users will only be affected by this if you are parsing these strings.

🚀 Features

  • Projections, group-by, and aggregations added to the processing framework (#712)
  • Reduce memory footprint of head and tail methods (#583)
  • Per symbol parallelisation for write batch metadata method (#476)
    • This can result in significant performance improvements when using this method over many symbols.
  • Per symbol parallelisation for append batch metadata method (#548)
    • This can result in significant performance improvements when using this method over many symbols.

🐛 Fixes

  • Ensure content hash is copied during restore version + fixing timestamp-uniqueness-related flaky tests (#600)
  • Restrict supported string types to type equality rather than has isinstance (#704)
  • Incorrect initialisation of LoadParameter::load_from_time_ (#697)
  • Ensure compact_incomplete and recursive normalization obey the library setting for pruning (#705)
Uncategorized
  • Update release process to detail the process for pre-releases (#688)
  • Unify release and pre-release hotfixing (#725)
  • Skip test_diff_long_stream_descriptor_mismatch on MacOS (#693)
  • maint: Remove VariantStorage (#695)
  • maint: Rename datetime64[ns]-related fields and datatypes (#592)
  • run C++ tests for conda build / ci (#486)

v1.6.2

Released 2023-10-04 · GitHub release

This release is a patch release, backporting bug-fixes to v1.6.1

🐛 Fixes

  • Fixed a bug in key data retrieval, which could lead to incorrect behavior and segmentation faults (#912 )

v1.6.1

Released 2023-08-09 · GitHub release

🐛 Fixes

Uncategorized
  • Remove SemVer version validation as it doesnt support version strings such as 1.6.1rc0 (#678)
  • Docs 672: Add docstring for DataError class (#686)
  • Black output diff rather than just erroring in build (#685)
  • Correct docs for read_batch return type (https://github.com/man-group/ArcticDB/pull/664)

v1.6.0

Released 2023-07-25 · GitHub release

⚠ API Changes

  • Modify read_batch to return a DataError object rather than raising an exception (#629)

For Library.read_batch, a DataError object will now be returned in the position in the list returned corresponding to the symbol/version pair there was an issue retrieving. This contains the symbol and version requested, and the exception string thrown. For two well-defined categories of error, the error category and specific error code are also included:

ErrorCategory.MISSING_DATA - ErrorCode.E_NO_SUCH_VERSION: The version requested by the user does not exist
ErrorCategory.STORAGE - ErrorCode.E_KEY_NOT_FOUND: At least one of the keys required to read the specified version does not exist in the storage.

Otherwise these fields of the DataError object are left as None.

See also the API docs.

  • Accessing a Library that does not exist now throws an arcticdb.exceptions.LibraryNotFound rather than an internal exception.

🚀 Features

  • Support negative integers in as_of (#589)

This PR enables negative indexing to select "the last-but-nth version". This is equivalent to negative indexing in Python. See API docs.

For example:

read(...as_of=0...) will select the first version
read(...as_of=-1...) will select the most recent version
read(...as_of=-2...) will select the version prior to the most recent version

🐛 Fixes

Various fixes to make our use of LMDB more correct. These only affect LMDB-backed Arctic instances (not S3). These should resolve the segfaults users have been experiencing with LMDB.

  • Only open library once in the interpreter lifetime (#585)
  • Fix for LMDB DBI Handling (#597)
  • Delete relevant part of LMDB tree upon library delete (#601)

Also,

  • Fix for write batch with dedup (#595)
  • Fix for read batch when as of is TimeStamp (#617)
  • Fix numeric isin filtering for some cases with a mix of signed and unsigned integers (#604)
Uncategorized
  • Update demo notebook (#570)
  • fix: Remove uneeded FMT_COMPILE (#578)
  • build: Use Pandas 2.0 forward compatible API (#582)
  • Make batch tests much faster (#586)
  • Remove tests skips in test_storage_lock.cpp for Windows (#550)
  • Update python line in issue-template so it works on windows (#608)
  • Update faq.md (#616)
  • Windows is Beta (#610)
  • Rename lib to be consistent - lib vs library (#622)
  • Update docs to point how to use AWS_PROFILE (#619)
  • maint: Remove VariantStorageFactory and its implementations (#625)
  • docs: Add development guidelines for testing combinations (#644)
  • conda-build: Pin pybind11 to < 2.11 (#647)
  • Add checklist to pull request template (#643)

v1.5.0

Released 2023-07-11 · GitHub release

🚀 Features

  • ☁️ ArcticDB now supports Azure Blob Storage! (#427 #464) ☁️.
    • Note: This does functionality is not yet available in the Conda release of ArcticDB - it is only available in the PyPI release.
  • Performance improvements for:
    • write_batch (#467)
  • Added optional library encoding option (#401)
  • Add environment variable controlling whether AWS S3 should verify SSL certificates (S3Storage.VerifySSL). This defaults to 1 - set to 0 to disable. (#553)
  • Specify compilations optimizations for Windows build (#543)
  • Improve documentation for Arctic 1.0 -> ArcticDB migration (#546)
  • Add test for filtering down the string pool with Nones and NaNs (#533)
  • ArcticDB now supports logging to a file for log messages (#573)

🐛 Fixes

  • Rename write_batch_pickle to write_pickle_batch. Note that this is a breaking API change. (#516)
  • Fix compilation errors issued by clang-16. (#542)
  • Fix for read batch when filtering by column. Previously this would raise an error - it now works. (#567)
  • Suppress Pandas warning about nanoseconds being discarded (#571)

v1.4.1

Released 2023-06-27 · GitHub release

🚀 Features

  • Significant performance improvements for get_description_batch, exploiting per-symbol parallelism. Bench-marking suggests an order of magnitude improvement for 1000 symbols.

🐛 Fixes

  • get_description_batch datetimes now include UTC tzinfo (fixing the batch equivalent of issue #197).

v1.4.0

Released 2023-06-23 · GitHub release

📣 Notices 📣

1.4.0 has changed the default value of prune_previous_versions.

Prior to 1.4.0, if you did not pass a specific value into prune_previous_versions, prior versions would have been removed after successful completion of the write, update or append operation. By changing the default value, previous versions will now be kept by default.

To maintain the behaviour of previous releases, pass prune_previous_versions=True or manually call prune_previous_versions.

🚀 Features

  • Support an option to allow Arctic to override storage endpoint and credentials. Useful if replicating a bucket containing existing ArcticDB libraries to another region. (#502)

Other Changes:

  • If list_symbols is expected to be slow (no recent cache), a warning will be printed. (#489)
  • Major refactor to the analytical engine of ArcticDB (#471)
  • prune_previous_versions has been set to False by default (#485)
  • Fix logging levels not being configurable from env var (#490)
  • Add UTC timezone info to dates returned by SymbolDescription::get_description (#480)
  • Use RFC 3986 URL encoding for S3 interactions (#503)
Uncategorized
  • Demo notebook (#477)
  • conda-build: Use fmt < 10 (#479)
  • Add guide on how to release ArcticDB to PyPi and conda-forge (#478)
  • Try removing enum-compat (#491)
  • Address post merging comments from prune_previous_versions set defaul… (#487)
  • Add requirements file (#501)
  • Remove reference to Arcticc and fix file config (#508)

v1.3.0

Released 2023-06-12 · GitHub release

🍎 New Platforms

  • This release coincides with the release of 1.3.0 on MacOS Apple Silicon on conda-forge!

🚀 Features

  • Multiple improvements to the reliability and stability of the symbol list caching logic. (#393)
  • Add get_uri function to the Arctic instance (#433)

🐛 Fixes

  • Significant performance improvements for read_batch, read_metadata_batch and get_description_batch (#415 #435)
    • Note that the performance improvements for read_batch will largely only be visible if no QueryBuilder query is passed in.
  • fast_tombstone_all option has been removed - in code this is assumed to always be true. (#366)
  • Simplify log config and fix #426 #453 (#474)
Uncategorized
  • rename isin test to avoid being skipped (#418)
  • github: Adapt bug report template (#429)
  • github: switching action for micromamba (#424)
  • #146 Add Black formatting to build (#431)
  • Higher utility GitHub Actions changes (#438)
  • fix black formatting issues previous PR (#444)
  • dev: Add simple pre-commit setup (#445)
  • build: Do not build with Remotery by default (#466)
  • follow-up: dev: Add simple pre-commit setup (#448)
  • cmake: Improve the resolution of LZ4 (#451)
  • fix: Required adaptations for MSVC 14.29.30133 support (#459)

v1.2.1

Released 2023-05-31 · GitHub release

This is a bugfix release:

  • #419 #409 Build fixes to enable the Conda build (and therefore publishing to conda-forge)
  • #388 ArcticDB will now allow updates on data if the existing data was written prior to v1.1 regardless of whether that existing data is sorted or not

ArcticDB distributes:

  • wheels for Windows and Linux on PyPI, the latest being installable with:

    pip install --update arcticdb

  • conda packages for MacOS and Linux on conda-forge, the latest being installable with:

    mamba install -c conda-forge --update arcticdb

v1.2.0

Released 2023-05-22 · GitHub release

🚀 Features

✅ defragment_symbol_data  method added (#180)

  • This will defragment fragmented symbols. Fragmented symbols are typically caused by frequent small appends (e.g. 1 row every hour) and result in sub-optimal read performance. Reading defragmented symbols can result in significantly improved performance compared to a more fragmented equivalent!

✅ Write API for column-level statistics added. (#253)

  • The structures created using this API will later be used to improve the performance of queries in later versions. As of version 1.2.0 however, the structures are not consumed at query time.

✅ Configuration variables set via environment variables are now case insensitive (#364)

🐛 Bug fixes

✅ add_to_snapshot  and remove_from_snapshot  no longer leave unreachable data on disk if a symbol-version pair being removed from a snapshot is the last reference to this version (#253)

✅ Reads with QueryBuilder parameters that contain a groupby/aggregation  clauses now only fetch data segments containing columns relevant to the query (#253)

v1.1.0

Released 2023-05-10 · GitHub release

New Platforms

✅ Version 1.1 includes the first official release of the Windows version of ArcticDB!
Versions 1.1 includes the following caveats:

  • Please have a recent Visual C++ Redistributable for Visual Studio 2015-2022 installed.
  • Windows does not support fixed-width NumPy strings and so will not be able to write fixed-width strings contained within a NumPy array.
  • Writing pickled data can fail on machines with limited memory, see #340 for details and a workaround.
  • The LMDB store on Windows pre-allocates the max storage size on disk, so we limited the size to 128MB. We will provide an option to increase the size in the future (#229).

✅ Version 1.1 now supports Python 3.11!

New Features

✅ batch_get_descriptor method added (#219)
✅ batch_get_metadata method added (#220)
Both batch methods do not achieve optimal parallelism - we are aiming to address in 1.2.

Changes

✅ 1.1 pins to Pandas < 2.0 whilst we address a few remaining compatibility issues 🐼 (#237)
✅ 1.1 supports Protobuf V3 as well as V4
✅ Out-of-order data updates now raises exceptions should they prevent future indexed reads (#203)
✅ PCRE is now statically linked (#321)
✅ stream id which is a confusing term has been renamed to symbol in error messages and should be better understood (#311)
✅ Pickled data supports up-to 4GB (#260)
✅ Debugging toolbox added (#209)
✅ Introduced a base exception type arcticdb.exceptions.ArcticException. Exceptions are exposed in arcticdb.exceptions module with the following hierarchy:

RuntimeError
└-- ArcticException
    |-- ArcticNativeNotYetImplemented
    |-- DuplicateKeyException
    |-- MissingDataException
    |-- NoDataFoundException
    |-- NoSuchVersionException
    |-- NormalizationException
    |-- PermissionException
    |-- SchemaException
    |-- SortingException
    |   └-- UnsortedDataException
    |-- StorageException
    |-- StreamDescriptorMismatch
    └-- InternalException

Next version:

NEW FEATURE: defragment_symbol_data method added (#180)