DuckDB – Data power tools for your laptop, now in Clojure (2023)
This article discusses the integration of DuckDB into the Clojure-based tech.ml.dataset platform to improve data processing efficiency. It highlights how the transition from row-based to batched column-major processing allows for better handling of large datasets without needing complex cluster infrastructure.
Our in-memory column-major data processing platform, tech.ml.dataset (TMD), drives the future of functional data science. When data gets large enough not to fit in memory, one can continue with TMD by operating on samples of data, or otherwise filtering to relevant subsets to fit in bounds imposed by the working environment. Moreover, one can accomplish persistence, for data small and large, with nippy, arrow, or parquet .
Get the full story
Sign up for Headlinne to unlock AI insights, political bias analysis, and your personalized news feed.
Create free accountAlready have an account? Sign in