Qbeast-io/qbeast-spark ↗
Created Jun 7, 2022 · View the Qbeast-io/qbeast-spark repository page
Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!
An animated visualization of Qbeast-io/qbeast-spark's Git history from September 2021 to June 2022, built by 7 contributors. The codebase reached 1,815 lines, mostly Scala. The video runs 34 seconds.
Want this for your repo?
Render a free sample of any GitHub repo in seconds.
Contributors
7
Lines of Code
1,815
From
Sep 23, 2021
To
Jun 3, 2022
Advertisement
About Qbeast-io/qbeast-spark
Qbeast Spark is an Apache Spark extension designed to enhance data processing in data lakehouses by providing multi-dimensional filtering and efficient data sampling capabilities. Built on Delta Lake, it maintains ACID properties for data integrity while enabling faster queries through intelligent indexing and sampling strategies. The project targets big data environments where query performance and statistical accuracy are critical.
The extension implements multi-column indexing using the Qbeast format, allowing users to filter and sample data across multiple dimensions simultaneously. A key feature is the improved sampling operator that can read statistically significant subsets of files, with a table tolerance model that lets users trade off sampling fraction against query accuracy. In demonstrated benchmarks, the tool achieved query execution approximately 22 times faster than Delta format alone with only 0.034% error using 1% sampling.
Users can work with Qbeast through both Scala code and SQL syntax, with operations including indexing datasets, dynamic insertion of records, and table optimization to improve query performance. The project maintains compatibility with Apache Spark 3.5.x, Hadoop 3.3.x, and Delta Lake 3.1.x, and includes comprehensive documentation covering quickstart guides, algorithm details, and cloud storage recommendations. A Python index visualizer is also provided for examining the index structure and sampling metrics.
Advertisement