If you are a data scientist or do anything with data... duckdb is like a swiss army knife. So many great ways it can help your workflow. The original video from CMU in 2020 [1] is a classic. Minutes 3-8 present a good argument for adding duckdb to your data cleaning/processing workflow.
And if you want to add a semantic layer on top of data, Malloy [2] is my favorite so far (it has duckdb built in):
What you may be remembering were reports of exceptional cases where it didn’t handle out of memory errors well. I was one of the people affected. I was running complex analytic queries on 400 GB parquets and I only had 128GB memory. It used jemalloc which didn’t gracefully degrade. They fixed a lot of the OOM issues so it’s more robust now. I haven’t had a crash for a long time.
If you are a data scientist or do anything with data... duckdb is like a swiss army knife. So many great ways it can help your workflow. The original video from CMU in 2020 [1] is a classic. Minutes 3-8 present a good argument for adding duckdb to your data cleaning/processing workflow.
And if you want to add a semantic layer on top of data, Malloy [2] is my favorite so far (it has duckdb built in):
[1]: https://www.youtube.com/watch?v=PFUZlNQIndo [2]: https://docs.malloydata.dev/documentation/
Thank you for the recommendation on that video! I've already adopted to using DuckDB for my ad-hoc analytics work but I didn't know the background
Analytics with type-safe raw SQL (including DuckDb’s awesome extensions) is pure gold:
https://github.com/manifold-systems/manifold/blob/master/doc...
Over the years I've seen anecdotes here on HN that DuckDB crashes often for several people. Is this still an issue for anyone?
We use it heavily at my workplace. It doesn't crash at all if you use it as OLAP. But if you use it incorrectly, it will crash.
It's pretty solid.
I use DuckDB daily.
In short — It doesn’t crash often at all.
What you may be remembering were reports of exceptional cases where it didn’t handle out of memory errors well. I was one of the people affected. I was running complex analytic queries on 400 GB parquets and I only had 128GB memory. It used jemalloc which didn’t gracefully degrade. They fixed a lot of the OOM issues so it’s more robust now. I haven’t had a crash for a long time.
On normal sized datasets it never crashes.
Never seen this and have several products that use it...
The actual slides are linked from the intro-text:
https://github.com/DBatUTuebingen/DiDi
That was a good start for understanding DuckDB internals!!!
Unfortunately it does not seem that there are lecture videos.
thank you! Learned why DuckDB is named this way
Am I missing something or is the content empty?
https://github.com/DBatUTuebingen/DiDi
Thank you, I didn't realize all of the course counted as "slides and auxiliary material" haha
edit: Really great stuff in here. Every day at work I think about how much I love DuckDB
What do you use it for? What's the best part for you?
Computing page rank on par with NetworkX: https://github.com/idesis-gmbh/WikiExperiments Educational local DW from Github Archive events: https://github.com/idesis-gmbh/GitHubExperiments
It is quite fast for OLAP applications. It works on low cost hardware.