Please take a minute to remember that it exists!

Some great HN posts:

https://news.ycombinator.com/item?id=27016630

https://news.ycombinator.com/item?id=31318708

https://news.ycombinator.com/item?id=26580614

Oh, then you'll love: Hosting SQLite databases on GitHub Pages or any static file hoster (2021), https://news.ycombinator.com/item?id=27016630

This seems really neat. Curious if you saw this magic SQLite -> Static file tool shared here a couple of weeks ago: https://news.ycombinator.com/item?id=27016630

IIUC It would enable this tool to work with a dataset in sqlite that could be recompiled as static files periodically, and the relevant queries could be made on the front-end without having to load the whole data-set or select a CSV file from the filesystem. Would also lead to much better performance if you want users to be able to host generated charts on a static file server, presumably the output could be committed to git and then synced to a github pages or something.

edit: It appears the author of that tool is using Plotly as well in the demo website. An integration of these would be incredible.

There was neat post https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?

I couldn't find do these wrappers support read-only SQLite databases with HTTP range requests, like in this famous post [1]. Phiresky's wrapper supports it, but it seems to be rebuilding the whole sql.js [2], I'd rather have it as VFS on top of sqlite.org's own WASM module. I like the idea of HTTP range requests, but I don't want to run a fork, that will be unmaintained in few years.

[1]: https://news.ycombinator.com/item?id=27016630

[2]: https://github.com/phiresky/sql.js-httpvfs

This is a great book, there’s a few tools I would add.

Datasette, clickhouse local (cli) and duckdb.

I think ripgrep is a big omission, ripgrep | xargs jq and find -exec jq is one of my most common data science workflows cause you can get stuff done in a few minutes. An example of where I use this is to debug Infrastructure as Code that is generated for many regions and AZs quickly.

Another one I like in this space is Bioninformatics Data Skills, for some reason Bioinformaticians use CLI workflows a lot and this book covers a lot of good info for those who are just starting out like tmux, make, git, ssh, background processes.

Two other techniques I like are git-scraping (tracking the changes of data over time or just saving snapshots of your data to git so you can diff it): https://simonwillison.net/2020/Oct/9/git-scraping/ I most recently have used this technique to diff changes to build artifacts over time.

This technique is not really CLI related per se, but I really like the http range query technique of hosting data: https://news.ycombinator.com/item?id=27016630 There’s simple ways to use this idea (like cooking up a quick h2o web server conf) to host data quickly.

I also like the makefile data pipeline idea, I believe the technique is described in the book but I first heard of it from this HN comment: https://news.ycombinator.com/item?id=18896204 The basic idea is that you can use make to orchestrate the steps of your command line data science workflow and let make figure out when your intermediate data needs to be generated again. A good example is this map reduce with make here: https://www.benevolent.com/news-and-media/blog-and-videos/ho...

Similar idea here that may be of interest: SQLite DB on static host via HTTP range + WASM.

https://news.ycombinator.com/item?id=27016630