Show HN: I Built 0xFast Stream – 100x Faster Ethereum Block Downloads

manishrjain.com

13 points by mrjn 13 days ago

Getting web3 data out of Ethereum nodes or node services is excruciatingly slow (at least days if not weeks). I had wanted a way to just download the blocks quickly since I started exploring the web3 space. Took me a bunch of trial and errors to get to a way where I can parse all the blocks since genesis within 12 hours. But, instead of moving on to the next thing, I put this service together over the last few days, so everyone can benefit.

high_byte 13 days ago

wow can't believe just yesterday I thought how cool it would be to have that - and today you made it! :D

downloading...

btw did you verify (PoW) blocks using the nonce? just to be on the safe side :)

  • high_byte 13 days ago

    also if you could use binary encoding rather than json you could probably cut the size by half.

    I already started download, but perhaps it's worth noting in your website to use "--compressed" flag like so:

    curl --compressed -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/1-latest

    • mrjn 13 days ago

      Oh, --compressed is is a great point, thanks. Updating the blog.

      I wanted to stick to JSON -- it's just the easiest for everyone to work with. And it compresses well, so all good.

  • mrjn 13 days ago

    Glad it's useful! :-)

    > btw did you verify (PoW) blocks using the nonce?

    I've validated the data and checksummed while exporting from Geth. But, perhaps can validate data when serving it as well. Any pointers / articles on the best practices here?

kevinsimper 13 days ago

Are you exposing full node or archive node? Because Archive node is several terabytes :)

  • mrjn 13 days ago

    It doesn't have the states yet. So, equivalent of full nodes -- block, txns, receipts, logs (no traces, states).

    But, could be extended to include those, if needed.