Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Minor nitpick: using LZMA to compress large text files like this before distribution is normally better; here 7z is LZMA, zip is DEFLATE:

    739M    2013_data
    37M     2013_data.7z
    78M     2013_data.zip
Using [1] for reference, if the download speed is less than ~20MB/s, LZMA is faster than DEFLATE. Though, the data there is a bit less compressible than these csv files, so the break-even point for transfer rate would be higher here; even so, in my case, the download speed was much slower than 20MB/s.

1. http://richg42.blogspot.com/2015/01/parallelized-downloaddec...



Disclaimer: I work at Backblaze.

We tend to favor ZIP as a company because with no additional tools on all platforms (like Macintosh and Windows) it unpacks. No additional technology other than what the manufacturer provides.

In this case, we could provide the raw data in several formats in case bandwidth is a problem, plus I think we should provide the SHA2 or md5 of the resulting package just in case you are wondering if you got the correct download or whether somebody has messed with the contents.


Windows has the 7zip program which handles lzma just fine - it's overall a better program than zip anyway.


He said provided by the manufacturer by default, not something the user has to install themselves.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: