10 years ago I worked at a university that had a couple people doing research on LHC data. I forget the specifics but there is a global tiered system for replication of data coming from the LHC so that researchers all around the world can access it.
I probably don’t have it right, but as I recall, raw data is replicated from the LHC to two or three other locations (tier 1). The raw data contains a lot of uninteresting data (think a DVR/VCR recording a blank TV image) so those tier 1 locations analyze the data and removes all that unneeded data. This version of the data is then replicated to a dozen or so tier 2 locations. Lots of researchers have access to HPC clusters at those tier 2 locations in order to analyze that data. I believe tier 2 could even request chunks of data from tier 1 that wasn’t originally replicated in the event a researcher had a hunch there might actually be something interesting in the “blank” data that had originally been scrubbed.
The university where I worked had its own HPC cluster that was considered tier 3. It could replicate chunks of data from tier 2 on demand in order to analyze it locally. The way it was mostly used was our researchers would use tier 2 to do some high level analysis, and when they found something interesting they would use the tier 3 cluster to do more detailed analysis. This way they could throw a significant amount of our universities HPC resources at targeted data rather than competing with hundreds of other researchers all trying to do the same thing on the tier 2 clusters.
If I where you I would just buy a regular case that can fit a decent amount of HDDs like a fractal define 7 or one of its older versions and transplant your current computer into that with some new drives. 100tb is 5 20tb drives so you don't need that many.
USB enclosures are not a great way to handle storage as USB tends to be unreliable.
Yes, Ubuntu 20 isn't EOL yet. A lot of those downloads are probably IT staff or developers that are running Ubuntu servers or developing on those versions.
ETA: We still have some RHEL 7 and clones at my day job
20.04 and 22.04 were LTS versions, aka, long term support.
Any application that requires stability should run on LTS versions. Combined with Ubuntu being one of the most popular distros, makes 20.04 and 22.04 the most popular choices for anything in a home lab and many smaller business needs.
Whether you're building a server for home DNS, or a time server for a small business, then you're probably using Ubuntu as the base.
I think the next LTS version will be 24.04, so things might shift sometime after that.
Sounds like these 80 year olds need some friendly data hoarders to help them to digitise their collections. (Or for the BBC to promise to return the film, undamaged, once they've digitised them.)
If they want it so much why don’t they pay him? Sounds like if it weren’t for him (and the others he seems to allude to) we wouldn’t have this opportunity.
ZFS and BTRFS could update their codebase to account for these (if they haven't already), but I agree that their extra mechanical parts worry me. I really don't care about speed - if you run enough HDDs in your RAID then you get enough speed by proxy. If you need better speeds then you should start looking into RAM/SSD-caching etc. I'd rather have better reliability than speed, because I hate spinning rust's short lifespan as-is.
Timd to update your criteria, friend. Seagate hasn't been top of the failure stack for like 8 years now. The 3TB scandal era is long since passed. Now it's WD who has been shitting on quality control, sending out faulty SSD's that wipe user data, bait-and-switching HDD customers with a cheaper, much worse performing technology (SMR) WITHOUT TELLING THEM, them basically blowing corporate raspberries at everyone when people complain.
While i agree they were the best, HGST also hasn't even existed as a non-WD product for years....
Care to elaborate? Seagate is one of my favorite brand. And i read a lots of reviews and tech articles before purchasing any components. I am curious to learn about what i have missed about them. Thx
A lot of people have very strong opinions of brands based on a woefully inadequate sample size. Typically this comes from a higher than expected failure rate, possibly even much higher than expected. It could've been a bad model, a bad batch at manufacturing, improper handling from the retailer, or even an improper running environment. But even the greediest data hoarders only have a few dozen drives, often in just a couple of environments and use-cases.
Very few of these results are actually meaningful trends. For every person that swears by WD and will never touch a Seagate, there's someone else that swears by Seagate and will never touch another WD. HGST and Toshiba seem to have a very slight edge on reliability, but it's very small. And there are still people that refuse to touch them because of the "Death Star" drives many years ago.
It's also very difficult to predict which models will have high failure rates. By the time it becomes clear one is a lemon, they're already EoL.
I avoid buying WD new because of their (IMHO completely illegal) stance on warranty, but I'm comfortable buying their stuff used.
Don't worry too much about brand. Instead go for specs and needs. Follow a good backup strategy and you'll be fine
HGST is a part of WD and has been for quite a while.
But a big part of why the average consumer drive kind of sucks is that there is way more money in enterprise level drives so very little resources get put toward client drives.
Owned by, yes. Have their operations actually been integrated though? I haven't checked in a long time, but it was still a separate division last time I did.
Never ever going to buy Seagate again after the crap they've pulled on their Exos drives.
They simply decided to completely trash SMART and spin down commands. The drives simply won't give you useful SMART data nor they won't ever actually spin down, you can't force it, the drive will report is as if it was spun down but in reality its still spinning.
datahoarder
Oldest