Seagate has just publicly announced 8TB HDD’s in a 3.5” form factor. I decided to do some rough calculations to understand the density a bit better…
Note: I have decided to ignore the distinction between Terabytes (TB) and Tebibytes (TiB), since I always work in base 2, but I hate the -bi naming conventions. Seagate is most likely announcing an 8TB HDD, which is actually smaller than a true 8TiB drive. If you don’t know the difference it’s worth learning.
Rack Unit Density:
Supermicro sells a high density, double-sided 4U server, which can hold 90 x 3.5” drives. This means you can easily store:
90 * 8TB = 720TB in 4U,
720TB/4U = 180TB per U.
To store a petabyte of data, since:
1PB = 1024TB,
1024TB/180TB/U = 5.68 U.
Rounding up we realize that we can easily store one petabyte of raw data in 6U.
Since an average rack is usually 42U (tall racks can be 48U) that means we can store between seven and eight PB per rack:
42U/rack / 6U/PB = 7PB/rack
48U/rack / 6U/PB = 8PB/rack
If you can provide the power and cooling, you can quickly see that small data centers can easily get into exabyte scale if needed. One raw exabyte would only require:
1EB = 1024PB
1024PB/7PB/rack = 146 racks =~ 150 racks.
Raid and Redundancy:
Since you’ll most likely have lots of failures, I would recommend having some number of RAID sets per server, and perhaps a distributed file system like GlusterFS to replicate the data across different servers. Suppose you broke each 90 drive server into five separate RAID 6 bricks for GlusterFS:
90⁄5 = 18 drives per brick.
In RAID 6, you loose two drives to parity, so that means:
18 drives - 2 drives = 16 drives per brick of usable storage.
16 drives * 5 bricks * 8 TB = 640 TB after RAID 6 in 4U.
640TB/4U = 160TB/U
1024TB/160TB/U = 6.4TB/U =~ 7PB/rack.
Since I rounded a lot, the result is similar. With a replica count of 2 in a standard GlusterFS configuration, you average a total of about 3-4PB of usable storage per rack. Need a petabyte scale filesystem? One rack should do it!
Storage is getting very inexpensive. After the above analysis, I feel safe in concluding that:
Disclaimer: I have not tried the 8TB Seagate HDD’s, or the Supermicro 90 x 3.5” servers, but if you are building a petabyte scale cluster with GlusterFS/Puppet-Gluster, I’d like to hear about it!
You can follow James on Mastodon for more frequent updates and other random thoughts.
You can follow James on Twitter for more frequent updates and other random thoughts.
You can support James on GitHub if you'd like to help sustain this kind of content.
You can support James on Patreon if you'd like to help sustain this kind of content.