Scale-Out-File-Storage

Dave submitted a project plan on September 15, 2010.

Emails --

Dave: I shared the project plan with Ben Rogers, HCIS, and he sent some feedback.

Ben:

1. Silicon Mechanics servers: 6GB of ram is likely too low if you go with OpenSolaris as performance improves substantially with larger amounts of ram for caching. Might not be as big of deal on RHEL but based on what we are seeing with Lustre on CentOS I think this is a bare minimum there as well.
2. I wasn’t aware of the Trash plugin, could you point me to more info on that (quick google didn’t reveal what I was looking for).
3. Usable storage – We’ve seen between 50 and 55TB of usable storage on the Silicon Mechanics boxes depending on the raid configuration. I think 55TB is about the max you’ll see, probably won’t make it to 58TB, though that’s not a huge difference I thought I’d let you know.
4. We now have Lustre running on the cluster over Infiniband if you are still interested in taking a look at it.

I assume you are aware at this point that OpenSolaris is a dead product. We aren’t sure yet what we’ll be replacing it with, likely Solaris Express Edition, Illumos, or Nexenta.

Dave: Another note from Ben. He mentioned the idea of some shared spare kits on campus, since multiple groups have Silicon Mechanics gear now.

Mark: Ah – I saw #1 on the project plan and forgot to point that out – the quote is for 24GB of ram in each box and not 6GB

3 – could you ask Ben what his raid layout is? Is he using raid 6?

Ben: Ahh, 24GB should be fine. We’ve gone with 48GB but that’s because we’ve been using compression and have considered deduplication.

We are using Raidz2 (raid 6) on our systems. With two raid groups each with 18 drives we see ~55TB and with three raids groups of 12 drives each we see ~51TB of available storage. We aren’t using any hot spares, we just have the cold spares on site.

Dave: I asked Ben who already has Silicon Mechanics hardware on campus, and who’s managing it:

“At this point the groups I know of that have Silicon Mechanics gear are as follows:

Engineering – Iowa Institute for Biomedical Imaging – 3 units
Engineering – IIHR – 10 units (I believe they are on order but haven’t arrived)
ICTS – 5 units (Partial Ownership by Psychiatry)
Neurology – 1 unit
DNA Core – 1 unit

“Systems are housed in a variety of locations. Management is being done by individual groups but we are assisting and groups are generally using our puppet management system.”