Scale-Out-File-Storage

Dave submitted a project plan on September 15, 2010.

Emails --

Dave: I shared the project plan with Ben Rogers, HCIS, and he sent some feedback.

Ben:

1. Silicon Mechanics servers: 6GB of ram is likely too low if you go with OpenSolaris as performance improves substantially with larger amounts of ram for caching. Might not be as big of deal on RHEL but based on what we are seeing with Lustre on CentOS I think this is a bare minimum there as well.
2. I wasn’t aware of the Trash plugin, could you point me to more info on that (quick google didn’t reveal what I was looking for).
3. Usable storage – We’ve seen between 50 and 55TB of usable storage on the Silicon Mechanics boxes depending on the raid configuration. I think 55TB is about the max you’ll see, probably won’t make it to 58TB, though that’s not a huge difference I thought I’d let you know.
4. We now have Lustre running on the cluster over Infiniband if you are still interested in taking a look at it.

I assume you are aware at this point that OpenSolaris is a dead product. We aren’t sure yet what we’ll be replacing it with, likely Solaris Express Edition, Illumos, or Nexenta.

Dave: Another note from Ben. He mentioned the idea of some shared spare kits on campus, since multiple groups have Silicon Mechanics gear now.

Mark: Ah – I saw #1 on the project plan and forgot to point that out – the quote is for 24GB of ram in each box and not 6GB

3 – could you ask Ben what his raid layout is? Is he using raid 6?

Ben: Ahh, 24GB should be fine. We’ve gone with 48GB but that’s because we’ve been using compression and have considered deduplication.

We are using Raidz2 (raid 6) on our systems. With two raid groups each with 18 drives we see ~55TB and with three raids groups of 12 drives each we see ~51TB of available storage. We aren’t using any hot spares, we just have the cold spares on site.

Dave: I asked Ben who already has Silicon Mechanics hardware on campus, and who’s managing it:

“At this point the groups I know of that have Silicon Mechanics gear are as follows:

Engineering – Iowa Institute for Biomedical Imaging – 3 units
Engineering – IIHR – 10 units (I believe they are on order but haven’t arrived)
ICTS – 5 units (Partial Ownership by Psychiatry)
Neurology – 1 unit
DNA Core – 1 unit

“Systems are housed in a variety of locations. Management is being done by individual groups but we are assisting and groups are generally using our puppet management system.”

Work Machine - September 17, 2010

· Dell Latitude E6410 with Intel Core i5-540M (2.53 GHz, 3M cache) processor, 4 GB RAM

· Ships with Windows 7 32-bit, but you’d probably re-install with Windows 7 64-bit

· 14.1” screen

· Intel on-board graphics (upgraded graphics card would be $50)

· No modem

· 160 GB HD

· Add Bluetooth module

· No fingerprint reader

· Mobile broadband ready

· E-Port Plus docking station/port replicator

· Two 19-inch external monitors (Dell Professional P190S 19-inch HAS)

Tracking Quotas on NetApps

Mark is working on a Powershell script which will replace the vendor package, QFS from NTP Software.

I’ve now set a qtree tracking quota on every volume that has a qtree in it on itsnas01b – we will have to turn off quotas and back on again for the tracking quota to work so I will work through them all this week but going forward, if you need to add or change a quota on any of the qtree volumes, you can just hit resize and the changes should take effect immediately. Below are all the volumes I’m adding tracking quotas for:

* tree@/vol/shared1 - -

* tree@/vol/shared2 - -

* tree@/vol/shared3 - -

* tree@/vol/shared4 - -

* tree@/vol/shared5 - -

* tree@/vol/shared6 - -

* tree@/vol/IMFO - -

* tree@/vol/microbiology - -

* tree@/vol/library - -

The script goes as follows (9/10/2010) --

I’m putting together a howto in case we need to find a user who is using up too much space on their home directory:

below is how to do it in powershell using the netapp powershell toolkit – there is at least one caveat – the quota report is not available when the filer is initializing a new quota/volume/etc (this is true from within filerview as well). I’ve turned on the quotas on all but home1,3 and 5 so far – 3 is currently at 60% done. I will be turning on 5 when 3 is done and I will try to get to home1 tonight.

let me know if you have any questions

Download and install script are here:

http://communities.netapp.com/community/interfaces_and_tools/data_ontap_powershell_toolkit/data_ontap_powershell_toolkit_downloads?view=documents

once you have it installed:

PS E:\>connect-nacontroller itsnas01a

PS E:\>$q = get-naquotareport

PS E:\>$q | select-object quotausers,diskused,filesused,volume

PS E:\>$q | select-object quotausers,diskused,filesused,volume >> quotareport-09102010.txt

PS E:\> $q | ? {$_.quotausers[0].quotausername -eq "iowa\ltrapaneiii"}

Cannot index into a null array.

At line:1 char:23

+ $q | ? {$_.quotausers[ <<<<>

+ CategoryInfo : InvalidOperation: (0:Int32) [], RuntimeException

+ FullyQualifiedErrorId : NullArray

DiskLimit : -

DiskUsed : 0

FileLimit : -

FilesUsed : 1

Qtree :

QuotaTarget :

QuotaType : user

QuotaUsers : {IOWA\ltrapaneiii}

SoftDiskLimit : -

SoftFileLimit : -

Threshold : -

Vfiler :

Volume : home0

Cannot index into a null array.

At line:1 char:23

+ $q | ? {$_.quotausers[ <<<<>

+ CategoryInfo : InvalidOperation: (0:Int32) [], RuntimeException

+ FullyQualifiedErrorId : NullArray

VM Ware - Storage Team Meeting 9/13/2010

VMWorld Conference, Tom Kneese: Cloud Computing, VShell - firewalls on your individual NICs.

VSphere, Kevin Keyser: SEV1 Problem is believed to be only for 64-bit machines, scenarios VShpere 4.1 to 4.0 adn 4.1 to 4.1.

BLADE Technology: Thoughts to use that infrastructure before or after move to the new datacenter? BLADE, due to "powerbus" gives an estimated 33% savings.

ITS SecureFarm, Dave Bronder: Wanting some balance between secure and firewalled machines during build an maintenance. Too tight security control puts too many obstacles in the path of serviceability.

nfs exports on itsnas01b

mark: I made a change to the central web server exports (removed nfs3 and fixed a type on klieg) and when I did the exportfs –r I got an error on a storage export:

no such directory, /storage not exported

/storage pointed to /vol/storage_data which doesn’t exist and I’m guessing that was part of when dave created storage_depot and it was safe to delete – so I’ve removed it from the exports file.

Another thing I noticed – I did an exportfs to list the exports before doing the reload and there was this listed:

/linux -actual=/vol/shared5/linux,sec=sys,ro=itsesg01.its.uiowa.edu:itsesg04.its.uiowa.edu:itsesg03.its.uiowa.edu:itsesg02.its.uiowa.edu:itsesg05.its.uiowa.edu:itsesg06.its.uiowa.edu,root=itsesg01.its.uiowa.edu:itsesg04.its.uiowa.edu:itsesg03.its.uiowa.edu:itsesg02.its.uiowa.edu:itsesg05.its.uiowa.edu:itsesg06.its.uiowa.edu

But I don’t think /vol/shared5/linux exists and this export isn’t in the exports file

Anyone have any history on this one. It is exported to the SOFS cluster plus itsesg06

seth: We put the RHEL ISO on /vol/shared5/linux to install on all the IBM servers over NFS. I remember removing the share and the export after finishing so I’m not sure where that message came from. The file may have changed but the exportfs wasn’t executed

mark: Ah – ok

So you removed it from exports buy didn’t do a exportfs –r so the linux mount was still active.

It is gone now after I did the -r

NetApp LSA Move

Fyi, we discussed the October infrastructure day and the NetApp LSA move during the Enterprise Infrastructure LT meeting yesterday. It also came up during the monthly data centers conference call this morning.

Rex asked Pete and I to meet with the ITS LT on Tuesday afternoon to talk about the LSA move (Pete has been responsible for scheduling and communication for the infrastructure day). I expect that group to consider a few questions: can we tolerate the downtime now during the fall session; can we defer the move until January, when we're between fall and spring; is it worth the staff time to do the move more gradually?

I'm planning to schedule something for Tuesday morning so I'm prepped before the meeting with the LT that afternoon. I'll include a few questions in the appointment.

Meeting is at 10 AM on Sep. 7.

Q: Could we make the work transparent to VMware by using VMotion to move VMs between JH and LC?

A:

Not between the filers – we wouldn’t have enough space and would lose all the dedup savings and it would really mess up the snapshots. We could possibly do it from the itsnas01b to the lc-cx4 (if we had the space there) but again, we would really mess up the snapshots on the JHVM volumes. There would be issues around the fact that the windows vm’s no longer have BESR installed and so they wouldn’t have full vm recovery for however long they were off the NAS.

Q: How much time would that take (both in terms of effort, and in calendar time)?

A: a lot: new zoning, mounting new datastores on the clusters, formatting, storage vmotions – they would probably need to limit them to 2 to 4 at a time and let’s say they take an hour so that would be 63 hours at 4 at a time or 125 hours at 2 at a time (I just checked my sent mail – there was a problem with cpu spikes when they moved more than 2 at a time to the NAS – we may end up with the same problem moving them off)

Q: Any progress on the routing questions for the scenario where we move services between vifs? Is there a way to change the routing (and default gateway) without disrupting services? (Would we be confident enough to make that change live?)

A: I have a question in to netapp about the default route (sent yesterday). I’ve read a few conflicting support articles – one said you needed to bring the interfaces down in order to change the default route. Another I read said that routed probably wasn’t even needed anymore and could be turned off because of fast path. I’m still researching.

Group Meeting, September 1

Future SAN Team agendas: 9/13 FLARE 30

SAN Team is Unix (Pete Brokaw, Dave Bronder, One More), Windows (Guy Falsetti, Tom Kneese, Kevin Keyser), and IDDM (Bill Knight, John Kane, One More). It is a mid-tier for the Storage Group to advertise upgrades that the groups then propagate to their customers. Somestimes, it is needed to upgrade servers with software, administered by the Storage Group, and is then coordinated through this team.

• FLARE 30: Migration to Upgrade PowerPath, New Service Taskbar. Unix and Windows groups security concerns and vendor documentation is not good. The Storage Group uses the software (Control Agent) from EMC, but it needs to be installed by the groups due to admin authorization.

Outbound proxy

Set up a proxy to resolve outside links to upgrade expanded functions of Red Hat Linux, Fedora, for example. Use its own VM; Seth will use "Squid" to do it. Mark has already used it on his machine.

Infrastructure day

Mark: Move from 62 (CIFS) to 41 (VMWare) can be done on the fly and we should be able to do it soon.

EqualLogic disk upgrades

EqualLogic needs more disks -- Exchange has received more space, more than what we actually have currently, so if they wanted to use all of it, they couldn't.