Storage Wisdom!

Back in the day…

When I was a lad my elder brother and father were busy tinkering with the internals of a ZX Spectrum.  Unfortunately I was a little too young to be allowed into the same room as these amateur hackers (I would probably have done the annoying little brother thing of grabbing some ZX internals and running off to flush it down the loo).  The hackers enjoyed copying endless pages of code from magazines to perform tasks that only a hardened techie would get excited about.

I am now considerably older and only slightly more grown up, however I feel that I missed a key part of my computing education by not being able to fiddle in the fundamentals of a computer.  (The first hacking I did started on a 286 PC running DOS).

However all is not lost because a project based in Cambridge, UK is building a cheap and small computer that is designed for people to fiddle, furtle and poke it. The project is run by the Raspberry Pi foundation and the stated aim is to:

“…see cheap, accessible, programmable computers everywhere; we actively encourage other companies to clone what we’re doing. We want to break the paradigm where without spending hundreds of pounds on a PC, families can’t use the internet. We want owning a truly personal computer to be normal for children.”

So the aim of the project was to instil the enthusiasm for computing in the “up and coming“ generation.  There has been a huge hoo-ha here in the UK recently with people questioning the value of ICT lessons in schools.  The main issue being that the curriculum is too focused on using Office products rather than the fundamentals of computer science.

The hardware consists of the basic board (about the size of a credit card) with the chip (with RAM stacked on the top), SD slot for storage and OS, USB, HDMI, Power connector, RJ45 network port, Audio jack and a few connector ports.  The aim is that enthusiasts build add-on boards that link to the basic board to extend the functionality.  One of the Raspberry Pi team has already developed a board for operating motors. (http://www.raspberrypi.org/archives/500)

So what would you use it for?

Well other than it being fun to play with you could use it for a tiny media PC.  There is an example of the Pi being used as an AirPlay receiver! (http://www.raspberrypi.org/archives/564).  If you were doing the honourable education thing then it should become the core of some interesting school computing projects.  But make the projects interesting, some examples are:

- Parent warning system (pressure sensors in the landing carpet)
- Bedroom alarm system for unauthorised access to off limit areas
- Smelly breath detector for preventing those awkward discotheque moments
- Mobile DJ rig

But can I afford to have this toy?

So when I went home one evening and told my wife that I knew what my next gadget purchase would be, she rolled her eyes and imagined a 6 month delay to the kitchen re-fit.  However when I told her the price she smiled in a “whatever makes you happy” way.  The price is a staggeringly low $25 for the 128MB RAM model and $35 for the 256MB model! 

So hats off to the Cambridge boffins… go and get a slice of the Pi and support a good educational cause!

check out – http://www.raspberrypi.org/

Flash – AAAAH   aaaah…

 So EMC have released the first of their flash only arrays!  The VNX5500-F is based on the standard VNX chassis hardware but has a load of SSD drives shoved up it.

 The architecture is similar to the IBM STorwize v7000 model of disk system for block and additional “servers” to provide the file protocols.  EMC call the file modules, data movers (alternatively called X-Blade enclosures).

 EMC provide 3.5″ or 2.5″ SSD options for the VNX5500-F.  With the 2.5′ version you can install 25 x 100GB or 200GB SAS SSD’s.  The max number of trays you can attach is 15, meaning you can have 50TB of capacity (46.5TB of formatted storage).

 This capacity range seems to match the other “flash only” products such as the FA-310 from Pure Storage and the Violin 3100 series.  I am sure there would be lots of arguments from these flash purists that they would wipe the floor with the EMC VNX5500-F however EMC’s name does a lot to overcome these misgivings.

 The great thing with the VNX5500-F is that you can actually intermix all the other drive types that you can put in a standard VNX (SAS 10k, 15k and Near-line).  This mix of storage will then allow the tiering of storage as you would typically with an intermixed VNX.

 The other grand thing over some of the other smaller flash vendors is the functionality you get with the VNX such as, Block and File protocols, NDMP, Automatic tiering etc etc.

The future is flash…

 Anyway I am sure that this is the start of the trend that will become the norm. SSD will be the place your production data lives and the array will dynamically move your data through the tiers and then out to tape and into the vault.  A few years off but a very realistic prospect.

October was a busy month for announcements for IBM with the SVC 6.3, v7000 Unified, and TSM Flashcopy Manager (FCM) 3.1 amongst the announcements.  We have mentioned the v7000 unified previously but haven’t covered the FCM announcement.

Flashcopy Manager…

… is a software product in the Tivoli brand that allows you to manage snapshots of your IBM storage (and windows VSS snaps on any hardware) from one console.  It allows you to optionally perform a backup to TSM of the resulting hardware snaphost.  All of this functionality is performed through one interface.

What is in FCM Version 3.1

In the announcement in October IBM included the following new features:

• Advanced data protection and recovery features for VMware vSphere environments
• Enhanced data protection capabilities for Microsoft® Windows®, including support for New Technology File Systems (NTFS) and custom applications, and enhanced user interfaces for Microsoft Exchange and Microsoft SQL Server
• Support for IBM DB2® and Oracle databases (with or without SAP environments) on IBM AIX®, Solaris SPARC, Linux® x64, and HP-UX IA64 platforms
• Support for custom business-critical applications on IBM AIX, Solaris SPARC, Linux x64, HP-UX IA64, and Microsoft Windows platforms
• Transparent integration with IBM storage systems such as IBM System Storage® SAN Volume Controller space efficient FlashCopy target volumes, IBM Storwize® V7000, IBM XIV® Storage System, and IBM System Storage DS8000™
• Can leverage the Microsoft Volume Shadow Copy Services (VSS) framework for integration with non-IBM hardware subsystems
• Database cloning support for UNIX® and Linux clients

The diagram below shows the process of using hardware snapshots and TSM for VE:

In this first diagram we see the VMWare environment, FCM Linux proxy (either VM or Physical) and the TSM server.

The next diagram show the FCM proxy initiating VMWare software snapshots of the vmdk files that are owned by the VMs that we are protecting.  This process is using the standard vStorage API calls that are used by other products such as TSM for VE.

The third diagram shows the FCM software initiating a v7000 hardware snapshot (flashcopy) and removing the VMWare software snapshots so that the performance of the VM is not impacted during the backup.

The line from the snapshot through the FCM proxy to the TSM server shows the backup of the resulting snapshot into TSM.  The format of the backup into TSM uses the Common Data Format, this means that the data in TSM is in the structure that allows you to use the TSM for VE Recovery Agent.  This in turn means that you are able to recover individual files from a hardware snapshot backup that is stored in TSM!

At the moment the following configurations / restrictions apply:

• At the moment the following configurations / restrictions apply:
• FCM proxy is Linux only (RHEL or SUSE) – later will be Windows too
• If the FCM proxy is a VM, it can’t be on a datastore that is being backed up! (surprise)
• No SAN attachment required to the FCM proxy
• RDM and iSCSI attached volumes not supported
• When VMs span data stores the underlying disk system needs to be the same

Money Money Money!
So you are now saying to yourself “all well and good, but that is gonna cost!”.  Well actually if you purchase a IBM Storwize v7000 you can configure it to come with FCM.  In this configuration you can access a bigger discount than if you were purchasing a native v7000!  This incentive means that all v7000 quotes that I have seen recently have actually got this FCM licence bundled in.

Another reason to go get yourself a v7000 for xmas!
For more info:

Announcement:  http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS211-363&appname=USN

 Rapid Application Storage: http://public.dhe.ibm.com/common/ssi/ecm/en/tsd03121usen/TSD03121USEN.PDF

Network Attached Storage is the NA(D)S!

So we all think it is a jolly good idea to take our file servers and migrate them to a NAS device.  We get the resilience of our file servers being on a dedicated storage appliance with failover components and RAID.  We are able to consolidate multiple fileservers into one appliance and so gain ease of management and administration.

Backup is always an afterthought

The one problem with consolidating file data onto a NAS is when it comes to backups!  Previously the backup agent on the fileserver scanned the files and backed them up to the backup server.  This may have taken ages because of the scan process, however you could at least take an incremental backup.

With a NAS you can’t put the backup agent onto the appliance hardware and so you are stuck with a number of different options:

1. CIFS / NFS mount the drives to a proxy.

With this option you are using a proxy server as the medium to get the data into the backup server.  You would install your backup agent onto the proxy server and then mount the shares that need protecting onto the proxy.

PROs – You are now able to do a progressive incremental backup (with TSM) or standard Grandfather-father-son backups using the standard backup agent.

CONs – You are dragging the data over the network twice.  The first hop is going from the NAS to the proxy and the second from the proxy to the backup server.  This could increase the backup time beyond the allowed window.  The other disadvantage is that the file systems are often sizable and the scan of the file system can be a very lengthy process.

2. NDMP backups

NDMP is a protocol that is implemented by the NAS vendors and allows a backup to be taken directly from the NAS to the backup server.  This backup can either be over the network to the backup server or other the SAN directly to tape.  The list of files that is included in the backup is transferred in a Table of Contents (TOC) and is stored alongside the data.  The backup is a block based backup and is very rudimentary.

A typical implementation would see a NAS being backed up using a Full backup every weekend and during the week differential (all changes since the last full) backups would be taken.

PROs – no additional server hardware is needed, backups direct to tape can occur, file and full recoveries are available.

CONs – NDMP is clunky and very simplistic, if you think about the size of your NAS, you have to perform a full backup of that during a weekend.  Let’s take an example of a 15TB NAS.  If you want to retain 4 weekly full’s and 12 months of monthly full’s you will need 240TB of backup storage (and capacity licence) on your backup server!  We then come to the recoveries, Full volume recoveries are great as it is a full block based recovery of your data.  File recovery (the one you are most likely to want) means that the TOC needs to be read first before the actual restore can be started.  essentially you are reading a file and folder structure that may have millions of objects in before initiating the restore of the single file you wanted.  This process I have seen run for the best part of a day!

3. 3rd Party integration

NetApp allow backup products to use the differential analysis of the changed data without using the NDMP protocol.  This SNAPDIFF function means that you need a proxy server with CIFS mount of the NAS shares and the standard backup agent set to know which file systems are on the NAS.  The Backup agent also needs to have stored the login details to the NAS so that it can logon and initial a snap.

PROs – brilliant for reducing the backup traffic for very large file volumes.  With this process not as many full backups are required.

CONs – You have to have a Netapp or IBM nSeries NAS

4. No traditional backup – use the force (of the NAS)

This is an option that a good number of people are taking when faced with the problem of how to protect a NAS.  Most NAS devices will have a snap-shotting ability.  These snapshots will be space efficient as they will only track the changes to the data, original data will be accessed via pointers.  The data in a NAS will also be replicated to a second site, and at this site too, snapshots of the data can be taken.  Some NAS vendors allow you to have different snapshot policies on the production and DR NAS heads.  This allows you to keep hourly snaps of the data in production and then daily, weekly, monthly snaps on the DR site.

PROs – Very space efficient, only the block level changes between snaps will be occupying space on the storage.  Very fast recovery, many NAS vendors integrate with VSS so the backups can be accessed by the user by right clicking on the file and looking at the older versions that are available.  Full recovery of volumes is possible using the replicated copy of the data.  You don’t clog up your backup system with repeated full backups of huge chucks of data.

CONs – Your primary storage and backups are in the same place!  You never really get a full copy of the data at a point in time (as the data is partly changed blocks and partly pointers).  You never get the data out to tape (sometimes this is a requirement).  You are locked into the NAS vendor tighter as your primary storage and backups reside within the one solution.

SUMMARY

The primary advantages of NAS’s are all well and good, however you need to consider how to protect the data before you decide on which one to go for.  Points to mull over:

1. How is the new backup method going to affect my backup licensing?
2. How will my backup window be effected?
3. What level of protection do I need (file, full, DR replica)?
4. What retention do I need on my backups (do I really need 125 million years of backups)?
5. Who do I want to perform restores? the user, the admins?

I personally would look to use the functions of the NAS as the first line of defence.  Using the snap-shotting tools gives some brilliant recovery advantages.  I would only look to NDMP as a DR “backs against the wall” recovery method.

IBM have gone all unified!!

In October this year IBM made a huge leap forward in the clash of the unified titans.  Netapp and EMC have already got Unified Storage systems (systems that provide file and block protocols), but IBM have now released the IBM Storwize v7000 Unified.

Recap!
The v7000 was essentially a 2 node IBM SVC plus the disk all in one box.  This was IBM’s attempt at stealing some of the mid market storage sales.  In my opinion the initial release (while lovely and compact and having a brilliant GUI) lacked a few key ingredients.  One of these was proper IP replication.  The majority of these original failings have been ironed out, but still the v7000 lacked the killer file protocol punch.

All unified and lovely
Well now it is there.  The v7000 Unified can provide FC, iSCSI, NFS, CIFS,  HTTPS and SCP protocols.  In addition to this, it isn’t like the usual IBM solutions where you have to log onto 13 different portals to operate it.  the functionality is provided using the updated v7000 software.  A GUI example of this is shown below:

As well as adding the file protocol goodness, IBM have also lobbed in ILM features to allow you to migrate your file data between tiers of storage based on the frequency of file use.  Again this is all driven through the GUI to build your migration policies.  This feature is based on the IBM Active Cloud engine, it allows the file data to be moved between tiers and even written out to TSM as the last tier – HOW COOL IS THAT!!  All of this is configured, yes you guessed it, in the GUI.  I think IBM should go and lie down.

The physical features of the Storwize v7000 Unified are the same as the standard Storwize v7000 but with some additional NAS head type boxes that are connected to the disk controllers using FC and Ethernet cables.  These additional boxes are called the v7000 Unified File Modules.

Some other additional features of the IBM Storwize v7000 Unified:

- Antivirus Software Integration – v7000 Unified is certified for use with the Symantec and McAfee ISV antivirus applications
- File replication – async replication for protection of NAS data across sites
- File level snapshots (up to 256 snapshots per file system).  This allows the recovery of data at a file level!

A practical application of the Storwize v7000 Unified:

QUESTION: Have you got your home directories or file share on a windows server that is a pain to backup and has loads of old data on?  Well all you need to do is:

- Buy an IBM Storwize v7000 Unified
- Migrate your CIFS share data onto it
- Setup ILM policies to migrate data older than 90 days to cheap storage (could be externally virtualised storage)
- Setup ILM policies to migrate data older than 6 months to TSM
- Write a report to tell your boss how good you are
- receive a 110% pay rise…

I must say IBM, you have excelled yourselves!

(screen grabs taken from https://www.ibm.com/developerworks/mydeveloperworks/blogs/ibmnas/entry/marriage_of_san_nas?lang=en with thanks)

Progress!!

So IBM has finally improved the electronic offsiting function with their TSM Server 6.3 release.

For those of you who don’t know the previous method of replicating data electronically from one TSM server to another, it was called virtual volumes.  This method allowed volumes from one storage pool to be sent to another server as an archive blob.  The second server had no concept of what is in the blob and so can’t look inside it and restore files.  To do a restore you had to restore the original TSM server then access the copies stored on the DR server.

IBM have improved this greatly with a new feature to replicate an individual node or group of nodes from one server to another.  Once at the destination server the data can be accessed and can be used for recoveries without the recovery of the original TSM server.

some of the features listed by IBM are:

True incremental replication-
      Only replicates directories and files that do not exist on target server
Deletes data on target server that has been deleted on the source server
Can recover client data directly from hot standby server
Can use with or without deduplication
Can have multiple servers replicate to one server
Remote vaulting without manual tape transfer
Efficient use of bandwidth through deduplicated replication
Allows hot standby at remote site
For the more technical amongst you the basic setup steps are:

1. Enable source server to target server communication (basic server-to-server comms)
2. Set target server as default target for replication (use new SET REPLSERVER command)
3. Enable nodes to replicate and modify default and filespace replication rules if needed (on the reg node command set the REPLSTATE=ENABLED and REPLMODE=SENDSYNC)
4. Set storage hierarchy and domain policies
5. At proper time, preview and initiate replication for desired nodes (REPLICATE NODE)

A real life application

So where could you use this new feature to great effect.  Well, it is now possible to replicate data between sites in a deduped format meaning that bandwidth can be reduced by using TSM server-side dedupe and this new replication feature.

This is a great, cheap way of achieving “VTL like” operations but using the TSM server code only.  With the TSM 6.3 server being able to handle 4 billion objects it helps with holding the hash table in the database for reasonably sized TSM servers too.

Summary
So now with 2x TSM servers and 2x cheap disk systems you can get the equivalent of a poor man’s  VTL!

huzzar!

Back 2 Basics

I thought I would give you a summary of the session I did at the Silverstring Back 2 Basics event last week.  The title of my session was:

Survival of the fittest - Is your data protection system evolving quickly enough?

The theme was to emphasise the massive advances that have been made in data centre technology in the past 5 years.  With the mass take-up of server virtualisation technology the estate that needs to be protected by our backup infrastructure has changed drastically.

One slide that I used to illustrate this rapid evolution is shown below:

The stats on the level of workload that will be using virtualisation are staggering.  60% of server workload will be in virtualised environments in 2013. That is a huge growth from 25% today.

The problem then is the increasing divergence between our slowly evolving backup environments and our rapidly developing front line services. 

Most of us are still backing up these virtualised servers as if they where physical assets.  Gartner states:

 The reason for this slow development is not due to the backup products not having the features available to us.  It is because backup isn’t sexy and doesn’t get the focus needed.   However, we are where we are, so what can we do about it?

Accelerate evolution of data protection
We need to rapidly evolve our data protection systems to bring them in line with the estates that we are now trying to protect.  To do this we need to apply the basic principles of genetic engineering to breed the traits we need into our data protection systems.  This will accelerate the development of our data protection systems.

So what gene pools do we have available to us?

The pools that we have available to choose from are shown in the slide above:

- Storage – protection driven by the storage hardware so offloading the workload, shorter retention is generally seen at this level.
- Hypervisor – data protection tasks are consistent with the hypervisor tasks and agnostic to the backend storage
- Backup – longer retention possible and versatility of recovery types (file level and full VM recovery)

To decide which of these genes to encourage and develop we would do well to use the IDC advice as a guiding principle:

Most people have made a strategic decision on what hypervisor technology and what backup product they use.  So following the IDC advice of creating a framework that can cope with physical and virtual backups, we may as well develop the genes from the products that we have already chosen.  It seems that we are more likely to get long term savings if we lock into the hypervisor and backup technology rather than the physical storage.

The physical storage layer needs to be able to change and grow, as a result it is not a good idea to lock in a data protection strategy to storage hardware that may only last a few years.  It is wiser that we push the data protection up the stack to the hypervisor and the backup products that are going to be around a lot longer.

Summary:
- Primary systems are evolving rapidly
- Data Protection is not altering in line with primary systems
- Need to genetically engineer data protection to accelerate improvements to bring data protection up to speed
- The Hypervisor and backup product are the smart places to spend investment as the storage layer should be changeable and abstracted from data protection.
- This approach will stand our data protection systems in good stead for the jump to the cloud!

Well it has been a mighty long time since my last post but thought I would shamelessly promote the event that I am speaking at on Thursday 29th Sept 2011.
We are running an event that will be focussed on getting the best out of the decisions that you have already made surrounding data protection systems.
We aim to highlight that the effects of virtualisation have rendered the old way of backing up useless and that the data protection should evolve in step with frontline services.
However it isn’t all bad news as we will be highlighting the main areas of focus for making over your data protection systems in the cheapest possible way.
the official agenda is:

Why use a VTL – recap!

As you would have read in previous posts I have made.  The main reasons for using a VTL in a TSM environment is for one of two reasons:

- Software features above those of native disk e.g dedupe, compression
- Dedupe aware replication!

From the Rut of dispair!

SEPATON had a stuttering start into the world of depupe and replication.  The initial TSM replication was called Site2 and had a very small take up in the market.  The initial stab at dedupe had wonderful intentions of being content aware and maintaining the most recent copy intact (forward referencing).  However these marvellous intentions were not matched in the delivery (unfortunately).

A few years on and a number of developments have occurred.  The version 6 of the SEPATON software came out with a new way of doing the dedupe.  The old content aware “PairWise” method is out of the window and the new “Armstrong” method is in.

The Armstrong dedupe method is back to how dedupe should be done (in my humble opinion).  The data hitting the VTL is split into 7MB chunks for a finger printing process which helps the later levels of analysis.  The next level is a core differencing which uses a rolling 256 byte window to identify the actual dedupe candidate data.  The Armstrong dedupe is still forward referencing so that much remains the same.

This Armstrong dedupe sounds very similar to other vendors methods of dedupe but I am not complaining as it is a big improvement!

Mini VTL – The DS3 range

Another SEPATON development is the DS3 range which is a mini departmental size VTL.  This range uses different hardware to the ES range (which now uses HDS AMS 2100′s having ditched the Engenio and Dothill arrays).  The DS3 is a single chassis that includes the SRE node (the bit with the VTL software on) and the disks (I believe Xyratex disks are used).

I think the DS3 range will be available with either 1 TB drives or 2TB drives, performance or capacity will obviously determine the model of choice!  I also think that there is a model with 10Gb Ethernet too (I assume this is for CIFS presentation of the storage rather than just traditional FC tape).

Data Domain – watch out!
So SEPATON have lifted themselves out of a bit of a rut and have come up with something to battle with the Data Domain chaps!  let battle commence!

A couple of articles ago I commented on Object –based storage systems and tried to outline what they are and what they would be used for.  I have been to a session today at the IBM Systems Technical University entitled:

Object Based Storage: The next big thing for cloud storage?

The session was an interesting technology overview of the concept of Object-based storage.  I want to list a couple of points of interest that came out of the session above and beyond the article I put together previously.

Security.

Because Object-Based storage has all the meta data embedded in the object it is possible to implement fine grain security models where only that particular object will be able to be accessed.  This means that the old ACLs that need to be maintained and managed with NFS and CIFS will not be necessary (These ACLs would be a pain to administer in the hugely multi tenanted environments like cloud storage).

Attributes of an Object:

As attributes belong to an object itself it is possible to have a standard set of attributes on an object (such as size, type, creation date etc) and then extend these attributes specially to the payload of the object.  So for instance if it was a video you could create an attribute to state what the bit rate was and how to read the file when streaming (not retrying failed bits etc etc).

These attributes could also determine the QoS that the object gets and the priority of access over other objects.  Attributes could be created determining the number of copies for the system to keep for data security reasons.  As you can see if you can relate the attributes on the storage to the type of data you can think of loads of possible benefits.

Data Optimisation

One other advantage of object based storage that relates to the fact that the storage system knows the context of the data it is managing is that it can optimise for performance.  For instance if you know that 10 blocks on the disk all relate to a particular object you can located them contiguously to prevent performance of the blocks being fragmented.  Performance optimisation can also be conceived for prefetch routines too as the storage controller can fill the cache with all blocks relating to the read request for an object.

IBM futures

So what are IBM doing about it as they are very lacking in the field at the moment.  Well I should think that IBM’s object-based storage will make its first appearance in the SONAS product which will essentially be a REST and SOAP interpretation for GPFS which is used by this product.  Once this is out I wouldn’t think it would be too long before the converging storage portfolio will be able to access the same code (XIV, v7000, SVC).  We will have to see over the next year.