Storage Management Automation

Recap!

Back in TSM 6.1 server side deduplication became available.  The data had to reside on disk to gain this benefit (obviously) and the dedupe process was post process (meaning it wasn’t performed at time of ingest).  This functionality works very well in the right circumstance and storage benefits can be gained.  IBM in version 6.2 have added a client side dedupe that extends the functionality of the server side dedupe.

Dedupe on the client!?

If you are technical, reading the IBM explanation will make you sit up and then mutter “gosh that’s clever”.  If you are not technical you will recognize the features usefulness.

Firstly what is client side dedupe?  It is the ability for the TSM client to identify chunks of data that have been sent to the TSM server previously and so not send them again.  The result being greatly reduced backup traffic which in turn reduces backup windows.

In order to implement the new functionality of client side dedupe you will need to be running version 6.2 server and client as the client dedupe is an extension of the server dedupe code.  This means that the TSM client API has been updated to cater for the new exchange of data hash queries.

So here is a simplified explanation of how it works:

1. Client inspects file to backup and chunks it up.

2. Client queries the TSM server to see if any of the chunks already exist in the server hash table

3. If it does exist then a local cache of the hashes will be built on the client to speed up the queries in the future

4. The new chunks plus hashes for the old chunks will be sent to the server.

As can be seen by the process above the dedupe for the client end is in-line so reducing the landing space needed on the disk pool at the server end.  Another funky feature is that client side and server side deduplcation share the same hash table, or pool of data chunks.  So the benefit being that if some nodes reside on slow network links they can share the chunks that have been populated by nodes that are local to the TSM server.  The explanation above mentions a local cache of the hash tables this is another attempt at reducing network traffic by preventing the process being too chatty.  Hash queries will first go to the local cache and if they are not satisfied they will be forwarded to the TSM server.

What about my API clients I hear you cry!?!  Well you can get the advantages for agents that use the TSM API however there are a few caveats when using TDP agents that haven’t been updated to understand what the API is doing to it’s data.  It is best to test it on development system or if all else fails read the manual!

I will leave you with a table that summaries the server side and client side deduplication methods:

 …mutter, mutter, mutter, those clever IBMers, mutter, mutter…

The event at IBM Bedfont went well today, thanks to all for supporting us.  The day was essentially to layout Silverstring’s RAP framework for Storage Management Automation(SMA). 

RAP stand for Reduce, Automate and Protect.  We figure these are the three results people look for from a successful SMA engagement.  For instance under the “Reduce” element we will put TPC data audits for identifying archivable data and dedupe for reducing primary and backup storage.  For “Automation” we will discuss the PREDATAR suite for automating TSM management.  Finally for “Protection” we will discuss our automated TSM recovery tool (PREDATAR Recovery Tracker) and consult offerings such as TSM Health checks and Recovery Audits.  So enough of the ramble, basically we based the day around the RAP theme.

We used the “Reduce” section to cover the official launch of the TSMagic product (we didn’t get the same cheers and clapping Steve Jobs gets, how does he do it!?).  It was nice letting the product out into the open for general inspection and prodding, we got some good interest in the product that is very encouraging! (phew, we didn’t waste months of development).

Ronnie De Giorgio covered the “Automation” section of RAP by giving us a comprehensive run through of the PREDATAR product set.  He gave an insight into the future of PREDATAR wearing his hat as PREDATAR Development Manager.

The final section of RAP, “Protect” was covered with one of the products from the security and encryption giant THALES.  Simon Taylor presented on the TEMS key management appliance that will allow a fully secured and offsited method of implementing key management for tape and disk products.

If anyone wants the materials that we used or other marketing bumf on the products then email me - ldavenport@silverstring.com

All in all a tiring but enjoyable day that will hopefully lead to some business…

Just to give you all an update on the developments of the TSMagic product.  I have been working on the marketing material and doing webinars on the product.  Things are shaping up we and we have a good set of marketing collateral.

I am also putting together the presentation material for the launch day at IBM Bedfont on June 10^th. The agenda for the day is below:

I will be presenting in the 11:00am slot on TSMagic.

If you want more information then drop me a mail (ldavenport@silverstring.com)

Life the Universe and TSMagic!

Life has been crazy busy over the past few months, in a good way though.  I have been working with the software guys here at Silverstring (the PREDATAR team) on our new product, TSMagic.  This new PREDATAR module is really rather fun!

Firstly I will tell you what the software does and then go onto explain the consultancy spin that we have put on it.

The Software

So imagine being asked the question:

“how many versions do we keep of the file quantum_bananas.pdf in TSM?”

Or

“TSM person, tell me the percentage of data in TSM that hasn’t been read in the last year”

These specific questions may not crop up regularly but you will almost definitely get questions about what data is in TSM, how many version are being kept and how much a particular application is occupying.  TSMagic uses the TSM database and gives a visual representation of the contents of usage of TSM storage.  So for example the screenshot below is one of the highlevel views that shows the breakdown of the backup data within a TSM system:

Breakdown of the file types using TSMagic

The particular designation of data types comes as a number of defaults that can be altered and added to using a nice easy GUI driven wizard.

Another couple of Screenies:

Breakdown of space occupied by the different application types

Split of application data types (SQL, Oracle, Exchange etc)

A useful Storage Resource Management type report showing the age of the data that is resident in TSM.

The Consultancy Spin! 

It is all very well having access to a load of pretty pictures, but what are you going to do about it!?  That is where the work that I have been doing comes into play.  Rather than deliver TSMagic as a software product, we have built it into a consultancy offering that uses the data to give a view of the following:

• Capacity - data and facts about the current capacity and storage use of TSM and where savings can be made. Our experience of the tests we have performed is that massive savings can be made by identifying all the unnecessary backups that occupy space that would never usually get spotted. One instance we saw a 25% storage saving. This was all unnecessary backups such as backing up the disk based TSM DB backups, back into TSM (eh, good idea!!).
• Compliance - we use TSMagic to check the required version and retention against what is really happening. This is one better than the usual test which is to check that TSM is configured to have the correct policies. The usual tests completely ignore if the right data is bound to those policies.
• Archiving - Using the SRM type reporting we can give a view on the age and profile of the types of data in TSM which gives a very good indication of the types of data on primary storage. Who cares!? I hear you cry! Well it gives you a very quick and accurate picture of what primary storage can be freed up by archiving unused data!
• TSM 6 readiness. Using the data type breakdown we can give a view of the level of dedupe that can be achieved in your TSM 6 environment. Usually this figure is based on a “finger in the air” estimate and not actual data from your own environment.

So the plan with the TSMagic Consultancy is to present, in an innovative way, the reductions in cost by managing TSM storage wisely and the reductions in compliance penalty payments by being able to rapidly identify and fix compliance issues.

The launch is on the 10^th June…

…to be continued

Tivoli User Group

The other week I presented at the Tivoli User Group on the

 subject of Upgrading to TSM 6.1.  I thought I would share the summary of what I had to say.

One of the main things I emphasised was the importance of preparation for upgrading.  I covered three aspects of the preparation needed:

1. DB and Log sizing
2. Cleaning your version 5 database
3. SQL Scripts

I will outline what I said in each section:

1. DB and Log sizing

The main point that that I wanted to get across here is that under sizing your log and DB will end in much pain!  I did a whiteboard session to describe the rules and gotchas for DB and Log sizing (I have typed up most of the session in the picture below)

TSM 6.1 whiteboard session

2. Cleaning your version 5 database

This is an optional step in the process but will speed up the upgrade process if performed.  The advice is to carry out the following steps to get your version 5 database clean before the upgrade:

• Estimate Dbreorgstats
• If significant savings are found then proceed with UNLOADDB / LOADDB
• UNLOADDB
• LOADFORMAT
• LOADDB
• AUDITDB

Some of the experiences of those in the audience stated that the cleanup operations knocked about 6 hours off an upgrade process.

3. SQL Scripts

There are a number of syntax changes to the SQL statements run from within TSM now.  These are detailed in the TSM 6.1 Upgrade Guide.  One example is below:

 “Like” changes to “in” when using nested SQL statements

select * from volumeusage where volume_name like (select distinct volume_name from volumeusage where node_name=’node1′)

Replace such usage with the in parameter, as in this statement:

select * from volumeusage where volume_name in (select distinct volume_name from volumeusage where node_name=’node1′)

I think during the presentation I managed to emphasised the critical nature of these preparation tasks and hopefully scared the crowd into thinking long and hard about when and how to move to TSM 6.1

Resources:

TSM 6.1 Admin and Upgrade Guides:
http://publib.boulder.ibm.com/infocenter/tsminfo/v6/index.jsp

TSM 6.1 Technical Guide (Redbook, draft at the moment)
http://www.redbooks.ibm.com/redpieces/abstracts/sg247718.html

Technote for Log Sizing:
http://tinyurl.com/ygj36js 

Fingers Crossed!

Once you have installed Tivoli Storage Productivity Center (TPC) a few times you are used to holding your breath and crossing your fingers in order to help the install go through error free.  However there are a few more scientific ways of helping the install complete successfully.

For those that are new to TPC there are several components that need to be installed in the right order.  These components and the order of install are:

1. DB2 database server
2. Agent Manager
3.  TPC DB Schema
4. TPC server and client components

The most painful step of the install in my experience is the second step, the installation of the Agent Manger.

Below are several tips to ease the installation process:

1. Agent manager fails to install with return code 2 and no other information

Often the installation of the Agent Manager will begin then bomb out with a very vague message such as a “Return Code 2″.  I am not sure what causes this but it has always occurred when logged into the TPC server as a domain admin.  The solution has always been to login as a local admin (db2admin will work as it gets created during DB2 install).

2. Agent manager install on a 64bit DB2 database server.

The Agent manager supports both 32-bit and 64-bit DB2 database server.  However the install wizard for the Agent Manager needs some spoon feeding to install on the 64-bit version.  If you just select the option in the wizard to install onto a 64-bit DB2 server you will get an error stating that the DB2 database on localhost is not responding.  The solution is to manually create the IBMCDB database manually before running the wizard.  The process is described in the IBM manuals:

http://publib.boulder.ibm.com/infocenter/tivihelp/v4r1/index.jsp?topic=/com.ibm.tpc_V41.doc/fqz0_t_installing_the_agent_manager_64bit_DB2_unix_gui.html

3. Install of TPC 4.1.1 DB schema fails with UTIL_HEAP_SZ errors

I have experienced a problem when installing the TPC database schema (step 3 in the process) where the install fails and there are messages in the <TPC install location>\log\dbSchema\install\dbSchemaInstall.log mentioning that UTIL_HEAP_SZ has run out of space.

UTIL_HEAP_SZ is a DB2 parameter.  This parameter indicates the maximum amount of memory that can be used simultaneously by the BACKUP, RESTORE, and LOAD utilities.

The solution that I found was to increase this parameter while the install process was underway (you can’t do it before hand as the TPCDB database doesn’t exist).  Below are the steps I used:

1. Start the TPC setup wizard and select a custom install and only install the DB Schema

2. While the install is running perform the following steps

3. Open the DB2 “Command Line Processor” This sets the environment for your command session.

4. Quit out of the “Command Line Processor” but keep the window open.

5. Enter the following commands:

db2 connect to tpcdb user <db2 username eg db2admin> using <password>

db2 update db cfg for tpcdb using util_heap_sz 524288

6. You can check that the setting has taken by using:

db2 get db cfg for tpcdb

7. Keep checking the parameter during the install because sometimes the setting seems to get changed automatically.

Hope someone finds these tips useful.  Happy installing!

Solutions for backing up virtualised server environments often leave a lot to be desired.  MS Hyper-V’s offering seems to be several years behind the rest and even the market leader (VMWare) has been lacking with its VMWare Consolidated Backup (VCB) framework.

Much Clunkiness!

For those that don’t know VCB is literally a bunch of command line executables and scripts that allows you access to the VMFS file system in various different modes.  There is no GUI and nothing pretty about it.  When 3^rd party products integrate with VCB there is still a good deal of clunkiness that is experienced.  VMWare users have often had to rely on specifically written applications such as vRanger, esXpress and Veeam to provide a GUI interface and scheduling of the VMWare backups.

So what does  vSphere provide?

 VMWare vSphere is VMWare’s latest and biggest upgrade of their Virtualisation platform for a good while.  With many features that will wow techies around the world.  One addition for backup bods is that they have introduced VMWare Data Recovery and will phase out VCB.  So if you are implementing a new backup solution for VMWare remember that VCB has a limited life.

VDR then, what is that all about?  Well VMWare have finally got round to spending some time on the Backup side of things and given us a native VMWare GUI.  It comes in the form of a virtual appliance (OVF) that you install on the Virtual Center server.  Once installed the plug-in is then visible from the vSphere client.

What features do you get?

•  GUI integrated in vCenter
• Wizard for creating backup jobs
• Backup to disk only
• Restore of individual files and entire images of a VM
• Understands backing up VMs that are migrated and will continue backups during a migration
• Full and incremental backups
• Deduplication - only the changes are backed up.  This means that many “full” backups can be kept as the are really just virtual checkpoints and not multiple copies of the files.
• Compatible with any storage that ESX supports- Fibre, iSCSI, NAS, or local

The main question that I had when I heard about VMWare Data Recovery was “how do you get the backups to tape?”.  The answer is “you don’t!”.  I think this is a major issue that will prevent many people using it for long term historical backups.  Where I can see an advantage is for keeping small retention of backups for fast restore of recent files and full VM recovery.

There are several places on the web that are focused on VMWare that have a lot more information than I have been able to disseminate here.  One of these is:

http://www.virtualizationadmin.com/articles-tutorials/vmware-esx-articles/backup-recovery/vmware-data-recovery-part1.html

 A useful YouTude video that shows the process:

 I am not convinced that VMWare have solved the VM backup conundrum with VDR but at least it looks prettier!  What do you think?

Continuing the series of articles about TSM 6.1, we will consider in this article the methods of upgrading from TSM version 5.

Upgrade or Import?

There are several methods that are provided by IBM to upgrade the TSM server, which we will discuss in a moment.  The first thing to consider is whether or not you actually need to do an upgrade before you go down that route.  As discussed previously we know that TSM 6.1 is a major upgrade and that you are likely to be investing in new hardware to satisfy the requirements.  I see this as an ideal opportunity to consider any design changes needed.  For instance do you want to use the new active data pools offsiting process for fast recovery in a DR?

If some design changes are desired then you have the option of building yourself a fresh TSM 6.1 environment and importing historic backup data directly from the old version 5 TSM server (using the EXPORT NODE and IMPORT NODE commands).  This gives several advantages:

• New design and implementation of new features is possible
• TSM can be tested by having both systems running - no down time or missed backups are experienced
• The upgrade can be staged by migrating a few nodes at a time to the new system

Life is rarely as simple as there only being one best fit option for all, and so we see here that the export and import method may not always fit.  For instance if you have lots of archive data then the export / import tasks will be time consuming.  Another consideration relates to the generation of the offsite copies of the storage pools.  These will have to be recreated if the import method is used, this may cause a problem if the number of tapes is restricted or if the data has got to be sent over a WAN using server-to-server communications.

So there is the “Fresh TSM 6.1 with node data exported and imported” option and I would strongly suggest that this method is considered long and hard before embarking on the upgrade path.

Onto the upgrade methods,  IBM have given you four possible upgrade options, these are:

1. Same System, media - the upgrade is on the server that is currently running version 5. The upgrade method of “media” means the data is extracted and inserted in two distinct operations. The extract writes the data to *.ost files. These can then be used for the insert process. Having the media available means you can retry the insert without rerunning the extract.
2. Same System, network - this upgrade is again performed on the server running version 5 but this time the extract and insert are done in the same operation. No intermediate space is needed as the staging area to dump the data before it is inserted into the DB2 database.
3. New System, media - upgrade will be done on a new server. See point 1 for details on the “media method”
4. New System, Network - upgrade will be done on a new server. See point 2 for details on the “media method”

My preference for performing an upgrade would be number a variant on number 3.  This would involve backing up the DB on the original server, restoring it onto the new server and then perform a Same System, media upgrade on the restored database.  This means you can test the upgrade without impacting production and if there is a problem during the upgrade you can quickly revert to the working version 5 environment.

In summary

I believe the TSM 6.1 upgrade should be used as an opportunity to take a fresh look at the TSM design and so running in parallel the TSM 6.1 and the old TSM 5.x server makes sense.  The data can be migrated into the new environment a node at a time and no down time is needed.

If an upgrade is needed I would suggest the following things:

1. Test, test and test again before doing the live upgrade (get an idea of the timings)
2. Use the New System, media method (or variant).

For more information about the TSM 6.1 upgrade there is a manual specifically for the upgrade (http://publib.boulder.ibm.com/infocenter/tsminfo/v6/index.jsp)

Next time we will be considering the upgrade commands, timings and process for a specific scenario!

Step Change

TSM 6.1 is the most exciting change for TSM bods that has occurred in the last 10 years or more.  For those of you who don’t know IBM Tivoli Storage Manager has always used a relational database at its core to make policy management and object tracking possible.  The database that has previously been used is propriety and can be queried as a SQL database but isn’t a true SQL database. 

This DB works pretty well until you start getting beyond about 100GB then admin and maintenance tasks start becoming troublesome.  Anyway the big change is that TSM 6.1 will dispense with the old propriety database and replace it with the well know IBM DB2 Database Engine.   As you may imagine changing from a DB that was well entwined into the application code to one that is completely abstracted will cause some changes!

Pros and Cons

It needed doing! Is the first point I will make although changing the DB engine does give some challenges to TSM admin’s.  The first challenge is migrating from version 5.5.x to 6.1.  The process isn’t pretty!  The data has to be exported and imported (as you would imagine) and there are a number of considerations that can present stumbling blocks along the way (that I won’t discuss here).  Another challenge is that there are several DB and LOG manipulation commands (resizing and changing the location thereof) that are not available anymore due to the fundamental change of the database.  This means a TSM database restore is required to change the location of the TSM database!

However I am sure that in time the Cons will be addressed and the positives will out way these issues.  The big positive from day one is the possibility for vastly more scalable TSM databases.  This results in TSM servers that have been split due to the TSM DB size issue now being able to be consolidated, making management so much easier.  There are lots of feature advantages as well such as active pool offsiting, dedupe of file storage pools etc.

You are about 4 months late!

Yes TSM 6.1 did come out in May 2009 but like most TSM major releases many people leave it to the brave to try out first before seriously considering the upgrade (especially a major upgrade like this).  So I think now is the time that we should all start planning the upgrade.  The upgrades should be carried out  in the new year to allow plenty of time for planning/testing/ getting the new hardware and working out what migration method to use.

Plenty more I could mention about TSM 6.1 so I will leave that for other posts…

What is the point of tiered storage?
Well it is the categorising of your data into types so that they can be located on different media.  The reason they go onto different media is to save cost by not keeping all data on the expensive high performance media needed for a fraction of applications.  So for instance you may have Tier 1 as being Fibre Channel 15k drives, Tier 2 might be SATA drives and Tier 3 could be tape.

Price, no longer a problem
Sometimes implementing the tiers and maintaining them can be time consuming and complex.  Especially when you need to elevate and demote LUNs from one tier to another.  Having done some work with IBM XIV it is clear that the old Tier 1 and 2 makes no sense at all.  When the price per TB is about £4k the price argument for tiering goes away.  The other argument of performance remains until you are convinced that the XIV can perform as well as any traditional Tier 1 disk system.

Storage QoS
So considering IBM XIV and the other buzz in the industry at the moment “cloud storage” we could well be seeing the demise of tiering as we knew it.  Tiering would be replaced with storage QoS whereby the media and location remain constant but LUNs for key applications will get precedence over others.  This seems like a more manageable arrangement whereby you can dynamically alter the performance of your LUN without migrating from one media to another.

I can’t see tape and archive media being scrapped just yet but at least primary storage will be simplified if people apply this modal with the new technology that is coming through.