An Ounce of Prevention – Why Backups are important

In the past few blog posts, I have gone over the What, When, and Where of SharePoint Disaster Recovery planning.  So, why? Why do you need any type of Disaster Recovery plan? You have a farm that has ten web front ends that are all load balanced and set for automatic failover.  You have two app servers with mirroring set between them, as well as two separate servers dedicated solely to Search with mirroring as well. You have a clustered database server set for automatic failover. Your high security/high availability set up is second only to the one at Microsoft HQ.  The reason that you need a Disaster Recovery strategy is that disasters will strike even the most robust of environments. Besides, it is always better to have it and not need it, than to need it and not have it. Murphy’s Laws being what they are, simply having a good disaster recovery strategy will probably prevent you from needing it. 

So what kind of disasters could befall such a glorious and robust SharePoint farm that would bring it to its knees and make Administrators, Architects, and Developers weep? Two words; “End Users.” End users are notorious for finding new and creative ways to make things not work correctly. End users are famous for accidentally obliterating months old objects that they discover were needed mere seconds after they were irreparably removed or damaged.  So much so that I have come up with a name for the kinds of chaos and havoc that End Users inflict on SharePoint environments; “Tiny Disasters.”  Just like a toddler sticking a PB&J in your Blu-Ray player, End Users are the worst enemy of healthy SharePoint environments. They are also the reason it exists, so we can’t realistically say “Out of the pool!” (Wouldn’t that be nice though?)   There are ways that you can try to minimize the damage that End Users will inflict, but there are always exceptions.  To minimize the damage that users can unleash upon your SharePoint farm, a good set of governance is a brilliant place to start. Limit who can do what. If you have not yet created a custom permission that is CRUD (Create, Read, Update, Delete) what are you waiting for? Not every contributing user needs to be able to delete. You also need to make sure that only the stuff that is supposed to go in SharePoint does. You don’t really need 1,300 copies of Bruno Mars mp3s stored in your farm. Ultimately, be ready for the end user that manages to upload something that causes damage. If you aren’t prepared for it, it will happen.

One of the most powerful tools that your end users have is the Content Query Web Part.  Unfortunately, it is powerful enough to really mess things up.  Bad code, whether it was developed in house or from a third party, can also exist in your farm. Yes, even ISVs will write code that may not play nice with your farm. Maybe not directly, but ISVs are not known for testing compatibility with their competitor’s products. I know that last bit from personal experience, as I worked at an ISV, and when trying to test compatibility with another ISVs product, there was some friction.   

How do you mitigate this unending flow of terror that is constantly unleashed upon your precious environment? Have backups. Have a Disaster Recovery strategy in place. Test it. Test it often. Just because it worked once doesn’t mean it works now. And the easier it is to recover from a Tiny Disaster, the stress of having them will fade from major paranoia to merely mild inconvenience.  Practice with Tiny Disasters also makes it much easier to manage the catastrophes.  You will already know your recovery plan and know how to be the most effective with it. Knowing you can recover from anything your End Users throw at your environment will help you sleep easier at night. 

Ultimately the best reason to get backups of your SharePoint farm, is so you won’t need them.

 

Backing up the backed up backups of your backups – Where do you keep your backup files.

If you have been following along the past few posts, then you have a pretty good idea what you need to backup, and when you need to back it up.  So once you do have a governance policy in place that says what you need to backup, and when your backups need to run, you are set to go right?  Sure, but where are you going to keep the backup files?  I would like to say that this is a "no-brainer", but I have seen it many times where the backup files were stored on the very server that was backed up.  I will admit that I am guilty of it as well, although under very unique circumstances. (Those circumstances being that it was a testing environment and I didn’t really care if the entire farm went belly up, I was in a position to lose exactly no critical data at all.) So where do you keep those files? When deciding where to keep your backup files there will be several things to consider; you will need to weigh the importance of each aspect to decide which option is best for your organization.

Backup Latency

Backup Latency is how quickly a backup file can be stored to the selected location. Selecting a local drive to store the backup file will be extremely fast, while selecting to store your backup in the cloud will likely be significantly slower.

Safety

While storing a backup file locally is very quick, it is not very secure. If something happens to your server that makes it inaccessible, chances are your backup file is also inaccessible, which means it is useless.

Restore Latency

Restore Latency is how fast you can get the information out of your backup and into your environment. Although there is a close relation between Backup Latency and Restore Latency, they are not identical. Often you do not need to restore the entire contents of your backup, but rather a small portion of it. (I plan to address this in more detail in a future post.)

So where are the best places to store backup files? Below is a table* that lists some common locations and how they rank in my experience. Keep in mind that your mileage may vary.

 

Storage	Backup Latency	Safety	Restore Latency
Cloud	Low	High	Low
Tape	Low	Medium	Low
File Share	Medium	Medium	Medium
Portable Hard Drive	Medium	Low	High
Local Drive	High	Low	High

Many people will wonder why I said Tape only has medium safety. It can be very safe, if it is stored offsite. If you keep your tapes next to your server it isn’t much better than a shared drive in terms of safety. (Which makes it the one of the worst options to go with, unless you have a strict policy around tape management that will prevent the tapes from being stacked on top of the server rack.) The same can be said of the removable hard drive. Typically they are left plugged into the server and just stacked on top of the rack. If there is a strictly enforced policy that says the the portable drive must be kept off site unless it is being actively used in the backup or restore process, safety considerations go up.  The reason that this is important is because not every disaster is related to software or hardware failure. Sometimes environmental conditions are the cause of your disaster. If your server room floods and all of your backups are kept in the same room with your server, you didn’t really have a backup after all did you? You can tell that there are going to be other cost considerations as well. Convenience is one and the actual financial costs should always be a consideration. I’ve added those to the table* below.

 

Storage	Backup Latency	Safety	Restore Latency	Convenience	Cost
Cloud	Low	High	Low	High	High
Tape	Low	Medium	Low	Low	Medium
File Share	Medium	Medium	Medium	High	Medium
Portable Hard Drive	Medium	Low	High	Medium	Medium
Local Drive	High	Low	High	High	Low

 

You can tell that there isn’t really a best, “one size fits all” solution. Each has its advantages and disadvantages. It truly depends on what your needs are and what your organization can afford. Please note that there also is no real correlation between cost, safety, convenience, etc. I should also note that with Tape and Portable Hard Drives you can increase the safety, but it will also decrease the relative convenience. Keep all of this in mind and be sure to include it in your disaster recovery planning.  Cost can also increase with the amount of storage space required.   What do you want to keep in your backups? If you keep your SharePoint backups on the local drive and then through a different disaster recovery policy backup the entire server to the cloud, you just backed up your backup, and that uses storage space. Your backup will take a little longer, and use up a little more storage space.  This will eventually add up. Instead of backing up SharePoint to the local server and then backing up the local server to the cloud, including the backup, it probably makes more sense to backup SharePoint to the same location initially. This also means that in the event of a disaster you don’t have to restore the backup from the restored backup.  (If we keep this up we will end up in a cyclic redundancy loop, like having Doritos Locos Tacos flavored Doritos. Are we next going to have some Doritos Locos Tacos flavored Doritos Locos Tacos?)

*The values in the tables are not from any official source, but just from my personal experience. Your experience may vary greatly from mine. This is only meant as a guide to what you should consider when choosing a location to store your backup files.

OBD-II Diagnostics – When do you backup.

In my last post I spoke about what needed to be backed up. In this one we will address something else that you need to consider in your disaster recovery plan – when.

Ideally you want to create a backup of your content when no one is using it.  Here’s part of why. If you start a backup at 8:00 am on Monday morning, you are probably going to have some unhappy users.  When you start a granular backup, using a 3rd party tool, or the built-in tools like Powershell or Central Administration, the first thing that SharePoint does is lock the Site Collection that is targeted in your backup.  (Doing full farm or database backups does not cause this behavior, but there are other limitations.) While the Site Collection is locked, the users can’t do anything but look at the content in SharePoint.  Anything that would change the contents of any list or library is stopped.  Workflows don’t start. Users cannot update list items or create new documents.  Everything grinds to a halt.  Suddenly SharePoint becomes more like a stuffy museum instead of a petting zoo. (SharePoint was meant to be more like a petting zoo.)  If this happens to your users they will probably not be happy. There is a way around this.  You can opt to not lock sites when you create your backup. This presents another problem that Sean McDonough went into here.

So how do you know when to backup your SharePoint content?  You can assume when no one will be using your SharePoint site and just create backups over the weekend, but this is only hit and miss, and you might be impacting more of your users than you originally thought.  This is why they make diagnostic and performance monitoring tools for SharePoint.

Think of diagnostic tools for SharePoint like the OBD-II sensors in your car. Your car has a myriad of sensors to monitor its performance; fuel sensors, air sensors, and so on. When something is amiss, your car will let you know by a little light on your dashboard. It may not tell you what is wrong, just that something is wrong. When you take the car to a mechanic however, they can pull a code and know exactly what is wrong and what needs to be repaired.  Without a diagnostic tool like this for SharePoint, you may know that something is wrong, but may not be able to determine what. And a diagnostic tool will also let you know if something is going to go wrong before it does, so that you can address it before your end users notice.

So how does a diagnostic tool fit into a DR strategy? You need to know when your SharePoint farm will suffer the least from performing a backup, and when your backups will perform the best.  You don’t want to do a backup when your server is under a high load and there is a lot of traffic on it. It’s not a good idea to guess when an ideal time to perform a backup is. By using tools like performance monitors you can know for sure what kind of impact your backup will have on your end users.

Something else you need to keep in mind about when you backup, is how long your backup will take.  You need to know how long your backup will take to complete so that you can manage your backup within the window that you have determined by your diagnostic tool. The diagnostic tool not only tells you when it is okay to start your backup, but when it should finish by.  Is this really an issue? Absolutely. When I was doing quality assurance testing for Idera’s SharePoint Backup, some of the tests I would perform took days. Not minutes or hours, but days to complete. (This was of course under a very rare and unusual circumstance in a unique environment that you likely don’t have, but it is worth noting.)  Starting a backup on Friday afternoon that doesn’t complete until sometime Wednesday morning that locks up your entire SharePoint farm is not going to make your end users very happy. It is also a good idea to use the performance monitoring tool to see how taxing a backup is on your farm. And to constantly adjust your DR strategy around it. SharePoint is not a static environment. Your end user’s habits change. You need to stay up to speed with what their needs are so that you can accommodate them and work around them.

Fortunately most of the companies that sell DR products for SharePoint also sell performance monitoring tools for SharePoint. There is a reason that those tools exist, and this is one of them.

Target Discrimination – What do you backup?

For those that don’t know me, I am an avid firearms enthusiast.  Handguns, Rifles, etc.  One of the things that responsible shooters practice is target discrimination.  Target Discrimination is the delicate art of hitting exactly what you want to hit, and missing exactly what you want to miss.  So how does this relate to Disaster Recovery? All too often, there is a corporate policy for Disaster Recovery that I call “Carpet Bombing.”  Everything is backed up. Everything. Hard drive snapshots are created every 6 hours. A full system image is created every night. Information that has been sitting dormant for 4 years, 7 months, and 13 days, gets backed up every night, making a new copy of the same information. Even if a backup system is advanced enough to do differential or incremental backups, an index of this unchanged file is created every time.

Perhaps it would be better to imagine this in terms of physical files, and not computer files.  You have a file cabinet that is full of thousands of documents.  Every night, it is the responsibility of one person to photocopy every single file, and then return them to their respective file drawer.  In the case of incremental and differential backups, the file itself is not copied, but the index card that says where the file is, what is in it, and when it was last changed, among other identifying features, is copied. If the change was more recent than what ever the disaster recovery policy says, the actual file is found and added to the xerox pile.  If the company policy is that no photocopies older than 7 days are allowed to be used, every 7 days every single file gets dragged out of the cabinet and photocopied again.

This is where target discrimination comes in. Some files do change, and some don’t. If the file folder for the 10 year lease on the building doesn’t ever change, why would we drag it out and create a backup of it every week? It is apparent that it doesn’t make a lot of sense in a paper world, but we do it in the computer world daily and think nothing of it.  We drag entire hard drives over to the photocopier and make a copy of the whole mess, whether they need it or not. We end up with 137 copies of the exact same file, spread across backups, taking up valuable storage space, and potentially creating a new kind of disaster.

In the paper world, there is one backup. Maybe several versions of the same file, but not many. Actual physical space was much more expensive than computer storage space, so a policy was usually created to only keep so many backup copies of a file. After a certain amount of time, they were shredded. Here is an example. Go to your file cabinet, and pull out your tax forms from 1997. You probably don’t have them anymore. There is no need for them, so they got pitched, shredded or burned in the barbecue pit. Now pull out your tax forms from last year. You still have those. You may not need them, but you have them just in case. (You should anyways.) Now, do you make a backup copy of these every week? Kind of silly to do so isn’t it? If they were computer files though, then for some reason it makes perfect sense.  This is what target discrimination is. Only backup what needs to be backed up. Discriminate the targets of your disaster recovery policy. Be selective about what gets backed up, and how often. Imagine that you have to create a physical print out of every file to get a back up, and then determine what is “mission critical” to be backed up, and what hasn’t changed since 1997.

Just like with physical paper files, it is okay to not create a backup of a file that has not changed since the last backup. Create backups of the important stuff, but let go of the theory of “we have to back up everything, everytime, just in case, just to be sure!”  When you do start discriminating your targets, you will notice something; your backups will take much less time to execute and your used storage space allocated for backups will shrink.

So how do you decide? Well, you are going to have to talk to your end users, and discuss what their needs are, and once they tell you, add what those needs are to some documentation that will govern how you manage your SharePoint farm. (I think it’s called “governance?”) Different groups in your organization are going to have different needs. Just like in your household, different aspects of paperwork have different needs. You don’t need to keep a copy of a school permission slip around for the same amount of time that you need to keep a copy of your mortgage. You probably don’t even need a backup copy of the permission slip, but should probably have several backup copies of other vital documentation, such as your mortgage.  You don’t need to keep a holiday newsletter from five years ago, but you probably should keep the financial records from five years ago handy.

A good way to determine how to discriminate your disaster recovery targets is to imagine that the files actually are physical paper.  Whatever the actual physical paper policy would be, it should be relatively simple to translate into computer disaster recovery terms.  If the file doesn’t change but once a year, you really shouldn’t feel the need to back it up once a week.

So, instead of “carpet bombing” your computer assets with a disaster recovery strategy that simply says “make backups,” a little bit of thought into discriminating what targets need to be backed up and how often, could save large amounts of storage space and system resources.  After all, you don’t need to have 137 copies of your teenage child’s 3rd grade Christmas concert program; one copy will suffice.

Setting “Cancel on first rejection” on an SPD workflow.

I’ve been working on a workflow for a department at work.  It turns out to be a bit complicated, since it is a multi-step approval workflow and I need to clean up the task assignment e-mails sent.  So here is how the workflow needs to work.  When an item is submitted to a library, the approval workflow starts.  One user has an approval task.  If that one user rejects the submitted item the workflow is over.  If he approves, then 5 additional people need to approve the workflow. There is no particular order, so running the second stage of the approval in parallel is ideal.  If any one of those individuals reject the item then the workflow is over.  There must be 6 approvals in total for the item to be approved.  Doing a two stage approval workflow in SharePoint designer is fairly easy, and for clarity I will add how to do that in this post as well.

Don’t they test that?

Well, today is my last day at Idera.  I was a Quality Assurance engineer testing their SharePoint disaster recovery product.  I have seen behind the curtain at a software vendor and I know a lot more about what goes on in the life cycle of a piece of software.  I can assure you that yes, it is tested. Part of my job was to test the functionality of the software and all of the new features that were included in each new release, and that has enlightened me to some very interesting things. 

The good side of the SharePoint community, a view from underneath.

An article was posted the other day by Bjorn Furuknap about the decline of the SharePoint Community. (Article here.) I don’t disagree with what he said, but I have a different point of view on it. First, Bjorn is a well known and respected member of the SharePoint Community.  I’m not saying he is wrong about everything that was stated in his blog post. He is very likely absolutely correct.  What I find is that sometimes it is a matter of the perspective. And my perspective is different from his, for a myriad of reasons.

• Why from underneath?

I am not a SharePoint MVP. I’m not a speaker at SharePoint events around the country. I have yet to speak at our local user group because I want it to be done well.  (I’m also not going to take someone else’s topic of expertise and try to make it my own.) I am a SharePoint Administrator, and I currently do quality assurance for a company that creates SharePoint software. I’m one of the people that is listening intently to the community, either on various blogs, or on Twitter. I do have a blog of my own, that has my own findings on SharePoint, (You’re reading it now incidentally,) however I am not one of the prolific bloggers that has a post a week. Either I don’t run into that many issues, or someone else has already blogged about it. The SharePoint Community is not just the bloggers, the MVPs, the speakers and the sponsors though. The community also has the listeners and the learners. SharePoint Saturdays are not for all of the SharePoint experts to gather around and listen to each other speak. It is so that we can hear what they have to say, and ask questions that we may be too afraid to ask in a different format. It is so that we can learn what we don’t know. And learning from a respected member of the community carries weight. It may not carry weight to the person being asked, but it carries weight to us.

• As the product grows the community will also.

SharePoint has gotten bigger. It almost goes without saying that the community will have to grow also. New SharePoint User Groups, SharePints, and SharePoint Saturdays are popping up all over the place. It is to the point now where there is very likely some kind of SharePoint event going on right now. Our experts and mentors cannot possibly attend every single one of these events. In order to fill that gap, the community grows. We will have more MVPs, more experts, more speakers because of this growth. This isn’t a bad thing. It is just a fact of nature and part of the culture.

• Different users have different needs, and all of those need to be met.

Not every SharePoint user is the same. Some are administrators, some are developers, some are architects, some are power-users, and so on. Not every SharePoint farm is the same. Some farms can live happily on one or two machines and never have any issues. Some farms are giant lumbering behemoths with 35 servers. Multiply the different possible farm configurations, by the different kinds of users, and almost every SharePoint user has a different need for SharePoint knowledge. Some need information on disaster recovery, some need information on performance. Some might even need some completely non technical information.

• SharePoint is a monster, having experts in different areas, or “splintering,” helps us find the influencers we should pay attention to.

There is not a “One Size Fits All” SharePoint answer. Some would say that because of this the SharePoint community has “splintered.” I don’t agree. I think that the SharePoint community has specialized. There are certain SharePoint community members that you know will have authoritative answers to that component of SharePoint that you want to know more about. Some are more outspoken than others, but the experts are there. There is also the occasional SharePointer that just wants a recommendation for a good beer.

• One of the only online communities that is warm and inviting from the outset, you don’t have to “prove” anything.

Once upon a long time ago I was a member of an IRC chat room. In order to be accepted into this community I had to prove something to them. I had to prove that I was worth asking them a question.  The SharePoint community isn’t like that, and I am grateful that it isn’t. If you have a question about something SharePoint, ask. No one will look down on you. No one will tell you to “RTFM.” (Well maybe, but they’ll also give you a link to the exact page in “TFM” that you happen to be looking for.) One of the reasons that I love SharePoint as much as I do, has nothing to do with SharePoint. It is the community. This community could be supporting the “Lets call it beer and drink it anyways” society and it wouldn’t matter. The SharePoint community is built of friendly, helpful people.  That warm and inviting atmosphere inspires more SharePoint users to become SharePoint speakers and experts.

• Only community I know of where individuals who work for companies that are bitter rivals, can be cooperative and engaging.

Most of the time in the corporate world, if you work for the competition, you are somehow not as nice a person. Magically this is not so in the SharePoint community. SharePoint experts who work at competing companies often follow each other on Twitter. And they are friends in real life. Their companies may hate each other, but they don’t hold a grudge and respect each other. I wish that our politicians were more like our SharePoint community.  I’m sure that it is not all fluffy in the community, but for the most part, at least the part that we can see, all of these people act like lifelong friends. I would even dare to say that some of them are my friends, even though we may have only met twice.

So although I can see Mr. Furuknap’s point I don’t think he is entirely correct. Sometimes it helps to consider the perspective and I thought I would add my two cents to the conversation.

It may not be your Content Databases that need upgrading

I recently had to recover from a disaster with one of our SharePoint testing environments. (This happens to be the only testing environment that is not on virtual hardware.) Fortunately, I do quality assurance testing for a disaster recovery product and it wasn’t long before I had new hardware for the failing database server and could rebuild the farm. I did run into some interesting issues however. After I reinstalled the WFEs and had central administration working again, I reattached the content databases that I had recovered. After all of this my farm worked fine and I had access to all of my Site Collections again. Disaster averted. The health analyzer on the farm kept warning me about upgrading databases though. Specifically the error message that I got was the following; "Database is in compatibility range and upgrade is recommended."  Seems simple enough. There are Powershell commands that I can use to upgrade the databases.

$contentdb = Get-SPContentDatabase | Where-Object {$_.Name -eq "WSS_Content"}

Upgrade-SPContentDatabase -Identity $contentdb

I ran those commands from a Powershell prompt (the SharePoint 2010 Management Shell since that one automatically loads the SharePoint snap-in.) I was then immediately irritated when the upgrade failed.

WARNING: Database [SPContentDatabase Name=WSS_Content] does not need to be upgraded.

So, Central Administration tells me that my database needs to be upgraded, and then when I try to update the database, it tells me that it doesn’t need to be upgraded. I was a little confused. I figured I would try a different approach, so I ran PSConfig on the farm with the “build to build” switch.

psconfig -cmd upgrade -inplace b2b -wait -force

This only resulted in more errors.

I then tried to detach and reattach the content databases, since, when attaching a content db, SharePoint should typically upgrade the database if it needs it. That didn’t work either.

I talked with Sean McDonough for a few minutes about the situation and he enlightened me that there is a chance that it is not my content databases that need upgrading. How do you really know though? Time to poke into your content databases to tell for sure.  The first database you need to check is your SharePoint Config database since that is the one that shows you what version your farm is currently using, be it a service pack, or one of the myriad cumulative updates. Run the following in a Sequel Server Management Studio Query.

use SharePoint_Config
SELECT Version
FROM Versions
WHERE VersionId = ’00000000-0000-0000-0000-000000000000′
ORDER BY Id DESC

This will show you every version that your farm has evolved through. Mine showed one version; 14.0.4762.1000. (This is incidentally the RTM version of SharePoint 2010) I then ran the same query against one of my databases that supposedly needed to be upgraded and found more than one version. 4762 was there, but so was 5128. It would seem that it wasn’t my content databases that needed to be upgraded, but rather the rest of my farm. Now my problem was figuring out which version I needed to end up with after upgrading. Fortunately, Todd Klindt has an awesome reference page here that will tell you which Service Pack or Cumulative Update each version number means with a link to download what you need. I downloaded the Cumulative Update for October 2010, installed it on my farm, and then ran psconfig again to finish everything up. All of those “database needs upgrade” errors were gone and the health analyzer on my farm settled down again. Just to be sure I went to the Farm’s Central Administration, then to “Upgrade and Migration” and clicked on “Review Database status.”  Every database stated “No Action required” instead of the upgrade message.

So, if you have to recover a farm and reattach your databases, or you are attaching your content databases to a new farm, keep in mind that if SharePoint tells you that databases need to be upgraded, it might not be the databases that are in need of an upgrade.

Many thanks to Sean McDonough for his input, and to Todd Klindt for his blog for reference. 

 

Who’s SharePoint installation are you training your users for?

The proliferation of SharePoint in the business world brings new users to SharePoint everyday.  Because SharePoint is so easily customizable, many businesses have optimized their SharePoint installation for their own business needs. Some use SharePoint as a CRM system, some use it for ECM, and yet others use SharePoint as a full blown intranet. While some users may have used SharePoint at a previous company, many others have not.  So if you ask someone if they have used SharePoint you’ll get an array of answers and experience.  The real question should be, “Have they used your SharePoint?” 

Where do you get started?

A comment on one of my other blog posts asked me “Which SharePoint role would be a good starting position?” (I did paraphrase the question a bit. The blog post I am referring to is here.) What an awesome question. Where do you begin? When you are first branching out into the vast SharePoint world and looking for a career, what should your first steps be? What books should you read? What classes should you take? What certifications should you pursue? What’s with all the bacon? Hopefully I can shed a little light on this, and tell people the right place to start to begin on a career involving SharePoint.