At last week’s Campaign Tech Conference, I participated in a panel entitled “If it crashes, they can’t come: Planning uptime in politics.”  During the panel, we discussed a lot of ways to minimize downtime from a technical perspective.  Choosing the right host provider.  Optimizing your site to improve performance.  Using CDNs to protect server resources.

This technical stuff is really important.   If you don’t make good technical decisions at the beginning of the project it can be costly and time consuming to fix mistakes. 

But as vital as the technical stuff is, in my experience the number one cause of downtime is human error.  People make mistakes more often than machines break.

With that in mind, below are some tips on how to minimize the impact of human error on your web program. 

1. Set Up a Proper Development Environment

One of the biggest causes of downtime is developers making changes and enhancements  directly in the live, production environment.  When you do this you are working without a net – if you screw up all the people trying to access the site will see your mistake.  Political campaigns, who want everything done RIGHT NOW and are in a constant state of panic, often fall into this bad habit in an effort to save time.

A ton of problems can be prevented by setting up a proper development environment and having the discipline to use it.  The ideal set up consists of the following:

• A development site that serves as a sandbox for designers and developers.  This is where you build things.
• A staging site that is used for testing purposes.  Once quality control on a new features has been completed in the development environment, the work is moved to the staging site for final testing.  The staging site is usually a complete mirror of the production site in terms of content.
• A production site that is the version of the website that your visitors actually see.   You move files from staging to production when they are ready for the world to see.

At a minimum, I would make sure your web team has an up to date development environment to work from. 

2. Test Your Work

Back in June of 2011, the Romney campaign launched a new version of their website at midday.  The website more or less worked, but the it contained placeholder images, links to their development site and an action center with no actions for people in the District of Columbia.  The problems were sorted out within a few hours, but a lot of people noticed the errors. 

Basic testing of the Romney site before roll out would have caught these errors.   Web teams should have multiple people test all changes before they are pushed to a production server for the world to see.  This can prevent both technical errors that lead to downtime and incomplete work from being published.

3. Deploy During Off Hours

The launch of a new version of your site is the time when you are most vulnerable to downtime.  You should go into deployments assuming something will go wrong.  At best, it might take 10 minutes for you to copy all the files to the live environment and get everything set up the way you need it to be.  The site will likely be jacked up during the period when you are copying things.  At worst, there could be more series problems that cause the production environment to be buggy for hours while you sort things out.  To minimize this risk, perform large scales deployments during off hours when no one is paying attention.  Minimize risk. 

4.  Establish a Basic Workflow

Another big cause of downtime is staff making changes to the website without proper training.  Examples I’ve seen over the years:

• Staff copying Word documents directly into the CMS HTML editor and publishing, accidently destroying the site’s layout.
• Un-optimized images getting posted, slowing load times and using up site bandwidth.
• Unpaid interns making changes to Drupal templates on the production environment, taking the site down in the process.
• Use of the web server as a file sharing system, slowing down load times by draining the servers resources.

As the saying goes: “This ain’t Nam.  There are rules.”

Make sure that your staff is trained to perform the work you ask of them.  Consider setting up a workflow where someone with experience reviews materials before they are posted to the website.  If someone is only going to be doing one thing on your website (like posting blog entries), restrict access in your CMS so that that is the only privilege they have.

Some basic rules and procedures can prevent a lot of problems.

5. Keep up with Site updates

Open source CMS solutions like Drupal and Wordress are terrific, low cost options for political websites.  But when using these systems keep in mind that the software requires ongoing maintenance.  Drupal and WordPress frequently launch new versions, often to fix security holes in the software.  You should plan to update your software on a quarterly basis, at a minimum.  If a critical security update comes out, make the changes right away.  If you don’t update your site, you open yourself up to hacking attacks from opponents or bots that crawl the web looking for sites with vulnerabilities.

Similarly, you need to make sure the operating system of the hosting provider you are using gets updated on a regular basis. 

Ask your provider or development firm their process for managing both software and server updates.

6.  Take backups

If you are hosting a site on an inexpensive provider like Go Daddy or Blue Host they do not back up your site on a regular basis.  Not having a  backup process in place is a huge risk. 

As an example, say you built your site on WordPress and it gets hacked for some reason.  When WordPress sites get hacked, the hackers usually delete the majority of content and turn the site into an ad for an online pharmacy.  If this happens and you are not keeping backups, you may not be able to recover your data. 

For sites we manage, we take backups every night.  We typically keep a week of daily backups, a month of weekly backups and three months of monthly backups on hand at all times.  We always store our backups on a different server than the site is hosted on as an extra safeguard against a catastrophic hardware problem.  Make sure your development and/or hosting firm sets up a procedure for backing up the site.

7. Slow Down, Plan

Most mistakes political campaigns make are a result of working too fast and skipping steps.  While it may not be possible, following some basic protocols and making sure there is adequate testing time will save a lot of headaches and prevent downtime.  Slow down a bit, and try to plan things in advance.