Introduction to Technical Matters
To get the Technical Matters section started, I have written the first of a series of articles which reflect some of the issues that I expect most Blackboard users have had to, or will have to, address. These have had a significant impact on our learners and staff during the implementation and rollout of Blackboard. The first is about a scary experience with backup and recovery and, to follow, will be hands-on articles on the pitfalls involved in upgrading/installing Blackboard Version 6 (enterprise).
Back up and Recovery of the Blackboard Learning System : A Scary experience
To give you some technical background to our particular implementation of Blackboard, we installed Blackboard Learning System 5.6 on a top of the range Dell server (as specified by Blackboard) running under Microsoft advanced server operating system, with an SQL Server 2000 back-end database. We fully integrated and automated our data imports using the Blackboard snapshot process with our SIS, personnel, agency and timetabling systems. We have also implemented single point of authentication for staff and students via the LDAP facility of our network operating system, Novell 6.
As you would expect from a responsible College ICT department, prior to the rollout of any new system, we discussed and implemented a backup and recovery plan for Blackboard. This was, in fact, an extension of the current College system plan for its other IT systems. Our worst case scenario was one where the original Blackboard server would be totally irrecoverable. To protect against this we decided to replace the hardware in order that we could completely restore the system without loss of any significant amount of data.
Our initial plan was to automate a daily backup of the database and transaction files of all the relevant databases on our back-end SQL Server 2000 database platform ( i.e. master, model, msdb, bb50 and portalserver). We also decided to initiate a daily backup to a set of daily, weekly and monthly tapes of all the required data, including system state information, using the College backup software, Veritas. Finally, we tested the backup by recovering various directories from the backup tapes. Job done, we thought, and congratulated ourselves on a good job.
All went well for the first nine months and then disaster struck. I got the dreaded telephone call from the server room, “Houston we have a problem”! Somehow, and don’t ask me how, we had a multiple hard disk and raid controller failure which, after frantic telephone calls with Dell hardware support, turned out to be irrepairable. Fortunately, this was not too bad, as we could replace the hardware with next day support, recover from backup tape and be up and running within no time. Next day new hardware arrived and we went for our full system restore from the backup tapes, but only 98% restored. So, to cut a very long and frustrating story short, after spending many hours on the telephone with Veritas support being told 'it should work…’ it didn’t! My main concern at this point was not only about recovering the Blackboard Learning System and all its data, but also about recovering all of the bespoke integration work that had been done in order to link Blackboard with several of our College systems (MIS, Personnel, Timetabling). After final consultation with Veritas and our technical support, it was decided that the only course of action left open to us was for a fresh installation of the operating system.
The plan of action was agreed and a new Windows 2000 advanced server installation was installed. The backup software application was installed and the previous installation system state was recovered from tape. Then, after a reboot, hey presto, we were back in business. A further restore from tape of the D: drive of the previous installation, which contained the Blackboard application and data and, with only minor tweaking, we were operational again without any loss of data.
So what did we learn from our scary experience?
• We quickly came to realise during the downtime that we had grossly under-estimated, even at the early stage of our implementation of Blackboard (1st year), the speed at which the system had become a “mission critical” system.
• The system downtime had a significant effect on the confidence that academic staff had in the reliability of the system and, therefore, on the uptake of the system by staff and learners, and this, in turn, took several months to overcome.
• The absolute need to have a flexible but comprehensive hardware and software backup/recovery plan, (and especially ‘flexible’ because, as we found, some things do not always do what it says ‘on the box’), and very few of us are in a position where we can test a full system restore.