We performed 2 big upgrades to our production Blackboard environment last week. We had been running 7.0 for almost 2 years, but it was time to upgrade to 7.3. The upgrade path was pretty straightforward except for the use of hot fixes in place of the base application pack.

Before walking through the steps it’s probably best to mention the environment a little more. Our Bb environment was running 6 RedHat Enterprise Linux (RHEL) 3 application servers, and an Oracle 10g R1 (10.1.0.5) Real Application Cluster (RAC) database running on an addition 4 RHEL3 nodes. All of the appserver and database nodes are Dell PowerEdge 1850’s with 6GB and 8GB of memory respectfully.

In recent months we were fortunate enough to receive a brand new NetApp to replace our current EMC Storage Area Network (SAN). So in preparation for the upgrade, we migrated the ASM, Voting disk, Oracle Cluster Registry (OCR), content mounts, etc over to the new storage. This made it possible to have a much smaller down time window allocated because of NetApp’s snapshot technology. Rather than having to run an entire RMAN backup we could simply snapshot the datafiles on the off chance a disaster occurred. The downtime window would have needed to be roughly 6 hours larger to accommodate the initial cold backup and rollback using RMAN, rather than an instant backup and rollback by snapshot.

So the backup began by shutting down services on all 6 of the application servers so that a snapshot of content could be made as well as a backup of /usr/local/blackboard. Then a snapshot of the database was made. The cluster specific setting had to be rolled back as well as our custom authentication module for our single sign-on portal. At that point we kicked off the upgrade to 7.1. Now at this point I’d like to note that we built up a development environment very similar to the production one in which to roll all the way through these patches. In development it took just over 4 hours to perform the upgrade to 7.1. It took around 12 hours in production. The only differences that we could identify were the following: /usr/local/blackboard/content was next to empty in dev, and that the storage was using fiber-channel rather than NFS over ethernet. Once it wrapped up we truncated the bb_bb60.activity_accumulator table and turned off archiving to help speed up the upgrade from 7.1 to 7.3. This portion completed within 2.5 hours which was just a little longer then its 1hour 45 minute run time in dev.

The upgrades went smoothly overall, but we did find some bugs in the 159.0 release that weren’t caught while testing in development. If we had used the hotfix #2 release it would have resolved a lot of the little bugs. Because their patches are cumulative, they can be used in place of the base install. I’m not really sure why they would even provide the link to the bugged versions…. Why not just take them down of the site?

Next we upgrade the nodes to 10g R2, RHEL4 and 64bit.