After several years of beta testing Blackboard software, I've come up with a list of the things we at Seneca College try to accomplish as part of our internal beta testing process.  If you are chosen to be a Beta Program partner you may find this useful.  From what I understand, Blackboard receives more requests to be part of the Beta Program than they can handle (literally).  While more is better, it's often more effective to have a smaller group that is more focused and active in testing than larger groups.  That being said, it's important to be active so here's what we do.


Try to Attend All of the Beta Conference Calls.

While conference call attendees get asked what testing they have done since the last call, they are also given important information about issues and time lines.  Blackboard will also bring in various resources to demonstrate new features and discuss issues as appropriate.  Time lines, while prearranged, may change depending on the feedback and issues reported so it's important to keep current.

Seems Simple but... Test Out the "New" Software.

The beta software will contain new or rewritten software tools that beta testers will most likely be unfamiliar with.  In this early stage, there is usually no documentation so testing is largely through self-discovery.  Sometimes the beta conference calls will discuss the new functionality but it's usually demonstrated and is just not the same as hands-on usage of the new functionality.  Beta testers should "play" with the software by reviewing all of the portions of the new or enhanced tool thoroughly and it should be tried out with various pedagogical scenarios.  I personally try to figure out the software without learning anything about it beforehand.  Consider that this may be how your faculty are introduced to these new changes so "walk in their shoes" and try it like you think it should work.

Review your Support Tickets and Enhancement Requests.

As has been the case recently, portions of the Blackboard product are being rewritten.  This opens the door for enhanced functionality that you may have requested through enhancement requests or issues documented in  lower priority support tickets.  As a beta tester, see if any of your requests have been implemented.  You will occasionally be pleasantly surprised.

Support tickets also get reviewed when new releases come out so check your support tickets to see if anything has been changed to resolve them.  If so, test thoroughly and do a final test when in production and don't forget to update your tickets.

Include Key Faculty in Beta Testing.

I'm sure that everyone has particular faculty that either "champion" a particular tool or have enhancement requests or concerns that they have voiced.  If a particular tool is enhanced or is included as new functionality, include those faculty in the beta process and have them test it out as they will use it in class.  The beta process is not just for system administrators even though the majority of the effort may come from them. Faculty can be included in the beta process as well because the non-disclosure document you sign covers the entire institution.

If Upgrading to this New Version, Test the Upgrade Process.

I think the biggest issue we had with Blackboard 6.0 was the fact that we beta tested the software on a test server but we did not try a complete upgrade from 5.6 in our beta process.  We learned the hard way that we needed to. If I can stress one point, it would be to test out the upgrade and do your beta testing on the upgraded test server.  I realize that sometimes test servers are not the same hardware configuration as your production servers but upgraded software and data is what you will be working with in production when you do your production installation so this needs to be tested. You should also try to beta test the software with the same or near same server configuration.  We ran into technical issues several times when beta testing with upgraded data because faculty use tools differently than what the software was originally written to do and these use cases were not tested.

Trust me; the beta process is when you want to catch these issues; not in production.

Collaborate with Fellow Beta Testers.

Part of the beta process usually includes a course site on a Blackboard server specifically for beta testers that houses several tools including a discussion board for discussion of issues and general feedback.  Check this often and participate in providing feedback.  This is vital to the beta process and generally no feedback means that everything is perfect (just like tech support, right?). So, post plenty of feedback.

Report your Issues.

I've talked to several Blackboard support people over the years and the most frustrating thing is that common issues usually never get reported.  This is a beta test so now is not the time to think that someone else "must have run into this issue so they must have reported it".  I think Blackboard would rather go through several reports of the same issue than not having it reported at all.  Don't hesitate to report them!

Don't forget to test the stuff you didn't think changed.

Beta testing should include general testing of all aspects of the software. Normally, particular focus in testing is given to new or enhanced functionality because, let's face it, we want to see what's new!  Equally important is general testing to make sure nothing else broke. There are smaller less visible fixes and enhancements that are made and the new enhancements may also indirectly effect other portions of the system (like Grade Center or email) so include everything in your general testing plan.  In Blackboard 9.0, the user interface was greatly enhanced but even if core functionality didn't change, we still went thoroughly through the software.  We also keep our "test plan" for participation in later betas and update it with the new features each time.

Commit to a regular weekly Beta Testing period.

I'm not sure anyone can afford to focus on beta testing and do nothing else.  In our case, we schedule and dedicate one half to one full day of beta testing per week.  In recent beta tests we were able to focus on testing with everyone in one room, testing out various scenarios as we went through each major sub system or tool.  If you are able to do this, I think you will find that more testing done than trying to take a small portion of each day to test, for example.

New Build?  Time to start all over.

Depending on the beta process, you may get access to multiple iterations of the software known as "builds" to test.  I have seen major back-end changes occur between builds and functionality that changes from build to build.  While I know it gets a bit tedious, go through your test plan for every build.  It may save you some headaches later on.

 
Final Testing is Completed with the Production Release.

Nobody like surprises when you install software in production and things do change between various builds and the final "Production Release".  Remember that this version is just another build so it goes without saying that you should do complete testing with the production version of the software in your beta testing environment before you install this version in production.