Part four - Best Practices for Blackboard 5.5 Backend Administrations
This section outlines the best practices for Blackboard 5.5 backend administrations. It includes the procedure required to automate service restart after an outage, log file management, polling mechanism to ensure service availability and some practical tips on performance tuning. This section on performance tuning is applicable only to Blackboard 5.5 platform using Solaris 8 operating system and Oracle 8.1.6i database.
- Automated Service Restart
A procedure has been in place to restart the service should an unexpected outage like power failure occurs. Other than the industry practice of having an Uninterrupted Power Supply (UPS) unit to shut down the servers gracefully, we have added the capability to re-start automatically when the power resumes. This is useful for situations as power outages might occur in the wee hours when nobody is around in the office (e.g. 3am) to execute the manual service restart procedure.
In such a scenario, the Blackboard 5.5 application and web services as well as the Oracle 8.1.6i database services must be able to start up automatically while the servers are being booted up. Scripts have been written to bounce or start up services for Apache, WebLogic and the Oracle Database instance.
- Log Files
Log files for web access and system errors are useful resources when there is a need to perform investigation on systems malfunctioning or forensic analysis on illegitimate web access to course site repository (like on-line grade book).
The size of log file especially the web access log file can be huge, especially in an active eLearning environment like ours. In NTU, the daily access log file can be as large as 300MB during the peak period (usually, the first 3 weeks of the semester). Owing to this, log files must be properly managed for the Blackboard service to run responsively.
Ideally, the log files must be compressed and archived into a date/time stamped folder daily. The file compression will help save disk space and prevent unnecessary wastage of storage. In addition to the log files archival process, a set of new log files should be created daily at 12.00am. If this new set of log files creation is not done, all user access information will be logged and appended to the same set of log files that gets accumulated over time; this results in extremely huge file size. This may lead to slow server response time as every new log entry will require the servers to append it to the end of a growing humongous file.
- Polling Mechanism
For Blackboard sites that have multiple load balanced server configuration and are always heavily used, it might be useful to write scripts to poll the availability of the WebLogic service at regular interavls of, say 5 or 10 min. If the service becomes unavailable, the script will automatically restart it when it detects a malfunction during the polling process. The script can also be written to alert the systems administrators via email for the occurrence of a service restart. This mechanism will help provide fast service turn around time for the users with minimal human intervention.
- Tips on Blackboard 5.5 Performance Tuning
The systems architecture depicting the essential components for Blackboard 5.5 is shown in Figure 10 below. The areas that require performance tuning include the Apache web server, the BEA Web Logic portal server and the Oracle 8.1.6i database server.
Figure 10: Blackboard 5.5 Systems Architecture
It should be noted that the following tips are the experience of the NTU team while administering Blackboard 5.5 servers. The configurations are tabulated here for the purpose of sharing experience. For the optimal performance tuning for your institution’s Blackboard 5.5 servers, you may adopt the tips given here but you are advised to verify the server configuration settings with Blackboard Professional Services team.
In NTU, the Blackboard platform is configured using 3 x Blackboard 5.5 Application Servers (4 x CPU/8GB RAM) and 1 x clustered Oracle 8.1.6i database (6 x CPU/8GB RAM, active-passive failover). This set up is load balanced using 2 x F5 BIG-IP 5000 web load balancers (active-passive failover).
This platform configuration has been able to sustain high utilization rate of 2.1 million weekly page views (or 300,000 daily page views) generated by the academic community.
Technical details related to the tuning of the Apache web server can be found at http://perl.apache.org/docs/1.0/guide/performance.html.
The apache configuration file is found in this directory:
There are some parameters to tune the Apache server. They are discussed below. It would be useful to have a technical person to perform the tuning of the servers. Some of the values are set using trial and error for our environment (that is, nature of transaction, typical transaction length, transaction loads, etc)
- Max Clients: t his is the maximum number of child processes that the server will spawn. It can be translated to the number of simultaneous users that the server can support or the number of concurrent sessions at that particular instance in time.
The formula used to calculate this parameter is:
MaxClients = Available Memory/40MB
Thus, if you have 8GB RAM on your server, you may set the MaxClients setting to (8GB/40MB =) 200.
This is a theoretical calculation. Based on our experience, we set the parameter to 200 initially based on the above calculation but had lowered it to 175 during peak load when the servers were found to running low on memory resources for 8GB RAM.
When this MaxClient limit is reached (that is, the number of simultaneous sessions approach the set value of 200) and the server is unable to process any further new web request owing to limited resources, error message like “Internal Server Error” will appear on the client browser window of the user.
This is recorded in the web error log and the directory is located at
A typical entry looks like this:
[Sun Jan 24 12:05:32 2001] [error] server reached MaxClients setting, consider raising the MaxClients setting.
- Start Servers: t his refers to the number of child processes initially created to service the users. The default number is 10; NTU set it to 16. The number should not be too low as time and system resource are required to create a child process.
- MaxRequestsPerChild: t his is the maximum number of requests that a child process can handle before it dies off and restarts. The default is 300; NTU set it to 400.
- MinSpareServers: t his is minimum number of children awaiting requests. The default number is 5; NTU set it to 8.
- MaxSpareServers: t his is the maximum number of children awaiting requests. The default number is 10; NTU set it to 32.
- BEA Web Logic Portal Server
- File Descriptor: w hen the server experiences an error with the web logic portal server, and messages like “Portal Server not available” appears, there is a need to increase a server parameter called File Descriptor.
A File Descriptor is used whenever the application opens a file or communicates across a network. This parameter can be found in this directory
/usr/local/blackboard/apps/weblogic/weblogicctl ulimit –n 1024
NTU sets it to 1024 for stable performance.
- Web Logic Threads: When user complaints of “slowness logging onto Bb server” start streaming to your helpdesk, there is a need to review the number of web logic threads assigned. This thread is used whenever there is a connection made to Director Server (LDAP) for user authentication.
The web logic properties file is located at
/usr/local/blackboard/apps/weblogic/weblogic.properties. Weblogic.system.executeThreadCount 200
The recommended number is 15; NTU sets it to 200 for optimal performance
- Oracle 8.1.6i Database Server
The cluster Oracle 8.1.6i database is powered by a high-end 6 CPU and 8GB RAM configuration.
The Database parameters are tabled below for information sharing:
db_block_buffers = 512000
shared_pool_size = 268435456
sort_area_size = 4194304
log_buffer = 4194304
To get started with eLearning is easy but to get it going smoothly requires established processes to support good customer services and strong support right from the top management to trainers and frontline operational staff. There is also a need to stay close to technology partners so that support is always rendered promptly when the call for help is activated. From the technological view- point, there should be sound processes in place that provides proactive information alerts of any malfunctioned system that require immediate attention from the systems administrators.