Secure Web-based Standard-setting System

Educational Testing Service

Background

The Educational Testing Service (ETS) develops standardized tests used nationwide and revises them annually. To assure the highest standards of quality and consistency, test items are developed by a panel of judges, consisting of dozens of educators and other experts from across the country.

The collaborative process had always been a conventional one of assembling all participants in a room to develop test content. To save money and time, and to make the decision process more objective, ETS required a totally customized Web-based conference platform in which judges could review and discuss potential questions. The process had to be highly secure and enable facilitation by ETS administrators.

The Solution

Fulcrum IT built a Web-based system for collaboration, facilitated by a human moderator.

• Judges gather in a virtual meeting space, evaluate questions, and compare their evaluations with those of others in real time. The system employs a robust Oracle relational database to track and present evaluations.
• Discussion and debate takes place in a threaded discussion in which judges can view one another's comments and suggestions, and modify their positions.
• Results are mathematically processed to generate standards for the next exam.
• The highest standards of security keep the test questions private.

The Results

ETS accelerated the adoption schedule of this system because it generates several clear benefits:

• Decision making among experts is faster and more accurate. ETS judges save significant time and arrive at better results.
• The freedom to schedule electronic sessions allows judges to meet more frequently, respond faster to immediate needs, and address critical issues and concerns more effectively.
• The Web-moderated process itself is quicker and more efficient than face-to-face meetings, saving time while maintaining the energy and focus of the participants—producing a better product.
• The collaboration tool results in enormous savings in travel and per diem expenses, freeing budgetary resources for other pursuits.