Supporting System for the Calibration of Items by the Procedure Based on Expert Judgment
DOI:
https://doi.org/10.37467/gka-revedutech.v2.960Keywords:
Items, Calibration, Experts, MoodleAbstract
Evaluation is a key issue in learning and item banks are very common resources used to perform activities related to it. Professional evaluation deals with banks that are calibrated, which means that the parameters of the items (for example, their difficulty) have previously been estimated. One option for item calibration consists in gathering the assessments of a panel of experts and using them to determine the values of the item parameters. This paper describes CALLIE-PRO: a system that customizes and automates the process of item calibration based on expert judgment. It guides the user by recommending a choice for every decision to be made, adapting to changes in the specification, and warning of non advisable or incorrect choices. It also creates an e-learning content package that depends on the process specification and can be included in a learning management system such as, for example, Moodle. This package is responsible for gathering the assessments of the experts via the internet and generating an executable workflow that will include the activities to be performed, which are mainly analyses and calculi, by using all gathered data. Finally, the system provides estimates for the parameters (difficulty) of the items.
References
Armendariz, A. J., López-Cuadrado, J., Tapias, A., Villamañe, M., Sanz-Lumbier, S. y Sanz-Santamaría, S. (2003). Learning environments should follow standards: ELSA does . World Conference on E-Learning in Corporate, Government, Healthcare, & Higher Education, Phoenix, Arizona (USA), Association for the Advancement of Computing in Education (AACE).
Arruabarrena, R. (2010). E-learning y la calibración de ítems de test: Teoría de Respuesta al Ítem versus calibración basada en juicios de expertos. Un estudio empírico . San Sebastián, España: Lenguajes y Sistemas Informáticos, Universidad del País Vasco/Euskal Herriko Unibertsitatea.
Arruabarrena, R., Pérez, T. A., Gutiérrez, J., López-Cuadrado, J. y Vadillo, J. A. (2002). On Evaluating Adaptive Systems for Education . AH2002, 2nd. International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Málaga, España: Springer-Verlag.
Fallon, C. y Brown, S. (2003). E-learning Standards: A guide to purchasing, developing and deploying standards-conformant e-learning . Ed. ST. Lucie Press.
Kingsbury, G. G. y Weiss, D. J. (1983). A comparison of IRT-based Adaptive Mastery Testing and Sequential Mastery Testing Procedure. En D. J. Weiss (eds.), New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing (pp. 257-283). New York, United States: Academic Press.
López-Cuadrado, J. (2010). Test adaptativos informatizados de ingreso en un sistema e-learning. Saarbrüken, Germany: Lambert Academic Publishing.
López-Cuadrado, J., Armendariz, A. J., Latapy, M. y Lopistéguy, P. (2008). A genre-based perspective for the development of communicative computerized adaptive tests. Journal of Educational Technology & Society, 11 (1), pp. 87-101.
López-Cuadrado, J., Armendariz, A. J., Pérez, T. A. y Arruabarrena, R. (2008). Helping tools for item bank calibration and development of computerized adaptive tests . International Technology, Education, and Development Conference (INTED2008). Valencia, España: International Association of Technology, Education, and Development.
López-Cuadrado, J., Armendariz, A. J., Pérez, T. A., Arruabarrena, R. y Vadillo, J. A. (2009). Computerized adaptive testing, the item bank calibration and a tool for easing the process. En A. Lazinica & C. Calafate (Ed.), Technology Education and Development (pp. 457-478). New York, United States: Intech.
López-Cuadrado, J., Pérez, T. A., Vadillo, J. A. y Gutiérrez, J. (2010). Calibration of an item bank for the assessment of Basque language knowledge. Computers & Education, 55 (3), pp. 1044-1055.
Mark, M. A. y Greer, J. E. (1993). Evaluation Methodologies for Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education. Special Issue on Evaluation, 4 (2/3), pp. 129-153.
Muñiz, J. (2010). Las teorías de los tests: Teoría Clásica y Teoría de Respuesta a los Ítems. Papeles del Psicólogo, 31, pp. 57-66.
Pérez, T. A. (2000). Un hiperentorno adaptativo para el aprendizaje instructivo / constructivo. San Sebastián, España: Lenguajes y Sistemas Informáticos, Univ. País Vasco/Euskal Herriko Unibertsitatea.
Presedo, C., Armendariz, A. J. y López-Cuadrado, J. (2012). Calibración de ítems para test informatizados: Descripción detallada de las fases en la construcción de test de evaluación adaptativos mediante ordenador . Saarbrüken, Germany: Editorial Académica Española.
Riley, K. y Mills, D. (2008). IMS Common Cartridge (CC) Authorization Web Service v1.0 Final Specification . IMS Global Learning Consortium, Inc., accedido el 10/6/2104 desde www.imsglobal.org/cc/
Smythe, C. y Nielsen, B. (2007). IMS Content Packaging (CP) Specification Primer v1.2 Public Draft. IMS Global Learning Consortium, Inc., accedido el 10/6/2014 desde www.imsglobal.org/content/packaging/
Smythe, C., Shepherd, E., Brewer, L. y Lay, S. (2002). IMS Question & Test Interoperability: ASI Information Model Specification. Final Specification Version 1.2 . IMS Global Learning Consortium, Inc., accedido el 10/6/2014 desde www.imsglobal.org/question/
Wainer, H., Bradlow, E. T. y Du, Z. (2000). Testlet response theory: an analog for the 3PL model useful in testlet-based adaptive testing. En W. J. van der Linden & C. A. W. Glas (Ed.), Computerized adaptive testing: theory and practice (pp. 245-269). Dordrecht, The Netherlands: Kluwer Academic Publishers.
Wainer, H., Dorans, N. J., Flaugher, R., Green, B. F., Mislevy, R. J., Steinberg, L. y Thissen, D. M. (2000). Future challenges. Computerized Adaptive Testing: A Primer (2nd edition) (pp. 231-269). Hillsdale, New Jersey: Lawrence Erlabum Associates.
Downloads
Published
How to Cite
Issue
Section
License
Those authors who publish in this journal accept the following terms:
- Authors will keep the moral right of the work and they will transfer the commercial rights.
- After 1 year from publication, the work shall thereafter be open access online on our website, but will retain copyright.
- In the event that the authors wish to assign an Creative Commons (CC) license, they may request it by writing to publishing@eagora.org
