Document Type : Research Paper

Authors

1 Associate Professor, Psychology Assessment and Measurement, Allameh Tabataba'i University, Tehran, Iran

2 Psychometrics, Classroom Assessment, and Measurement, University of Saskatchewan, Saskatoon, Canada

Abstract

The aim of this study was to investigate the effect of geographical clustering on the estimation of ability parameter (before ranking) as well as the ranking of examinees in large-scale tests (such as national university entrance examination). The design of this study can be considered as survey and because of analyzing the data previously collected by the National Organization Educational Testing (NOET), this project can also be considered as a secondary data analysis. The statistical population of this project includes all the candidates of the Mathematics group who have participated in the national exam of 2013-2014. The sample used in this study includes data on 3,000 examinees in the mathematics group from regions (quota) one, two and three, that have been provided to researchers by the NOET. The main data used in this study are the examinees’ scored responses to test items (1 for a correct answer and 0 for an incorrect answer) and the code related to the selected quota. In summary, it can be concluded that differences between the three regions, or any type of clustering, is considerable when the level of analysis is the overall sample and beyond clusters. In such a situation, it is possible to observe a significant difference between rankings using different methods, even when the value of intra-class correlation (ICC) is very low (like this study).

Keywords

پیک سنجش (1393الف). راهنمای داوطلبان برای انتخاب رشته و ضوابط پذیرش دانشجو در آزمون سراسری 93، سال نوزدهم، شماره 14.
پیک سنجش (1393ب). نمره کل آزمون چگونه ساخته می شود؟، سال نوزدهم، شماره 16.
تیلور، کاترین، اس. (2013). روایی و رواسازی. ترجمه یونسی (1398). انتشارات دانشگاه علامه طباطبائی.
خدایی، ابراهیم (1393). توضیحات رئیس سازمان سنجش درباره سهمیه ها.  https://www.tabnak.ir/fa/news/
شارع­پور، محمود (1386). جامعه­شناسی آموزش و پرورش. تهران: انتشارات سمت.
فاطمه اسلامی هرندی، فریبا کریمی، محمدعلی نادی (1398). شناسایی مؤلفه های عدالت آموزشی در آموزش و پرورش ایران، فصلنامه پژوهش در نظام های آموزشی، 13(47)، ۵۵-۷۵.
لرد، فردریک، ام. (1980). کاربردهای نظریه سؤال- پاسخ. ترجمه دلاور و یونسی (1391). انتشارات رشد.
مگنوسون، داوید. (1966). مبانی نظری آزمونهای روانی. ترجمۀ محمد نقی براهنی (1370)، انتشارات دانشگاه تهران.
یونسی، جلیل اسکندری، فرزاد، ، دلاور، علی، فلسفی‌نژاد، محمدرضا، فرخی، نورعلی (1393). مقایسه توانمندی رویکرد بیزی مدل IRTچندسطحی و مدل کلاسیک چندسطحی: تحلیل داده‌های آزمون فیزیک تیمز پیشرفته (2008)، فصلنامه اندازه‌گیری تربیتی، 5 (15)، ۱۶۶-۱۸۶.
یونسی، جلیل، دلاور، علی، اسکندری، فرزاد، فلسفی‌نژاد، محمدرضا، فرخی، نورعلی (1393). توانمندی رویکرد بیزی مدل چندسطحی: تحلیل داده‌های آزمون ریاضیات تیمز پیشرفته (2008)، فصلنامه پژوهش در نظام‌های آموزشی، 8 (24)، ۲۹۵.
 
References
Bennett, C. , Both, C. & Yeadle, S. (2001). Mainstreaming equality in the committees of the Scottish parliament. University of strathclyde.
Beretvas, S. N., & Kamata, A. (2005). The multilevel measurement models: Introduction to the special issue. Journal of Applied Measurement, 6, 247-254.
Bloom, H. S. (2005). Learning more from social experiments: Evolving analytic approaches.
Brennan, R. L (2006). (Ed.). Educational measurement (4th ed.). Westport, CT: Praeger.
Embereston, Susan E.; Reise, Steven P. (2000). Item Response Theory for psychologists. Lawrence Erlbaum Associates, Publishers, Mahwah, New Jersey. London.
Feinberg, R. A., & Rubright, J. D. (2016). Conducting Simulation Studies in Psychometrics. Educational Measurement: Issues and Practice, 35(2), 36-49.
Feldman, S. (2001). Closing the Achievement Gap. American Educator25(3), 7-9.
Fox, J. P. (2010). Bayesian Item Response Modeling: Theory and Applications. New York: Springer.
Fox, J.-P. (2001). Multilevel IRT: A Bayesian Perspective on Estimating Parameters and Testing Statistical Hypotheses. PhD dissertation, University of Twente, Faculty of Behavioural Sciences.
Ganzeboom, H.B.G., Treiman, D.J., and Ultee, W. (1991). Comparative intergenerational stratification research: Three generations and beyond, Annual Review of Sociology, 17: 277-302.
Gee, J. P. (2008). A sociocultural perspective on opportunity to learn. Assessment, equity, and opportunity to learn, 76-108.
Goldstein, H. (2003). Multilevel Statistical Models, 3rd ed. London: Hodder Arnold.
Golstein, H. (1999). Multilevel Statistical Models. London: Institute of Education, Multilevel Models Project.
Herman, J. L., Klein, D. C., & Abedi, J. (2000). Assessing students' opportunity to learn: Teacher and student perspectives. Educational Measurement: Issues and Practice19(4), 16-24.
 Kamata, A., Bauer, D.J. & Miyazaki, Y. (2008). Multilevel measurement modeling. In A.A. O'Connell & D.B. McCoach (Eds.) Multilevel Modeling of Educational Data (pp. 345-388). Charlotte, NC: Information Age Publishing.
Kane, M. T. (2006). Validation. In R. L. Brennan (ed.), Educational measurement (4th Ed.) (pp. 17-64). Westport, CT: Praeger.
Kerckhoff, A.C. (1995). Institutional arrangements and stratification processes in industrial societies, Annual Review of Sociology, 15: 323-347.
Kline, R. B. (1998). Principles and practice of structural equation modeling. New York: Guilford.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd Ed.) (pp. 13-103). New York: Macmillan.
Mousavi, A. (2013, April). Analyzing data from educational surveys: a comparison of HLM and Multilevel IRT, Paper presented at the annual conference of National council on Measurement in Education. San Francisco. USA.
Muthén, L. K., & Muthén, B. O. (2012). Mplus 7.1 for Windows. Los Angeles, CA: Muthén & Muthén.
New York, NY: Russell Sage Foundation.
Nunnally. J. C. & Burnstein (1994). Psychometric Theory. Mc Graw- Hill book Co.
Raudenbush, S. W. and Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd ed. Thousand Oaks, CA: Sage.
Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data issues in secondary analysis and reporting. Educational Researcher, 39(2), 142-151.
Schmidt, W.H., & Maier, A. (2009). Opportunity to Learn. In Sykes, G., Schneider, B., & Plank, D.N (Eds.), Handbook of Educational Policy (p. 541-559). New York, NY: Routledge.
Snijders T. A. B., & Bosker R. J. (2000). Multilevel analysis. An introduction to Basic and Advanced Multilevel Modeling. Thousand Oaks, CA: Sage.
van der Linden, W. J. and Hambleton, R. K. (1997). Handbook of Modern Item Response Theory. New York: Springer.
Vartanian, T. P. (2010). Secondary data analysis. Oxford University Press.
Zimowski, M. F., Muraki, E., Mislevy, R. J., and Bock, R. D. (1996). BILOG MG: Multiple-Group IRT Analysis and Test Maintenance for Binary Items [computer software]. Chicago, IL: Scientific Software International.