Do C-Test and Cloze Procedure Measure what they Purport to be Measuring? A Case of Criterion-Related Validity

Authors: Masoud Saeedi, Mansour Tavakoli, Shirin Rahimi Kazerooni, Vahid Parvaresh

Abstract:

This article investigated the validity of C-test and Cloze test which purport to measure general English proficiency. To provide empirical evidence pertaining to the validity of the interpretations based on the results of these integrative language tests, their criterion-related validity was investigated. In doing so, the test of English as a foreign language (TOEFL) which is an established, standardized, and internationally administered test of general English proficiency was used as the criterion measure. Some 90 Iranian English majors participated in this study. They were seniors studying English at a university in Tehran, Iran. The results of analyses showed that there is a statistically significant correlation among participants- scores on Cloze test, C-test, and the TOEFL. Building on the findings of the study and considering criterion-related validity as the evidential basis of the validity argument, it was cautiously deducted that these tests measure the same underlying trait. However, considering the limitations of using criterion measures to validate tests, no absolute claims can be made as to the construct validity of these integrative tests.

Keywords: Integrative testing, C-test, Cloze test, theTOEFL, Validity.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1334810

References:

[1] J. W. Oller, Language tests at School: A Pragmatic Approach, London: Longman, 1979.
[2] A. D. Cohen, "On taking tests: what the students report," Language Testing, vol. 1, pp. 70-81, 1984.
[3] C. J. Weir, Communicative Language Testing, Hemel Hempstead: Prentice Hall, Inc., 1998.
[4] J. D. Brown, "Correlational study of four methods for scoring Cloze tests," Unpublished master-s thesis, University of California, USA, 1978.
[5] J. W. Oller, and J. Jonz, Eds. Cloze and Coherence, Cranbury, NY: Bucknell University Press, 1994.
[6] M. R. Sampson, and L. D. Briggs, "A new technique for Cloze scoring: A semantically consistent method," Clearing House, vol. 57, no. 4, pp. 177-79, 1983.
[7] M. Pikulski, and A. W. Tobin, "Cloze procedure as an informal assessment technique," In J. J. Rikulsky and T. Shanahan, Eds. Approaches to the informal evaluation of reading, Newark: International Reading Association, 1982.
[8] J. C. Alderson, "The effect on the Cloze test of changes in deletion frequency," Journal of Research in Reading, vol. 2, pp. 108-18, 1979.
[9] A. G. Sciarone, and J. J. Schrool, "The Cloze test: or why small isn-t always beautiful," Language Learning, vol. 39, no. 3, 415- 38, 1989.
[10] M. Soudek, and L. Soudek, "Cloze after thirty years: New uses in language teaching," ELT Journal, Vol. 37, no. 4, pp. 335-40, 1983.
[11] J. C. Alderson, "The effect of certain methodological variables on Cloze test performance and its implications for the use of Cloze procedure in EFL testing," Paper presented at the fifth International Congress of Applied Linguistics, Montreal, 1978.
[12] J. C. Alderson, "The Cloze Procedure and Proficiency in English as a Foreign Language," In J. W. Jr. Oller, Ed. Issues in Language Testing Research, Rowly, Mass: Newbury House, 1983, pp. 205- 217.
[13] J. D. Brown, "A closer look at Cloze: validity and reliability," In J. W. Oller, Ed. Issues in Language Testing Research, Rowley, Mass: Newbury House, 1983, pp. 237-250.
[14] J. B. Heaton, Writing English Language Tests. London: Longman, 1990.
[15] J. W. Oller, "Cloze tests of second language proficiency and what they measure," .Language Learning, vol. 23, pp. 101-105, 1973.
[16] P. Irvine, P. Atai, and J.W. Oller, "Cloze, dictation, and the Test of English as a Foreign Language," Language Learning, vol. 24, pp. 245-52, 1974.
[17] J. B. Stubbs, and G. R. Tucker, "The Cloze test as a measure of English proficiency," The Modern Language Journal, vol. 58, pp. 239-41, 1974.
[18] K. G. Aitken, "Using Cloze procedure as an overall language proficiency test," TESOL Quarterly, vol. 11, pp. 59-68, 1977.
[19] C. Klein-Braley, and U. Raatz, "A survey of research on the C-test," Language Testing, vol. 1, no. 2, pp. 134-46, 1984.
[20] U. Raatz, and C. Klein-Braley, "Ein neuer Ansatz zur Messung der Sprachleistung. Der C-test: Theorie und Praxis," In R. Horn, K. Ingencamp, and R. Jager, Eds. Tests und Trends, Jahrbuch der Padagogischen Diaanostik. Weinheim: Beltz: 1983, 107-38.
[21] C. J. Weir, Understanding and Developing Language Tests, Prentice Hall, Inc., 1993.
[22] A. Cohen, M. Segal, and R. Weiss, "The C-test in Hebrew," Language Testing, vol. 1, pp. 221-225, 1984.
[23] R. Grotjahn, "Test validation and cognitive psychology: Some methodological considerations," Language Testing, vol. 3, pp. 159- 185, 1986.
[24] R. Grotjahn, "How to construct and evaluate a C-test: A discussion of some problems and some statistical analyses," In: R. Grotjahn, C. Klein-Braley, and D. K. Stevenson, Eds. Taking their Measure: The Validity and Validation of Language Tests, Brockmeyer, Bochum, Germany, 1987, pp. 219- 253.
[25] R. Grotjahn, Ed. "Der C-test: Theoretische Grundlagen und praktische Anwendungen," AKS-Verlag, Bochum, 2002.
[26] U. Feldmann, and B. Stemmer, "Thin____ aloud a____ retrospective da____ in c-te___ taking: Diffe______languages - diff_____ learners - sa____ approaches?" In: C. Faerch, C. Kasper, Eds. Introspection in Second Language Research, Clevedon: Multilingual Matters, 1987.
[27] C. Cleary, "The C-test in English: Left-hand deletions," RELC Journal, vol. 19, pp. 26-38, 1988.
[28] C. Chappelle, and R. Abraham, "Cloze method: What difference does it make?" Language Testing, vol. 7, pp. 121-146, 1990.
[29] M. Hood, "The C-test: A viable alternative to the use of the cloze procedure in testing?" In L. Arena, Ed. Language Proficiency. Plenum Press, New York, 1990, pp. 173-189.
[30] Z. Dornyei, and L. Katona, "Validation of C-test among Hungarian EFL learner," Language Testing, vol. 9, pp. 187-206, 1992.
[31] T. Kamimoto, "An inquiry into what a C-test measures," Fukuoka Women-s Jr. College Studies, vol. 44, pp. 67-79, 1992.
[32] T. Kamimoto, "Tailoring the test to fit the students: Improvement of the C-test through classical item analysis," Fukuoka Women-s Junior College Studies, vol. 30, pp. 47-61, 1993.
[33] A. Jafarpur, "Is C-test superior to Cloze?" Language Testing, vol. 12, pp. 194-216, 1995.
[34] A. Jafarpur, "Can the C-test be improved with classical item analysis?" System, vol. 27, pp. 79-89, 1999.
[35] G. Sigott, J. Kobrel, "Deletion patterns and C-test difficulty across languages," In: R. Grotjahn, Ed. Der C-test, Theoretische grundlagen und praktische anwwndungen, vol. 3, pp. 159-172, Brockmeyer, Bochum, 1996.
[36] C. Klein-Braley, "C-tests in the context of reduced redundancy testing: An appraisal," Language Testing, vol. 14, pp. 47-84, 1997.
[37] C. Ikeguchi, "Do different C-tests discriminate proficiency levels of EL2 learners?" JALT Testing & Evaluation SIG Newsletter, vol. 2, pp. 3-8, 1998.
[38] E. Babaii, and H. Ansary, "The C-test: a valid operationalization of reduced redundancy principle?" vol. 29, pp. 209-219, 2001.
[39] A. J. Hastings, "The Focal Skills Approach: an Assessment," In F. Ekman, et al., Eds. Second Language acquisition Theory and Pedagogy, Hillsdale, NJ: Lawrence Erlbaum, 1995, pp. 29-43.
[40] R. J. Mislevy, "Test theory reconceived," Journal of Educational Measurement, vol. 33, no. 4, pp. 379-416, 1996.
[41] L. F. Bachman, Statistical Analysis for Language Assessment, Cambridge: Cambridge University Press, 2003.
[42] A. Hughes, Testing for language teachers, 1st ed. Cambridge: Cambridge University Press, 1989.
[43] G. Henning, A guide to language testing. Cambridge, MA: Newbury House, 1987.
[44] L. F. Bachman, Fundamental Considerations in Language Testing, Oxford: Oxford University Press, 1990.
[45] P. Groot, "Language testing in research and education: The need for standards," In J. De Jong, Ed. Standardization in Language Testing, AILA Review, vol. 7, pp. 9-23, 1990.
[46] M. T. Kane, "An argument-based approach to validity," Psychological Bulletin, vol. 122, no. 3, pp. 527-35, 1992.
[47] E. Hatch, and H. Farhady, Research Design and Statistics for Applied Linguistics. Rowley, Massachusetts: Newbury House, 1982.