Steps in Assessment Development

Second in the series on Development of a Bilingual Test for Spanish-English Children, this presentation is a general overview of the process for developing a new assessment test.

Elizabeth D. Peña

DOI: 10.1044/cred-pvd-c13007

The following is a transcript of the presentation video, edited for clarity. Presentation slides are available for download via the PDF button in the toolbar.

Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Videos in this series are:

  1. Challenges in Assessing Bilingual Populations

    (Elizabeth D. Peña, University of Texas at Austin)

  2. Steps in Test Development

    (Elizabeth D. Peña, University of Texas at Austin)

  3. Bilingual Phonology Assessment Design

    (Brian A. Goldstein, La Salle University)

  4. Bilingual Pragmatics Assessment Design

    (Aquiles Iglesias, Temple University)

  5. Bilingual Semantics Assessment Design

    (Lisa M. Bedore, University of Texas at Austin)

  6. Bilingual Morphosyntax Assessment Design

    (Vera F. Gutierrez-Clellen)

Plan the Test

What are the steps in test development?

Well, first you plan the test. What are the domains that you want to address?

In our case, we looked at semantics, morphosyntax, phonology, and pragmatics.

Write Items for Each Domain

Then what we did is we wrote items for each of these domains. We wrote probably three or four times the items that we would eventually want.

A clinical test is one that is pretty efficient. You want to be able to give a test, and have each subtest take maybe 20 minutes, right? You don’t want to do a three- or four-hour test. But to start, you have to start out with a three or a four hour test to figure out which items are your best items for this population. That’s basically what we did, and each person is going to talk about how we did that for each domain in particular.

Administer to a Small Sample

Then you administer all the items to a small sample. Usually 50 kids. If you’re developing something like the GRE, you would administer that to several hundred. We’re not doing the GRE, so 50 or so kids was a good sample size to start with.

Conduct Item Analysis

Then you conduct item analysis. And through that item analysis, you start throwing out the items that don’t work.

Administer Revised Test and Cross-Validate.

Then, we take this shorter, revised test. And we administer it to another sample. A bigger sample.

Then we continue to do this, doing item analysis and cross-validation as we start to step through all these steps and reduce the number of items that we have in each of the different domains.

Item Difficulty

I’m going to tell you a little bit about item difficulty. Item difficulty is the proportion of a population that got an item correct. Those are p values. And p values go from 0 to 1. If they are close to 0, that means nobody got that item right. So it’s way too hard, so that doesn’t tell you anything, so you get rid of it.

If everybody got that item right, then it’s way too easy. So you might include one or two so you don’t totally bum out the kids, but you don’t want too many really easy items, because that’s not going to tell you anything either.

Then the items that you select depends, of course, on the purpose of your test. So we use these same principles for item difficulty and apply it for different kids at different levels of exposure to two language as well as for kids with and without impairments that we were trying to identify.

Item Discrimination

The next thing you do is look at item discrimination. Item discrimination is the comparison of the percentage of typical kids who got the item right, and the percentage of children with impairments who got that item right. We subtract and look at the differences between those two. There we’re going to get a difference of, again, 0 to 1. And the greater the difference, the more sensitive that item is to the impairment you’re trying to identify.

If you can build your test based on those items, items that are sensitive to that impairment, but don’t have big discrimination values for level of exposure, for example, or age of first language exposure, then you have a better chance of having a measure that is going to help you make distinctions between your clinical and non-clinical group.

References for this Series

Allen, M. & Yen, W. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth.
Alt, M., Meyers, C. & Figueroa, C. (2013). Factors that influence fast mapping in children exposed to Spanish and English. Journal of Speech, Language, and Hearing Research56(4), 1237–1248 [Article]
Alt, M. & Suddarth, R. (2012). Learning novel words: Detail and vulnerability of initial representations for children with specific language impairment and typically developing peers. Journal of Communication Disorders45(2), 84–97[Article] [PubMed]
Anderson, R. T. (2001). Lexical morphology and verb use in child first language loss: A preliminary case study investigation. International Journal of Bilingualism5(4), 377–401 [Article]
Bedore, L. M., Peña, E. D., Summers, C. L., Boerger, K. M., Resendiz, M. D., Greene, K., Bohman, T. M. & Gillam, R. B. (2012). The measure matters: Language dominance profiles across measures in Spanish/English bilingual children. Bilingualism: Language and Cognition15(3), 616–629 [Article]
Bedore, L. M., Peña, E. D., Gillam, R. B. & Ho, T. (2010). Language sample measures and language ability in Spanish-English bilingual kindergarteners.Journal of Communication Disorders43(6), 498–510 [Article] [PubMed]
Bishop, D. V. (1998). Development of the children’s communication checklist (CCC): A method for assessing qualitative aspects of communicative impairment in children. Journal of Child Psychology and Psychiatry39(6), 879–891 [Article] [PubMed]
Bonifacio, S., Girolametto, L., Bulligan, M., Callegari, M., Vignola, S. & Zocconi, E. (2007). Assertive and responsive conversational skills of Italian-speaking late talkers. International Journal of Language & Communication Disorders42(5), 607–623 [Article]
Brice, A. & Montgomery, J. (1996). Adolescent pragmatic skills: A comparison of Latino students in English as a second language and speech and language programs. Language, Speech, and Hearing Services in Schools27(1), 68–81[Article]
Cotton, E. & Sharp, J. (1988). Spanish in the Americas. Washington, DC: Georgetown University Press.
Carrow, E. (1974). Austin Spanish articulation test. Austin, TX: Learning Concepts.
Hammond, R. (2001). The Sounds of Spanish: Analysis and application (with special reference to American English). Somerville, MA: Cascadilla Press.
Goldstein, B. (2007). Measuring phonological skills in bilingual children: Methodology and clinical applications. In Centeno J., Obler L. & Anderson R. (Eds.). Studying Communication Disorders In Spanish Speakers: Theoretical, Research, & Clinical Aspects. Clevedon, UK: Multilingual Matters.
Goldstein, B. & McLeod, S. (2012). Typical and atypical multilingual speech acquisition. In McLeod S. & Goldstein B. (Eds.). Multilingual aspects of speech sound disorders in children. Clevedon, UK: Multilingual Matters.
Goldstein, B. & Gildersleeve-Neumann, C. (2012). Phonological development and disorders. In Goldstein B. (Ed.). Bilingual language development and disorders in Spanish-English speakers (2nd edition). Baltimore: Brookes.
Gray, S. (2004). Word learning by preschoolers with specific language impairment: Predictors and poor learners. Journal of Speech, Language, and Hearing Research47(5), 1117–1132 [Article]
Gray, S. (2005). Word learning by preschoolers with specific language impairment: Effect of phonological or semantic cues. Journal of Speech, Language, and Hearing Research48(6), 1452–1467 [Article]
Gutiérrez-Clellen, V. F., Restrepo, M. A. & Simón-Cereijido, G. (2006). Evaluating the discriminant accuracy of a grammatical measure with Spanish-speaking children. Journal of Speech, Language, and Hearing Research49(6), 1209–1223 [Article]
Gutiérrez-Clellen, V. F. & Simón-Cereijido, G. (2007). Evaluating the discriminant accuracy of a grammatical measure with Latino English-speaking children. Journal of Speech, Language, and Hearing Research50(4), 968–981[Article]
Hodson, B. (1986). Assessment of phonological processes-Spanish. San Diego: Los Amigos Research Associates.
Jacobson, P. F. & Schwartz, R. G. (2005). English past tense use in bilingual children with language impairment. American Journal of Speech-Language Pathology14(4), 313–323 [Article] [PubMed]
Kohnert, K. J. & Bates, E. (2002). Balancing bilinguals ii: Lexical comprehension and cognitive processing in children learning Spanish and English. Journal of Speech, Language, and Hearing Research45(2), 347–359[Article]
Leonard, L. B., Eyer, J. A., Bedore, L. M. & Grela, B. G. (1997). Three accounts of the grammatical morpheme difficulties of English-speaking children with specific language impairment. Journal of Speech, Language, and Hearing Research40(4), 741–753 [Article]
Mcgregor, K. K., Newman, R. M., Reilly, R. M. & Capone, N. C. (2002). Semantic representation and naming in children with specific language impairment. Journal of Speech, Language, and Hearing Research45(5), 998–1014 [Article]
Mason, M., Smith, M. & Hinshaw, M. (1976). Medida Española de articulación (Measurement of Spanish Articulation). San Ysidro, CA: San Ysidro School District.
Mattes, L. (1985). Spanish articulation measures. Oceanside, CA: Academic Communication Associates.
Melgar de Gonzalez, M. (1976). Como detectar al niño con problemas del habla [Identifying the child with speech problems]. Mexico City: Trillas.
Paul, R. & Norbury, C.F. (2012). Language disorders from Infancy through adolescence. St. Louis, MO: Elsevier.
Peña, E.D., Gutíerrez-Clellen, V.F., Iglesias, A., Goldstein, B. & Bedore, L.M. (2014). Bilingual English Spanish Assessment (BESA). AR-Clinical Publications.
Restrepo, M. A. & Kruth, K. (2000). Grammatical characteristics of a Spanish-English bilingual child with specific language impairment. Communication Disorders Quarterly21(2), 66–76 [Article]
Rice, M. L. & Wexler, K. (1996). Toward tense as a clinical marker of specific language impairment in English-speaking children. Journal of Speech, Language, and Hearing Research39(6), 1239–1257 [Article]
Sheng, L., Peña, E. D., Bedore, L. M. & Fiestas, C. E. (2012). Semantic deficits in Spanish-English bilingual children with language impairment. Journal of Speech, Language, and Hearing Research55(1), 1–15 [Article]
Toronto, A. (1977). Southwest Spanish articulation test. Oceanside, CA: National Education Laboratory Publishers, Inc.
Vermeer, A. & Shohov, S.P. (2004). Exploring the lexicon: Quantitative and qualitative aspects of children’s L1/L2 word knowledge. In Advances in Psychology Research. Hauppauge, NY: Nova Science Publishers.

Elizabeth D. Peña
University of Texas at Austin

Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Co-Presenters: Elizabeth D. Peña, University of Texas at Austin; Aquiles Iglesias, Temple University; Vera F. Gutierrez-Clellen, San Diego State University; Brian A. Goldstein, La Salle University; and Lisa M. Bedore, University of Texas at Austin.
Disclosure: All of the above-listed authors/co-presenters benefit financially from royalty payments from the Bilingual English-Spanish Assessment (BESA.).
Copyrighted Material. Reproduced by the American Speech-Language-Hearing Association in the Clinical Research Education Library with permission from the author or presenter.

Share on facebook
Share on pinterest
Share on twitter
Share on linkedin