Filter by Categories
Clinical Practice Research
Ethical Legal and Regulatory Considerations
Planning, Managing, and Publishing Research
Research Design and Method

Bilingual Pragmatics Assessment Design

Fourth in the series on Development of a Bilingual Test for Spanish-English Children.

Aquiles Iglesias

DOI: 10.1044/cred-pvd-c13009

The following is a transcript of the presentation video, edited for clarity. Presentation slides are available for download via the PDF button in the toolbar.

Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Videos in this series are:

  1. Challenges in Assessing Bilingual Populations

    (Elizabeth D. Peña, University of Texas at Austin)

  2. Steps in Test Development

    (Elizabeth D. Peña, University of Texas at Austin)

  3. Bilingual Phonology Assessment Design

    (Brian A. Goldstein, La Salle University)

  4. Bilingual Pragmatics Assessment Design

    (Aquiles Iglesias, Temple University)

  5. Bilingual Semantics Assessment Design

    (Lisa M. Bedore, University of Texas at Austin)

  6. Bilingual Morphosyntax Assessment Design

    (Vera F. Gutierrez-Clellen)

The pragmatics subtest is a little bit unique in several ways. One of them is that this is the only subtest that can be given in any language. It can be given in English, in Spanish, or bilingual.

We really don’t care so much about what form the kid is using, or what language the kid is using. What we’re looking at is their pragmatic skills. It’s also the only subtest that is not scored, and you’ll see this in a minute.

In terms of pragmatics, I hope you all know that pragmatics is using the appropriate message or interpretation in relation to the communicative context. If that means nothing to you, it’s what is your intent, what is the cohesion of your items, are you paying attention to what the other person knows, and so on and so forth.

I think what’s more important is something that Rhea Paul and Norbury mention. That is, “A ‘test’ of pragmatics is almost contradictory in terms. Since pragmatics involves the use of language for real communication, we need to assess it in a more naturalistic context.”

So actually developing a test for pragmatics was a very big challenge. Because creating a test, according to Rhea and Norbury, is insane.

Pragmatics is going to depend on the context and the experience and social demands of the situation. One of the things that’s interesting, and you’ll see why we don’t score the pragmatics subsection, is that pragmatic disorders are not unique to children with language disorders. So you can have kids with morophosyntactical problems that do not show pragmatic difficulties. Kids with semantic problems, phonological problems, that don’t demonstrate these pragmatics. One of the things that Brice and Montgomery have looked at also, in terms of L2 learners, is that they might encounter some difficulties, and I’ll address those difficulties that these children might encounter.

Instruments and Approaches

There are very few available instruments to assess pragmatics.

Some of them are observation of parent-child interaction, and you’re supposed to be marking what the interaction is like in terms of the pragmatic skills.

There are also questionnaires. Parents are required to fill out a questionnaire with items like, “Talks to people without any encouragement or starts conversations with strangers.” You know, does the kid initiate and so on and so forth.

Or they set up situations like, “Mary saw a little boy stealing candy at the store. She ran over to her mother and said, …” What is the kid supposed to say here? I think this is a very interesting part of pragmatics. It really depends on your experiences, and what’s acceptable and not acceptable. If you’re coming from a middle-class Anglo perspective, you say “Mother, somebody was stealing something.” Anything else would be totally unacceptable.

So, one of the things that also has been done is common, structured activity with temptations. These are things that the kid has done, within the child’s experience, and create a need for the kid to say or do something.

All of us are familiar with Nancy Creaghead’s making a peanut butter sandwich. Well, how many allergies do we have right now? We can’t be doing peanut butter at all. Plus, those of you who know about making a peanut butter sandwich, you know how messy it is, then somebody has to eat that. And the kid usually gives it to me, and I’m going, “Oh, no way. I’m not going to eat this.”

So we created something called, “A present for Diego.”

We focus on communicative intentions, primarily assertive acts. We look at requests for information, like, “What is that called?” Requests for action or attention, “Gimme that!” Requests for clarification, “¿Un qué?” / “A what?” or assertive comments like “This is a fun game.”

Test Example

So, what you actually will see is the whole test.

These are the objects you need. These are things that are in your closet. You need a box, wrapping paper, three short colored ribbons — and you’ll see why. They don’t have to be red, blue, and green. They can be black, orange, and purple if you want. Three long colored ribbons (not long enough to go around the box). Two tape dispensers — one of which is empty. A Mushki. Then we need a puppet or a doll, and this is optional, and you’ll see why we need it.

Looking at the score sheet, you will see it says that this is an optional activity that can be used to establish rapport with the child, as well as to observe how the child uses/understands pragmatic language. This activity is to be used for descriptive purposes only and does NOT contribute to any score.

What you have then is the stimulus on the left-hand-side, the expected on the right, and then whether the kid uses it or not.

So, the examiner says, “Let’s wrap Diego’s present. I bought Diego a great gift. It’s in the box. What do you think it is?”

There’s another alternative that we created. We realized — not our Puerto Rican kids or our Dominican kids, but our Mexican-American kids sometimes were not confrontational to an adult. As you can see, sometimes you have to be confrontational. So we created this puppet. So the puppet is the one these kids can say things to, not the person. So, using this option we would say, “Let’s wrap Diego’s present. This is my friend Timmy. He is going to help us wrap Diego’s present. Sometimes he can be silly, but don’t let him fool you. I bought Diego a great gift. It’s in the box…”

What you’re getting then is this responsive act, “A toy.” Or no response.

So you say, “It’s a Mushki!”

You’re expecting a request for clarification or a request for information. Again, we’re giving the kid options. You really don’t know what the kid is going to do, whether it’s a request for clarification or a request for information. “A what?” / “What is a Mushki?” / Or he gives you the puzzled look.

You say, “A Mushki, see.” Then you open the box. “We use Mushkis to bingle the waddles.”

So, the kid is going to say, “What?” Or he is going to say, “You’re crazy.”

You say, “Let’s see, to wrap the present, we need wrapping paper, tape and ribbon. Do you remember how to wrap a present? Hmm. Tell me.”

And the kid might say, “Bueno, lo primero que se hace es comprar todos los materiales necesarios para completar la tarea.” For those of you who speak Spanish — wow.

Or kid 2 says, “Box y ribbon pa el party.”

Folks, realize one of the things that’s very critical — I’m going to mention it later on. You are picking up a tremendous amount of information with respect to the kid’s language skills in this process.

So, we place the paper and the box on the table. And you say, “Let me have the ribbon.” And the ribbon is not on the table. So the kid has to say, “We only have paper and a box.” Or “There is no ribbon.” Or the kid will say something like, “Where is the ribbon?” Again, it’s assertive acts.

“Oh, I forgot, it’s in the bag.” And you pull out the red, blue, and green ribbon. Now, there were short ribbons and long ribbons. Because the kids might not know colors. And we want to make sure we don’t get into this labeling problem, so we have these three ribbons that he can choose. So you say, “I have a red ribbon, a blue ribbon, and a green ribbon. Which one do you want?” And he points to that one.

And I say, “Here it is.” And the kid says, “That one is green.” Or “That is not the one I want.” Or “Where is the red one?” Again, assertive acts.

“Oh, I’m sorry.” Then you give him the correct ribbon.

You bring up the empty tape dispenser. Which all of you have empty tape dispensers, as you work in the schools, and you begin wrapping the present.

And you say to the kid, “Please give me some tape.” Well, the tape dispenser has no tape on it. So you’re forcing the kid now to say something. “There is no tape.” or “Where is the tape?” or something to that effect.

“Oh, I better get another one.” Then you get the dispenser with tape. “Ok, now we have to put the ribbon.” But you keep the ribbon. And the kid says, “Let me have it.” A request for action. Or, “I don’t have the ribbon.” Or, “Where’s mine?” Again, we’re not really caring about the form, we’re caring about the use. And particularly, assertive acts.

So, you give the ribbon, but the ribbon is too short. You have to measure ahead of time, by the way. You might get these statements from these kids: “The ribbon is too short.” Or, “Do you have a longer ribbon?”

You say, “So what do you think we should do?” This is a request for information. So, the kid’s got to answer. The kids come up with great answers to this. “Not put any ribbons.” “Tape the ribbon.” “Tie two ribbons to make a long one.”

“Great idea. I think Diego is going to love this present. What do you think Diego is going to say?”

“Thank you for the present.” / “I think he is going to hate it.”

So, that is the pragmatics subtest. Again, it provides the context, the expected responses, and then you circle it yes and no.


Remember what I said to you — pragmatics skills are not necessarily tied to some of the morphosyntax or phonology scores, and this is very well proven by our data. The correlation between English phonology, English morphosyntax, and English semantics with pragmatics is very, very low. The same thing happens with Spanish.

What I think we need to look at is these language profiles. If the kid has morphosyntax OK, semantics OK, pragmatics OK, great. If a kid has OK morphosyntax, OK semantics, but low pragmatics: what kind of kids are these going to be? These are the kids that most of the teachers — and we have a lot of data of seeing this a lot — these kids who are in these classes, and their language skill, morphosyntax and semantics are okay, but they’re just sitting there not being assertive in the least bit. And the teacher thinks that there’s a language problem there.

Or you can have a low morphosyntax, low semantics, and OK pragmatics. This is the kid that we know that at least from a use perspective, we’re going to be able to build on that use in order to increase morphosyntax and semantics.

And finally we have these low, low, low kids. Which, you know what they mean.

So, what do we know, and what have we accomplished by giving this subtest?

>We have established rapport with the child. One of the things we did as we were collecting a lot of the data, was to actually give the pragmatics subtest first. It gave us an insight into the child’s phonology, morphosyntax, and semantics. And also, it gave us a good impression of how difficult it was to test this child. After you do this thing, if the kid has not been very cooperative, be prepared, you need to schedule another session for the test.

And also it has provided you with information on pragmatic strengths and challenges.

References for this Series

Allen, M. & Yen, W. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth.
Alt, M., Meyers, C. & Figueroa, C. (2013). Factors that influence fast mapping in children exposed to Spanish and English. Journal of Speech, Language, and Hearing Research56(4), 1237–1248 [Article]
Alt, M. & Suddarth, R. (2012). Learning novel words: Detail and vulnerability of initial representations for children with specific language impairment and typically developing peers. Journal of Communication Disorders45(2), 84–97[Article] [PubMed]
Anderson, R. T. (2001). Lexical morphology and verb use in child first language loss: A preliminary case study investigation. International Journal of Bilingualism5(4), 377–401 [Article]
Bedore, L. M., Peña, E. D., Summers, C. L., Boerger, K. M., Resendiz, M. D., Greene, K., Bohman, T. M. & Gillam, R. B. (2012). The measure matters: Language dominance profiles across measures in Spanish/English bilingual children. Bilingualism: Language and Cognition15(3), 616–629 [Article]
Bedore, L. M., Peña, E. D., Gillam, R. B. & Ho, T. (2010). Language sample measures and language ability in Spanish-English bilingual kindergarteners.Journal of Communication Disorders43(6), 498–510 [Article] [PubMed]
Bishop, D. V. (1998). Development of the children’s communication checklist (CCC): A method for assessing qualitative aspects of communicative impairment in children. Journal of Child Psychology and Psychiatry39(6), 879–891 [Article] [PubMed]
Bonifacio, S., Girolametto, L., Bulligan, M., Callegari, M., Vignola, S. & Zocconi, E. (2007). Assertive and responsive conversational skills of Italian-speaking late talkers. International Journal of Language & Communication Disorders42(5), 607–623 [Article]
Brice, A. & Montgomery, J. (1996). Adolescent pragmatic skills: A comparison of Latino students in English as a second language and speech and language programs. Language, Speech, and Hearing Services in Schools27(1), 68–81[Article]
Cotton, E. & Sharp, J. (1988). Spanish in the Americas. Washington, DC: Georgetown University Press.
Carrow, E. (1974). Austin Spanish articulation test. Austin, TX: Learning Concepts.
Hammond, R. (2001). The Sounds of Spanish: Analysis and application (with special reference to American English). Somerville, MA: Cascadilla Press.
Goldstein, B. (2007). Measuring phonological skills in bilingual children: Methodology and clinical applications. In Centeno J., Obler L. & Anderson R. (Eds.). Studying Communication Disorders In Spanish Speakers: Theoretical, Research, & Clinical Aspects. Clevedon, UK: Multilingual Matters.
Goldstein, B. & McLeod, S. (2012). Typical and atypical multilingual speech acquisition. In McLeod S. & Goldstein B. (Eds.). Multilingual aspects of speech sound disorders in children. Clevedon, UK: Multilingual Matters.
Goldstein, B. & Gildersleeve-Neumann, C. (2012). Phonological development and disorders. In Goldstein B. (Ed.). Bilingual language development and disorders in Spanish-English speakers (2nd edition). Baltimore: Brookes.
Gray, S. (2004). Word learning by preschoolers with specific language impairment: Predictors and poor learners. Journal of Speech, Language, and Hearing Research47(5), 1117–1132 [Article]
Gray, S. (2005). Word learning by preschoolers with specific language impairment: Effect of phonological or semantic cues. Journal of Speech, Language, and Hearing Research48(6), 1452–1467 [Article]
Gutiérrez-Clellen, V. F., Restrepo, M. A. & Simón-Cereijido, G. (2006). Evaluating the discriminant accuracy of a grammatical measure with Spanish-speaking children. Journal of Speech, Language, and Hearing Research49(6), 1209–1223 [Article]
Gutiérrez-Clellen, V. F. & Simón-Cereijido, G. (2007). Evaluating the discriminant accuracy of a grammatical measure with Latino English-speaking children. Journal of Speech, Language, and Hearing Research50(4), 968–981[Article]
Hodson, B. (1986). Assessment of phonological processes-Spanish. San Diego: Los Amigos Research Associates.
Jacobson, P. F. & Schwartz, R. G. (2005). English past tense use in bilingual children with language impairment. American Journal of Speech-Language Pathology14(4), 313–323 [Article] [PubMed]
Kohnert, K. J. & Bates, E. (2002). Balancing bilinguals ii: Lexical comprehension and cognitive processing in children learning Spanish and English. Journal of Speech, Language, and Hearing Research45(2), 347–359[Article]
Leonard, L. B., Eyer, J. A., Bedore, L. M. & Grela, B. G. (1997). Three accounts of the grammatical morpheme difficulties of English-speaking children with specific language impairment. Journal of Speech, Language, and Hearing Research40(4), 741–753 [Article]
Mcgregor, K. K., Newman, R. M., Reilly, R. M. & Capone, N. C. (2002). Semantic representation and naming in children with specific language impairment. Journal of Speech, Language, and Hearing Research45(5), 998–1014 [Article]
Mason, M., Smith, M. & Hinshaw, M. (1976). Medida Española de articulación (Measurement of Spanish Articulation). San Ysidro, CA: San Ysidro School District.
Mattes, L. (1985). Spanish articulation measures. Oceanside, CA: Academic Communication Associates.
Melgar de Gonzalez, M. (1976). Como detectar al niño con problemas del habla [Identifying the child with speech problems]. Mexico City: Trillas.
Paul, R. & Norbury, C.F. (2012). Language disorders from Infancy through adolescence. St. Louis, MO: Elsevier.
Peña, E.D., Gutíerrez-Clellen, V.F., Iglesias, A., Goldstein, B. & Bedore, L.M. (2014). Bilingual English Spanish Assessment (BESA). AR-Clinical Publications.
Restrepo, M. A. & Kruth, K. (2000). Grammatical characteristics of a Spanish-English bilingual child with specific language impairment. Communication Disorders Quarterly21(2), 66–76 [Article]
Rice, M. L. & Wexler, K. (1996). Toward tense as a clinical marker of specific language impairment in English-speaking children. Journal of Speech, Language, and Hearing Research39(6), 1239–1257 [Article]
Sheng, L., Peña, E. D., Bedore, L. M. & Fiestas, C. E. (2012). Semantic deficits in Spanish-English bilingual children with language impairment. Journal of Speech, Language, and Hearing Research55(1), 1–15 [Article]
Toronto, A. (1977). Southwest Spanish articulation test. Oceanside, CA: National Education Laboratory Publishers, Inc.
Vermeer, A. & Shohov, S.P. (2004). Exploring the lexicon: Quantitative and qualitative aspects of children’s L1/L2 word knowledge. In Advances in Psychology Research. Hauppauge, NY: Nova Science Publishers.

Aquiles Iglesias
Temple University

Originally presented at the ASHA Convention (November 2013) as part of the session Development of a Bilingual Test for Spanish-English Children: A Long and Winding Road. Co-Presenters: Elizabeth D. Peña, University of Texas at Austin; Aquiles Iglesias, Temple University; Vera F. Gutierrez-Clellen, San Diego State University; Brian A. Goldstein, La Salle University; and Lisa M. Bedore, University of Texas at Austin.
Disclosure: All of the above-listed authors/co-presenters benefit financially from royalty payments from the Bilingual English-Spanish Assessment (BESA.).
Copyrighted Material. Reproduced by the American Speech-Language-Hearing Association in the Clinical Research Education Library with permission from the author or presenter.