Experience sampling refers to a set of data collection methods for gathering systematic self-reports of behaviors, emotions, or experiences as they occur in the individual’s natural environment. These procedures may also be labeled as event sampling, real-time data capture, ambulatory assessment, diary method, or ecological momentary assessment in the research literature. Sampling strategies include interval-contingent sampling (participants provide responses at pre-determined times), signal-contingent sampling (participants are notified via beeper, text message, etc. at fixed or random intervals when it is time to provide responses), and event-contingent sampling (participants provide responses upon the occurrence of a pre-determined event).
Video: What is the Experience Sampling Method (ESM)?
Experience sampling methodology is a way of gathering data that uses repeated assessments in the individual’s natural environment. So, it’s a combination of sampling methods where somebody might get beeped or paged several times a day over the course of a week or two weeks. And each time they get that page, they write down certain aspects of what they’re doing.
So if, as a clinician or a researcher I was interested in somebody with aphasia and how they’re interacting with other people or who is around them in their environment when they are interacting, I might page somebody several times a day and ask them certain questions like: “How effective is the conversation you’re having?”; “How much do you enjoy the conversation?” You can measure other variables that are going on, so it might be, “Who are the people around you?”; “Do you know that person who’s around you?”; “Are you outside? Are you inside?”; “Are you with people who know about aphasia?”
Over the course of a week, or two weeks, or whatever the time period is, you get several or many data points. So you can get a profile of those particular types of behaviors that you’re wanting to measure. They are self-reported, so the individual reports them, but then you can look at them as a researcher or a clinician, and also you can look at them with the individual — or the individual themselves can look at them. So the client could keep a running track of how many times they are using their voice therapy techniques, or an individual with a head injury might be looking at keeping track of how many times are you using metacognitive strategies. There’s a great deal of application across the spectrum of individuals that we work with in communication sciences and disorders as well as in audiology.
What are some of the challenges of using ESM in research?
The method itself has been used across a wide variety of disciplines, from academia, measuring engagement from students in grade school, middle school, high school, college. In business, in terms of employee morale, employee satisfaction. But in speech-language pathology and audiology, there’s not a great deal of use.
I think one of the challenges is gathering the data itself. When this originated back in the ’70s, this was when it first originated, there were more technological challenges because randomly sampling somebody several times a day over the course of a week was much harder to do. The technology wasn’t there. So you would use the cumbersome pagers, you’d get an actual page, then you’d write things on notebook paper.
Now, technology has certainly made that much easier. There are apps, there are ways to program smartphones, sending text messages, programming — those kinds of things make that much easier. At the same time, there remain those challenges where we found, there’s a certain subset of the population age-wise that is not used to carrying around an iPod or iPad or having a phone. So what several of us might think of as, hey this is great, we just put the app right on your phone, others find that still very cumbersome and would rather have a digital watch that just beeps. So finding the right fit of the data gathering tools is important.
The other challenge is that you get a lot of data points. From a research standpoint, that’s great because it helps to increase, I believe, validity and reliability. That also creates challenges, though, in terms of more data points means more data to analyze and to gather. The data itself, from purely a research or statistical standpoint, there’s some complexities with the data because you have issues with the independence of the ratings within and individual, and then between several individuals, issues of the independence of the data. So repeated measures, assessments don’t always work so well. Often it requires, and is more and more requiring HLM, or hierarchical linear modeling, which is a fairly complex statistical method that not a tremendous amount of people do. So that may scare some people away.
I think it’s important to know that, while there’s certainly utility from maybe a research or statistical standpoint, there’s a great deal of value in purely the descriptive information — just quantifying, just like tally marks, how often was I doing that behavior. Challenges-wise, there’s a lot of data to be gathered.
What surprised you about implementing ESM in you research?
A couple things that surprised me. One was, I was really excited about the ability to incorporate this fairly seamlessly into mobile technologies. I thought that was going to take care of a lot of the problems that have been reported in the literature more historically. Like I mentioned earlier, I was surprised by the number of people that felt like, yeah carrying a phone or an iPod really was a burden. It was certainly an oversight on my part, in the sense that — it made sense after we talked about it. Some of that was just ego-centrism. We’re all used to carrying it around, it becomes like a part of our bodies really. Some people are not like that. There’s — we found an age issue, that those that tend to be older were less likely to be wanting to carry around that device. That was surprising.
The amount of compliance — so, how often does somebody answer the responses. We found people to be very dedicated to that, and to be very vested in the process. Even the idea, I wondered, do people find this intrusive, and is it a barrier, and does that become a problem. A lot of the literature talks about going for about the course of week. Shorter than a week, you don’t get as accurate a picture; longer than a week you tend to have more of a burden. So the study that I ran, we did it for a week. That seemed to be about right. People said, “Yeah, it was fine.” As a matter of fact people were actually interested in having it and were very interested in what their results showed. I was little but surprised at how vested people were in it.
What advice do you have for researchers looking to get started with ESM?
I don’t want to say that it’s the be-all, end-all because nothing ever is. And there are drawbacks to it. There are issues that people who do tend to use it more accurately or more consistently are probably more vested in whatever the behavior is. I think we can all relate to this if we have an app of MapMyRun or something where you’re charting your own eating. Usually you start out pretty well, and then things start to fade a little bit. But those ideas are also powerful in terms of the application of this. When you have that device with you and that sampling, there’s an increase in accountability.
While there are certainly flaws, I think that one of the takeaways is that there is a real potential not just for data gathering, but for utility for the clinician and the client relationship. It’s not just a tool for researchers, but it’s a tool for helping in terms of managing and moderating behaviors, targeting aspects that you and the client or patient have agreed upon that you want to work at. So, takeaway one would be there’s utility from both the clinician and the client standpoint.
Takeaway two would be that it can be high tech or low tech. You can make it as complicated or as simple as you can. It can be as simple as, “Before you go to bed at night, write on these three questions.” Or, “When you get up in the morning, take a minute and just talk into you recorder about what you plan to do for the day.” It can be that simple. And you can make it as complex — you can design computer programs to page and to sample and all these kinds of things. You really can vary the complexity. I think it fits in whatever your needs are for your client or patient, so I think there’s application there.
I think the third thing would be to not be scared off by the statistics. You have to be very mindful that we don’t want to be reporting invalid results and using statistics in a way they shouldn’t be used. There are some challenges in ESM, in experience sampling method, with that. So, there are times where higher-level complex statistics are needed. They are very necessary so we are not drawing false conclusions. There are other times though where descriptives are important and valuable. I don’t think that clinicians or researchers or clients should be scared off by the potential complexities, because there’s a lot to be gathered in just some of the descriptives. What are some of the targeted behaviors, and what’s going on around you? You don’t have to get standard deviations, and means, and p values. You have really rich discussions looking at that data or that information over the last week or two.
Tools and Examples
Conner, T. S. (2015, May). Experience sampling and ecological momentary assessment with mobile phones. [PDF] A summary of reviews of current software for conducting experience sampling studies. Available at http://www.otago.ac.nz/psychology/otago047475.pdf
Fitzgerald-Dejean, D. M., Rubin, S. S. & Carson, R. L. (2012). An application of the experience sampling method to the study of aphasia: A case report. Aphasiology, 26(2), 234–251. [Article]
Fitzgerald-Dejean, D. M., Rubin, S. S. & Carson, R. L. (2009). Experience Sampling Method: SLP Intensive Treatment Quality of Life Measure. [PDF] Slides from a presentation at the American Speech-Language-Hearing Association Covention, New Orleans, LA. Available at http://www.asha.org/events/convention/handouts/2009/1197_fitzgerald-dejean_donna
James, S., Brumfitt, S. & Cowell, P. (2009). The influence of communication situation on self-report in people who stutter. International Journal of Speech-Language Pathology, 11(1), 34–44. [Article]
Further Reading
Csikszentmihalyi, M. & Larson, R. (1987). Validity and reliability of the experience-sampling method.. The Journal of Nervous and Mental Disease, 175(9), 526–536 [Article] [PubMed]
Hektner, J. M., Schmidt, J. A. & Csikszentmihalyi, M. (2007). Experience sampling method: Measuring the quality of everyday life. Thousand Oaks, CA: Sage Publications.
Schwarz, N. (2007). Retrospective and concurrent self-reports: The rationale for real-time data capture. In Stone, A. A., Shiffman, S., Atienza, A. A. & Nebeling, L. (Eds.), The science of real-time data capture (pp. 11–26). New York: Oxford University Press.
Shiffman, S., Stone, A. A. & Hufford, M. R. (2008). Ecological momentary assessment. Annu Rev Clin Psychol, 4, 1–32 [Article] [PubMed]
