Instruments for Measuring Youth Digital Media Use A Comparison of App-and Web-based Mobile Experience Sampling Tools

In the context of adolescents’ increasing engagement with digital media, accurately assessing their media usage becomes challenging. This complexity arises from factors such as heightened multitasking, shorter attention spans, and peer influence. The Mobile Experience Sampling (MES) method has emerged as an alternative in digital media research with the strong advantage of measuring media use in situ several times a day over an extensive period. However, there is a lack of research investigating how MES studies among teenagers should be set up and which specific MES tools should be used to ensure high response rates. To answer this question


Introduction
For young people growing up today, daily life is permeated by digital media in various life contexts (Browne et al. 2021;mpfs 2021), resulting in a spatial and temporal fragmentation of media use (e.g., Thulin and Vilhelmson 2019), increased multitasking (Lau 2017), and shorter episodes of media use (Siebers et al. 2021).During adolescence, young people undergo significant developmental and behavioral changes (Sawyer et al. 2018), with rapid fluctuations in media habits.These factors make it increasingly challenging to measure digital media use in youth compared to adults accurately.The Mobile Experience Sampling (MES; Larson and Csikszentmihalyi 1983) method emerged to meet this challenge.It has become a central tool in youth

Mobile Experience Sampling with Adolescents
Diary studies have been considered the gold standard of time-use media research in recent decades (Twenge, Martin, and Spitzberg 2019).As a specific form of panel surveys, diary studies collect data once a day, several times a day, or weekly, typically over a more extended period (Scherer and Naab 2013).
The approach has been consistently validated for assessing media use and other activities throughout the day, allowing for investigation of individuals' specific time and media use patterns.Participants are asked to recall short time intervals as well as the context of their media use, such as location, time, or specific mood, all of which add up to their total media time (Orben and Przybylski 2019).The method emerged as an alternative to Likert scales, which are often more general and less precise than diaries (e.g., "How often do you use your smartphone on a typical day?"), compromising the validity of results (Scharkow 2016).From a time-use perspective, daily diary studies favor investigation of any compensation for changes in the total time spent on one or more activities (Bauman, Bittman, and Gershuny 2019).
However, due to rapid digitization, the traditional diary method has revealed substantial shortcomings in the precise measurement of media use across both cohorts, youth and adults: First, diaries are traditionally in a paper-and-pencil format, which places a high burden on participants and makes completing the diary cumbersome (Chatzitheochari et al. 2018).Second, diaries are often completed at the end of the day when participants have to recall what they did, leading to biased data.Studies of diary data demonstrate that respondents tend to miscalculate their media use (Parry et al. 2021;Scharkow 2019).A once-a-day diary is prone to memory gaps in increasingly fragmented digital media use: Individuals rely on cognitive heuristics to reconstruct past experiences rather than reporting moment-to-moment thoughts, states, and behaviors (Stone, Schneider, and Smyth 2023).Likewise, a study comparing retrospective survey responses with in situ measures found that WhatsApp and YouTube users retrospectively judged their usage time to be twice as long as assessed by in situ measures (Naab, Karnowski, and Schlütz 2019).Third, time-use diaries are sensitive to so-called fatigue effects (Verbeij et al. 2021), meaning that study compliance decreases over time, as the burden and efforts required of participants are high.While MES studies similarly prompt participants several times a day over a more extended period, they are thought to decrease effort and burden for participants as compared to complete diaries by offering participants the possibility of using their own mobile devices (instead of paper-and-pencil diaries) and asking them to recall shorter time spans during the day (e.g., the last hour).
Compared to adults, adolescents' digital media use is characterized even more by fragmentation and short usage episodes, which can be linked to fluctuating distraction resulting from increased time spent on social media (Siebers et al. 2021).

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 In addition, adolescents' usage episodes are likely to switch from brief instances, such as swiftly sending a snapshot via Snapchat or BeReal, to more prolonged and intense media activities, such as gaming or video streaming.For instance, bingewatching has become popular in recent years (Anghelcev et al. 2021).
Considering this evidence, the aforementioned methodological limitations are particularly apparent in adolescents' frequent and digitized media behaviors (Parry et al. 2021).The present study addresses the unique challenges of researching adolescents' digital media use arising from their intense use of social media, video streaming, gaming, and audio streaming.Digital media are pivotal during adolescence, facilitating social interactions and helping young people to navigate physical, emotional, and behavioral changes during puberty (Sawyer et al. 2018).Moreover, youth media use can pave the way for larger shifts in technology adoption (Botterill, Bredin, and Dun 2015) and can shape later life stages (Westlund and Weibull 2013).These factors underscore why it is crucial to assess digital media use during this life phase precisely.
The MES has emerged as a new methodology for assessing behavioral phenomena particularly in human-computer interactions, health, psychology, and communication ( van Berkel, Ferreira, and Kostakos 2018).One of the key advantages of MES over diary studies or other retrospective methods is that the in situ nature of MES allows media usage to be measured in its situational context, thus maintaining the high ecological validity of the method ( van Berkel, Ferreira, and Kostakos 2018).With media use becoming more digitized and temporarily blurred, MES increases the chance of capturing short-lived and transient media activities (i.e., checking WhatsApp, unlocking the home screen, scrolling through Instagram) (van den Heuvel et al. 2021) and significantly reduces recall bias (Roekel, Keijsers, and Chung 2019), participant burden, and administrative costs for respondents and researchers (Chatzitheochari et al. 2018;Rich, Bickham, and Shrier 2015).
Due to the distinct advantages of MES, the method has been used in an increasing number of studies, particularly with children and adolescents (e.g., Beyens et al. 2020;Valkenburg et al. 2021;Moreno et al. 2012).In these studies, the variety of existing MES tools and platforms becomes apparent, such as smartphone applications (e.g., Ethica Data Services, Inc.; MeTag, Hepp, Loosen, and Hasebrink 2021) and web interfaces (e.g., Qualtrics, SoSci Survey).However, so far, no systematic approach has evaluated the pros and cons of different types of MES modes.Therefore, we compared two distinct MES tools: SoSci Survey (web mode) and Ethica (app mode).SoSci Survey is a well-established tool in communication science research that is also suitable for conducting MES studies.The Ethica app has been previously used in various studies on adolescent media use (Meier et al. 2023;Verbeij et al. 2021) and met the criteria because it offers signal-based assessment.Moreover, we could

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024implement all relevant study variables at the necessary scale levels.Following, we will discuss important evaluation criteria that can help to guide the planning of MES studies among adolescents.

Evaluation Criteria to Assess the Suitability of Different MES Modes
Research summarizing the potential of mobile research tools is generally scarce (Schnauber-Stockmann and Karnowski 2020).Moreover, to our knowledge, no systematic studies have evaluated the advantages and disadvantages of different MES modes (with either adult or adolescent participants).This is critical, as web-and app-based platforms have distinct advantages and disadvantages, for instance, in terms of installation, user interface, type of notification, and costs (Table 1), which may affect (1) data quality and (2) perceived usability. 1Costs apply to researchers only.

MES
2 Participant costs vary: For both modes, owning a smartphone is a prerequisite.Moreover, while the Ethica app itself is free, the app installation and setting adaptation can be timeconsuming (non-financial cost).
There are numerous criteria for evaluating the design and implementation of MES studies that address the reliability and validity of MES modes (e.g., van Berkel, Ferreira, and Kostakos 2018).Criteria include response delay, compliance (e.g., risk of bias due to non-reporting of events due to fatigue; Wen et al. 2017), sample size and sample representativeness, ethics and data protection, and replicability and transparency of methods.In this study, we chose to examine response rate (i.e., compliance) and response delay, as these might be directly associated with the investigated modes and software.However, we will of course reflect on the findings

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 for these two indicators against the backdrop of our underlying sample characteristics.For ethical reasons, both tested modes are similar regarding ethics and data protection, as both align with the regulations of the General Data Protection Regulation (DSGVO).In general, it can be assumed that the selected quality indicators -response rate and response delay -are appropriate to assess the data quality generated in an MES study conducted in youth samples.In longitudinal data collection, such as MES, the response rate and response delay are essential quality indicators for the obtained data (van Berkel et al. 2017;Viechtbauer 2022;Tuten, Urban, and Bosnjak 2002).Response rate is the ratio of completed surveys to received surveys and is of critical interest for accurately estimating the parameters of the sampled population (Viechtbauer 2022;Vachon et al. 2019).Generalizations about a population can only be made if those who respond to an MES beep are not significantly different from those who do not respond (Tuten, Urban, and Bosnjak 2002); Low response rates may thus indicate biased data (Kovalchik et al. 2018).In addition, more ephemeral everyday moments collected with MES contribute to a more accurate and larger picture assembled from these individual MES data points, providing more precise insights into participants (van Berkel et al. 2017).Moreover, being particularly advantageous for accurately measuring youth digital media use because of its favorable in situ design, MES survey beeps should be answered with as little delay as possible (e.g., Viechtbauer 2022;Tuten, Urban, and Bosnjak 2002).Regarding the time lag between prompting the participant to answer a survey and actual survey completion (Viechtbauer 2022), response delay is of critical interest as an indicator of whether the MES study meets the in situ criteria.High response delay to single or multiple MES beeps may, again, indicate biased data where, for example, fast MES responders may differ from slow or delayed MES responders (Tuten, Urban, and Bosnjak 2002).
Perceived usability, in turn, includes participants' perceptions and evaluations of accessibility (e.g., installation), navigation or orientation, ease of use, and design features (Missen et al. 2019), as well as their usage experiences (e.g., integration into their daily lives) and possible suggestions for improvement (Ribanszki et al. 2021).Moreover, it has been shown that perceived usability may affect response rate and response speed, which are essential indicators of data quality (Tuten, Urban, and Bosnjak 2002).Among the studies on usability evaluations of apps for children, Missen et al. (2019) conducted a two-stage study consisting of 1) a systematic review of existing learning writing apps for Android and iOS operating systems and 2) participant-based usability evaluations of a subsample of these writing apps.Interestingly, they found a significant need to improve the user experience for children.Furthermore, despite the dramatic increase in the use of apps in different research areas, Maramba, Chatterjee, and Newman (2019) highlighted in their scoping review on usability testing methods in eHealth app development the need for a comparable

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024increase in the literature on usability awareness.The authors further criticize that among existing studies, standardized questionnaires are the most commonly used method, pointing to a continued need to identify challenges and areas requiring optimization.This is critical, as children and adolescents now use their smartphones and apps for multiple activities, and their voices should be included in the research (Dias and Brito 2021).Adolescents' perceived usability of app-and web-based MES tools is hence also part of our evaluation criteria.Based on the state of research outlined here and the identified research gap, we ask:

RQ1:
How do different MES modes differ regarding data quality?RQ2: How do different MES modes differ regarding perceived usability?

Study Design
This study is part of a larger project investigating digital media use, well-being, and academic performance of adolescents based on MES data (funded by the Austrian Research Fund under grant number P34431-G).
In order to answer the research questions, we performed a multi-method study among Austrian adolescents.The study was approved by the ethics committee of the University of Vienna (#00776).The first part consisted of a one-week experimental MES study (RQ1), in which participants' time use was assessed using either an MES app or a web-based survey.Subsequently, semi-standardized qualitative interviews were used to evaluate the perceived usability of each mode from an adolescent's perspective (RQ2).This multi-method approach thus combines a data-driven evaluation of quality criteria and individual experiences with the MES study design to provide a comprehensive analysis with a focus on the young participants.

Sample
We aimed for a quota-allocated sample to ensure that the data covered different age groups, genders, and educational and cultural backgrounds.We decided to cover a rather broad range by including participants aged 11 to 19 years.This approach allowed us to investigate the usefulness of the MES modes both in early and late adolescents, as it can be expected that their media use behaviors might differ due to contextual factors (e.g., parental control) and psychological factors (e.g., media literacy).The quotas were nearly filled.Nevertheless, some subgroups in the intended sample proved challenging to reach and were therefore underrepresented

Procedure
In the MES study, participants were randomly assigned to the following groups: half of the sample completed the MES via an app (i.e., Ethica), while the other half received a web-based MES survey (i.e., programmed on SoSci Survey).For an overview of mode characteristics, see Table 1.
The basic requirement for participation in the study was access to a (personal) smartphone with an Internet connection.For respondents under 14 years of age, written consent was obtained from parents and children.For respondents 14 years and older, consent was obtained directly from the participants.Participants in the app group had to download and install the app on their mobile phones and register for the study using a QR code.Instruction materials were provided for all necessary

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024steps.Prior to the start of the MES survey period, participants received a link to a brief questionnaire that collected sociodemographic data.Apart from minor differences in layout, the MES questionnaire was similar in both modes.All participants were invited to the online baseline survey via text message.On two consecutive weekdays (Tuesday, Wednesday) and one weekend day (Saturday), participants received three notifications to complete the questionnaire (nine notifications in sum).On weekdays, notifications were sent in random time slots between 12 a.m.-2 p.m., 3-5 p.m., and 6-8 p.m. On Saturday, notifications were sent between 10-12 a.m, 2-4 p.m, and 6-8 p.m. Participants received the invitation to the brief questionnaires either by push alert (app group) or by SMS (web group).In the web mode, the survey link would expire after it was completed once.In the app mode, the survey link did not expire automatically.
Once the MES study was completed, qualitative semi-structured interviews were conducted in the follow-up phase to learn about the participants' usability experiences with the study.The interviews were conducted online or in person.Each interview started with a welcome, including an introduction for the interviewee and a note on data protection.The data were recorded on a digital audio recorder and transcribed.The detailed interview guide is accessible online on OSF via https://osf.io/9eut7/?view_only=1317fc2754c342edb0a8de0dbf4206b4.

Measures
The MES survey included 25 questions about participants' time use in the last hour.Since this article takes a strictly methodological perspective, the content-related variables will not be the presented here.The following quality-relevant determinants were assessed: Total amount of completed questionnaires: Each participant -whether assigned to the web or the app mode -received n = 9 MES beeps in total (3 beeps/day).The response rate was measured by subtracting the total number of survey requests (beeps) received minus the actual number of surveys answered per participant (M = 5.19, SD = 2.54).
Response rate (in percent): The ratio of possible survey responses to actual responses indexed each participant's response rate (M = 0.72, SD = 0.20 range: 0-1).
Response delay: The response delay was measured by subtracting the actual response time for each survey beep minus the time each beep was transmitted (M = 2.16, SD = 7.17).The participant timestamp of the surveys (start and end time) allowed us to track whether a survey beep was answered with a delay or not at all.In this study, we chose not to filter out delayed responses above a certain threshold (e.g., responding more than one hour after being prompted) as we were particularly interested in the delay in both modes and the overall response rate.

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 The qualitative interviews followed a protocol developed based on the current state of research on the usability of mobile applications (Thüring and Mahlke 2007; Missen et al. 2019).The final interview guide comprised questions on the following thematic areas: Accessibility: Evaluations of accessing the study (e.g., downloading and installing the app, configuring smartphone push alerts).
Navigation: navigating through the app or web interface (e.g., start page web or menu app, the structure of the survey, answering survey questions) Everyday integration: Integrating the MES survey into everyday life (e.g., enjoying completing the survey in the web or app mode, situational context: When did filling in the survey become bothersome, what was it like when others were there?) Optimization: Suggestions for future research.

Analysis
To perform the descriptive data analysis (mean indices, frequencies) in order to assess MES data quality, data preparation and analyses were performed in R (version 2022.12.0+353).To prepare the data for analysis, we merged MES data collected from both SoSci Survey and Ethica into a single dataset and transformed the data from wide to long format.We ensured that each participant was accurately represented in the dataset nine times during the data cleansing process.Participants' IDs were represented fewer than nine times when they missed one or more MES beeps.Moreover, some app mode participants exceeded the dataset's nine-time limit, likely due to clicking on the questionnaire at other times than after the scheduled beeps, possibly out of curiosity.In such instances, we excluded these outliers based on the recorded timestamp.Aside from this, we did not exclude participants from our analysis.The interview transcripts were digitally recorded and transcribed using MAXQDA or Microsoft Office Word and then matched to the relevant sections of the interview guide for analysis (i.e., accessibility, navigation, everyday integration, optimization).

MES Study: Data Quality
The MES survey data were examined to answer the first research question.In detail, we analyzed the performance of both modes regarding their response rate, the total amount of completed questionnaires, and response delay.

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 A descriptive analysis (Table 3) demonstrates that the response rate was higher among the web group (85.2 percent, 138 out of 162 total beeps answered) compared to the app group (57.6 percent, 83 out of 144 total beeps answered).Figure 1 presents the relative number of completed questionnaires per day across beeps.As can be seen, the web mode response rate was substantially higher than the app mode response rate and steady across all three MES days.Over all beeps, the completion rate ranged between 77.7 and 88.9 percent.In contrast, the app response rate was substantially lower, ranging between 37.5 and 81.3 percent.Despite the increasing tendency, the app MES mode never reached the level of the web-based MES group regarding completion rates.Furthermore, while the web-based graph shows a plateau -therefore not being affected by the time of the survey invitation (beep number) -we see quite the opposite picture for the app participants.Adolescents in this group often failed to answer the third beep of the day, the evening beep.
In addition to the completion rate, we further determined the response delay for each mode (Figure 2).The results show that, on average, participants in the web group opened the questionnaire link 0.76 minutes (SD = 2.50) after receiving the invitation SMS.In turn, participants in the app group needed approximately six times longer to click on the push notification on their smartphones (M = 4.52, SD = 10.93).The mean response delay in the web group ranged between 0.04 minutes (first beep) and 3.38 minutes (sixth beep), whereby the sixth beep seems to be an outlier when considering the overall trend.In the web group, beeps 3, 6, and 9 -the last beeps of each study day -show the highest average delay in minutes.In the app group, the minimum mean response was 0.55 minutes (sixth beep) and 11.3 minutes (fifth beep).Moreover, it becomes evident that the participants answering the MES study via the app often failed to answer the first beep of the day in a timely manner.Response rate in percentage per mode across beeps (1-9).Note.The response rate per beep was calculated based on the ratio of answered beeps:invited beeps.Hence, a 100 percent response rate means that all beeps transmitted were actually responded to.

Accessibility
In the first part of the interviews, respondents were asked to report on their experience accessing the MES study and the repeated MES surveys during the study week on their personal smartphones in web or app mode.In the web group, registering for the study and completing the surveys did not require any preparatory steps other than opening and clicking on the text message that contained the survey link.In contrast, app users had to perform several steps before starting the study, such as installing the app, enrolling in the study using a QR code, and setting up or allowing push notifications on the smartphone.We provided thorough PDF instructions for all the required steps.While the majority of participants found the instructions easy to follow, some remarked that getting the app to work was challenging, stating that it was "just a time-consuming process, the registration and all that […] such a complicated Internet thing.How do you describe it?[…] The user interface is complicated and there are lots of intermediate steps and so on" (app user, 19 years, male).
Most participants did not mention that accessing the study via the app was particularly onerous: "Nope, I had no problems, it was all very easy" (app user, 13 years, male).
Yet some respondents argued that they were unsure whether the installation had gone smoothly and whether everything had been set up correctly: "I wasn't sure if I was doing it right the whole time" (app user, 15 years, female).
Another participant echoed this view: "Yes, I think it worked then.If I just did everything right, because the whole time I wasn't so sure if I was doing it right" (app user, 18 years, female).
However, respondents' experiences in setting up the app varied.There was also a considerable amount of positive feedback on how easy app access was: "I didn't really have any problems.I just had to download it, then, for example, accept whether I wanted to receive messages, all sorts of things.And yes, then everything actually went relatively well and then I was already in" (app user, 14 years, female).
In contrast, accessing the web survey via the text message link proved to be a very reliable option.Overall, participants found that accessing the study was simple and clear:

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024"When I receive a message, it is quickly displayed and all I have to do is click on it" (web user, 17 years, male), "Easy.Yes, very easy [...] you just had to click on it" (web user, 11 years, male).

Navigation
In terms of navigating the web or app interface and the survey itself, participants reported that it was fairly easy.Concerning navigation on the homepage of the questionnaire in the web browser or the app menu, experiences were comparably positive among both modes: "It was actually quite clear what you had to do because when you hit the notification, there was a field where you had to click and everything was actually easy there" (app user, 18 years, female).
There were some suggestions that navigation through the app homepage was not that clear at the beginning of the study and that the participants first get used to the interface in the app: "I think I was a bit confused at first [...].There was a sort of blue box, I would say, and somehow I didn't quite understand that I always had to go up there" (app user, 14 years, female).
Another participant in the app group reported confusion about being able to access the survey interface in the app outside of the set notification schedule: "I was a bit confused at first because the app had a feature that said you could still fill in" (app user, 15 years, female).
Another interviewee said that she found it difficult to set up the app notifications on her mobile phone and that she did not receive any, which left her confused and led her to enter the app interface proactively: "Then I remembered 'I have to do this' and I went into the app and filled it out and just didn't know if I did it right" (app user, 15 years, female).
General access to the app was always available, even outside the notification schedule, and could not be controlled by the research team.This was not the case in the web mode, as each survey could only be completed once: Afterward, the link expired.While a minority of participants in the app group mentioned these more technical ambiguities, the majority of participants in both groups found navigating through the MES surveys to be simple and easy to understand.

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25. 04.2024In almost all cases, participants indicated that the integration of the study into their everyday lives was relatively neutral and overall hardly disturbing: "I filled it out every time I saw the notification [...] also on the bus on the way home or at school during the lunch break, when you can use your mobile phone" (app user, 15 years, female).Only one comment from a respondent in the app group indicated that filling in the survey was annoying and interfered with daily life: "I was never in the mood, I found it rather annoying [...] It was bothersome during gaming" (app user, 16 years, male).Some suggestions from respondents in both modes were that MES beeps were perceived as temporally inappropriate (e.g., when on the bus or in school).At this point, the survey was often answered with a delay: "For example, if I was at school for a long time and then had to complete it at home, I just wasn't happy" (app user, 14 years, male) or "when I was really tired because I had a hard day.Then I thought to myself: okay, shit" (app user, 19 years, male).
Still, most respondents reported that answering the survey questions went quickly.Some interviewees found it interesting to take part in the study: "I found it very interesting to see what you had been doing in the last hour, you don't usually think about it [...]" (web user, 16 years, female).
Moreover, interview responses indicate that for some of the adolescents, there were times when they were not allowed by parents or school authorities to use their mobile phones.They noted that at these times, it was inconvenient to answer the survey questions: "I just got the notification and sometimes didn't see it until a little later because I was at school and we're not allowed to use our mobile phones until after school" (app user, 15 years, female).
Next, we asked respondents for their perceptions about completing the survey in the presence of other people.Generally, participants were relatively undisturbed by not being alone when answering the questions: "It didn't really bother me, someone asked me what I was doing and I just explained it.But that wasn't a disruptive factor or anything" (app user, 15 years, female).

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 Finally, participants were asked how much they enjoyed completing the questionnaire (repeatedly).The responses were overall neutral to positive in both modes.The individual feedback tended to be less about completing the questionnaire in the app or web mode and more about taking a survey in general and being asked about their individual time use patterns: "Well, it was fun because I find it very interesting to reflect on what I actually did in the last hour" (app user, 17 years, female).
Moreover, responses from participants expressed that they felt they "somehow couldn't say anything wrong" (app user, 14 years, female).
Again, in sum and across both modes, respondents reported that they enjoyed participating in the study.However, there was evidence that study participation, in general, became tedious over time and that participants lacked variety in the survey questions since these remained the same throughout the study: "Yes, the first two times I found it quite good, but then I found it strange that the third time it was always the same thing [...] I wished that other questions would come up" (web user, 17 years, male).

Optimization
In the final part of the interview, participants were asked what they would improve about the web or app survey mode and whether they would like to add anything else.The need for optimization was most prevalent in the app group and related to the app's notification settings.Although we carefully programmed the notification schedule in the system, some participants were notified either outside the actual schedule or more than three times a day (due to technical errors, incorrect in-app settings, and privacy settings, among others).Two respondents reported that the app continued to notify them after the study finished: "The notifications still came after the study was already over" (app user, 17 years, female).We planned the timing of the notifications to avoid disrupting the participants' school day and free time.Some respondents in both modes, however, reported perceiving the MES prompt beeps to come at inappropriate times.When this happened, they often answered with a delay.

Discussion
This study systematically analyzed the quality of data from an MES study obtained using two different data collection tools (web vs. app) and evaluated the usability of the tools from an adolescent perspective.Although extensive research in communication science relies on MES data, the evidence on the quality of data provided by these tools and how children and adolescents perceive them is scarce.However, this

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 knowledge gap is crucial to address, as the results of this multi-method study show that different MES modes are only somewhat suitable for measuring youth media use in terms of both data quality and usability.
In MES studies, researchers aim to lose as few participants as possible when conducting extensive and complex longitudinal designs such as MES studies.Panel mortality is a major problem since it reduces study power and comes with problems in the statistical analysis of MES data (Stone, Schneider, and Smyth 2023).To handle missing data in MES studies, researchers either rely on list-wise deletion of missing beeps (leading to skewed findings when those data points are not missing completely at random or at least missing at random; Stone, Schneider, and Smyth 2023) or have to deal with complex data imputation methods (Courvoisier, Eid, and Lischetzke 2012;Sun, Rhemtulla, and Vazire 2021).Hence, even prior to the data collection, researchers should thoroughly consider how they can optimize their study setting and design in order to avoid missing data.
In this study, the response rate was higher in the web group than in the app group.Our findings indicated that using conventional web surveys with text message invitations increases the chance that adolescents will respond to MES beeps when compared to the MES apps with push notifications.Another notable difference occurred regarding the response time.Although both modes showed a mean delay of less then 10 minutes, the distribution among those answering the questionnaire via the app was much worse than in the web group.Because the benefit of MES is in situ assessment, participants must respond to the questionnaire promptly upon receiving the notification; otherwise, the in situ character fades away.Hence, the tested web-based MES survey data can be deemed more reliable than the appbased data.It should be noted that in the web group, the response delay was highest after the last beep of the day, which might be due to the characteristics of the observed group: Most adolescents are firmly bound to the rules and structures in their families.It can be assumed that the evening slot often coincides with fixed family time, such as eating dinner or watching TV, therefore hampering a punctual response to the survey invitation.In addition, the results show that using the web questionnaire gives researchers more control and direction over data entry, thereby improving data quality.
In the interviews, some participants in the app group stated that they accessed the survey interface outside of the scheduled notifications, perhaps out of boredom or curiosity.Before the interviews, we had already observed this phenomenon in the data via the timestamp variable.Unfortunately, the app did not prevent such unscheduled access to the survey.This is another drawback of the tested MES app compared to the web interface, as it introduced noise into the data that needed to be accounted for.

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024 From the adolescents' perspective, the perceived ease of use was comparable, but again, the web mode had some advantages.The observed differences are also likely related to each participant's media literacy level (Martens and Hobbs 2015): Participants in the web group required fewer technical skills than those in the app group, for example, due to differences in access and navigation.Importantly, interview responses show that a baseline level of uncertainty about the correct app notification settings (e.g., allowing the app to display notifications on the smartphone's home screen) was prevalent among participants in the app group.This uncertainty is a likely reason for the apparent differences in response delay per mode: When a notification is not displayed on the smartphone's home screen immediately after a beep has been transmitted, the probability increases that the survey, which the beep is supposed to draw attention to, will be completed late.Smartphones allow for various notification settings on the app level compared to individual settings for text messages, which will likely impact how and when notifications are displayed and noticed.
Other factors for delayed or missed MES survey beeps -resulting in missing data and thus affecting data quality -have been debated in the recent literature (e.g., Rintala et al. 2020).The qualitative interview responses provide interesting clues to possible explanations for noncompliance and response delay in different contexts: Although digital media in general, and smartphones in particular, are considered to be firmly embedded in adolescents' digital lifeworlds, their use is often regulated by parents, schools, and other adults.Hence, timely response to an MES survey beep may be impacted by rules set forth by parents, caregivers, teachers, or other authority figures or by individual circumstances related to time and media use that enable or limit smartphone access (e.g., school schedules, commuting to school, homework, mealtimes).Therefore, in planning an MES study with youth, it is essential to evaluate the scheduling of MES beeps to determine whether on-time responses are feasible for the target sample and if delayed MES surveys can still provide reliable answers to the research questions (for similar findings, see Reinhardt, Mayen, and Wilhelm 2024).Nevertheless, there was no indication in the interview responses that, for example, data volume restrictions were a limiting factor for responding (on time) to survey beeps.
Our study proves that conducting an MES study through a smartphone captures young people in their natural environment and considers their needs.In addition, the analysis of our interview data revealed that interest in the study was relatively high and that adolescents were receptive to the concept of using their smartphones to participate in the study.This notion highlights that developing MES tools and technologies, which may also provide more user-centered approaches, is a fruitful area of research among tech-savvy adolescents (see also Mallan, Singh, and Giardina 2010).

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25. 04.2024No substantial evidence has emerged to suggest that participating in the study through either the web or app mode was perceived as complicated nor that lower digital skill levels represented a burden to participants.Younger participants displayed as much technological proficiency as their older peers and experienced no difficulties utilizing either the app or web tools.Nonetheless, some reported varying levels of individual motivation to participate.A few apparently did not follow the instructions, such as immediately responding to beeps, fixing app-related bugs, or seeking guidance, perhaps because they perceived them as unimportant and simply lacked the motivation to participate.This is an important finding: Adolescents today are part of a digital generation and are accustomed to installing apps and navigating interfaces on their smartphones.However, maintaining high data quality in an MES study necessitates minimizing the burden of participation for young participants.This entails avoiding or minimizing additional time and costs associated with installation processes and tool familiarization.
Furthermore, it becomes evident that researchers intending to apply MES in youth samples must confront the challenge of handling missing data.Despite their efforts to optimize various aspects discussed earlier, such as the choice of mode and software, timing of beep notifications, and the inclusion of detailed and easily understandable accompanying text (e.g., for installation and handling), MES studies are prone to higher rates of missing data than other methods, especially cross-sectional studies.Therefore, careful consideration of strategies for managing missing data is crucial in the data planning process, since neglecting this aspect can result in biased findings (see, for example, Reinhardt, Mayen, and Wilhelm 2024).
The study was subject to the following limitations.First, our sample demonstrates a broad age range, which is both a benefit and a limitation.On the one hand, this procedure allowed us to account for different developmental stages and adolescent (media) needs.On the other hand, the sample sizes were too small to go beyond descriptive analyses and test for significant age differences.Future studies should therefore take a deeper look at the effects of age on adolescents' performance in MES studies.From a descriptive standpoint, we do not expect great differences between younger and older adolescents (see Supplement).Since we analyzed the data quality descriptively at the beep level of the MES study, the power at this level is sufficient for our purposes.However, it is still possible that the data quality in the app mode did not suffice compared to the web mode because of technical issues or lack of motivation among some individuals.While researchers using SoSci Survey can adjust all settings fairly transparently, paid apps like Ethica represent more of a black box, which in turn may also affect replicability, especially when it is unclear how background settings influence data collection.Second, our sample was homogeneous regarding educational background, as all participants attended secondary school (middle school for 10-14-year-olds or high school).Future studies

Pädagogik
Zeitschrift für Theorie und Praxis der Medienbildung Medien www.medienpaed.com> 25.04.2024should include adolescents with lower levels of education.Third, we assessed perceived usability with qualitative interviews, but there are alternatives for measuring usability that are less reactive concerning social desirability (e.g., surveys).Finally, we tested two modes against each other, but multiple additional applications and web modes exist for conducting an MES study.Even though many of them are licensed, future studies should focus on this topic by testing a wider range of MES tools to examine whether the observed data quality and usability advantages persist when comparing the web mode with other applications.Until then, however, we stand by the finding that the web mode is less error-prone.

Conclusion
Despite the growing interest in MES as a method to accurately measure the ubiquitous use of digital media by young people, there is a need for research on existing MES tools in terms of their perceived usability and generated data quality.To date, the majority of methodological guidelines in this area have focused on sample and design characteristics, such as study length, notification schemes, and incentives for participants (e.g., Vachon et al. 2019).This study demonstrates that the perceived usability of a particular MES tool can impact data quality as well.Based on our findings, we recommend using a web interface and inviting participants via SMS to take part in each survey wave rather than relying on an MES app, since the web interface (1) produces higher quality data, (2) generates less confusion regarding installation and navigation, and (3) is more cost-efficient from a financial standpoint.This knowledge is important, as delayed responses and poor compliance offset the benefits of the in situ design in MES.For the planning of future MES studies with adolescents, we suggest that researchers consider all factors that could impact study compliance -including the timing of notifications embedded in the individuals' daily schedules structured by school, free time, and family time; motivation; compliance efforts; and individual factors such as digital skills or parental rules.We advise that thorough pretesting of MES tools be carried out before initiating data collection and that researchers become acquainted with protocols for handling missing data (e.g., Reinhardt, Mayen, and Wilhelm 2024).With the findings from our study, this article contributes to a deeper understanding of what is needed to obtain accurate and precise time estimates of adolescents' media use behavior.The findings can be used to inform future MES studies of adolescents.

Mode Installation/ Preparation of Use User Interface Notification Type
Participants were recruited through a research seminar.The sample for the experimental MES study comprised N = 34 students aged 11 to 19 years (M = 15.3,SD=2.2).Based on 34 participants receiving nine MES notifications, the total number of notifications was N MES = 306.Of those, N MES = 221 were finished (overall response rate: 72.2%).After completing the experimental MES study, N = 28 participants (n app = 15, M age = 15.9,SD=2.3; n web = 13, M age = 15.1,SD=2.4) agreed to participate in the qualitative followup interview.An overview of the sociodemographic characteristics of the sample can be found in Table2.

Response rate Response delay (min)
Response rate and response delay in the web and app mode (descriptive analysis).