Leading by the Hand Exploring the factors affecting individual student engagement with self-directed mobile assessment

This paper examines the role of the individual student and his/her engagement with self-directed mobile assessment. The findings presented in this paper are based on the results of the ALPS (Assessment and Learning in Practice Settings) CETL (Centre for Excellence in Teaching and Learning) programme, a 5 year examination of the use of mobile technology for interprofessional, formative, work-based assessment among health and social care students. Analysing the results of multiple focus groups across a number of health and social care professions we found that learner engagement with self-directed mobile assessment is dependent on internal and external factors. In this paper we explore the internal factors, namely that of learner goal, confidence and stage of self-direction (Grow, 1991).


1
Introduction Should mobile technology be used in education, and if so where?What are the pre-requisites for ensuring mobile technology «works» for education?In this paper we explore the use of mobile technology for self-directed learning and assessment and consider the pedagogical preparations required by students and education providers to implement such initiatives.We hope our article, reflecting on the role of the individual in mobile learning, provides an interesting perspective on the educational and conceptual frameworks underpinning the use of mobile technology in educational settings as highlighted in this special issue.We believe that in a world where technology is interwoven into the fabric of our lives and mobile phones are ubiquitous, it is strange not to use mobile technology within formal education.Higher Education providers have a responsibility to enhance students' employability skills wherever possible; using mobile devices during work-based placement is a good introduction for students to using this technology in a work context.The value of mobile technology within healthcare settings for work purposes has already been acknowledged (Sandars and Pellow, 2006) and its use on work-based placement provides an opportunity for the student to make the transition between the use of mobile devices for social and personal use to work related usage.
Themenheft Nr. 19: Mobile Learning in Widening Contexts: Concepts and Cases Moreover, mobile technology presents education with many opportunities.It provides possibilities to customize and individualise learning opportunities for students, (Sharples, 2000, Kukulska-Hulme and Traxler, 2005, Ally, 2009) thus involving students as partners in their own education (Bruns et al., 2010).Education providers have a chance to engage with students on their terms; allowing the student to find ways to devise meaning-making and increase their motivation to learn concepts that previously may not have interested them (Pachler, Bachmair and Cook, 2010).In 2004 a Centre for Excellence in Teaching and Learning (CETL) was funded by the Higher Education Funding Council for England to examine and improve the work-based learning experiences of health and social care students.The aim of the Assessment and Learning in Practice Settings (ALPS) CETL was to increase competence and confidence of the students on graduation.Previous studies had shown that students often do not feel prepared for practice; and this has serious implications for employers and graduates in their first year of working (Goldacre, 2003a, Goldacre 2003b).In 2006 ALPS bought 900 mobile devices to distribute amongst whole cohorts of students and staff from a variety of health and social care professions.These were for the purpose of completing formative interprofessional assessments created by ALPS based on the ALPS common competency frameworks of team working, communication and ethical practice.The tools encouraged students to engage in self assessment; but also to gather feedback from work-based assessors, service users (patients) and peers.For an example of the assessment process please see Figure 1.

Figure 1. The student-led mobile assessment journey
The student identifies a learning opportunity; perhaps they have had a complex or unusual encounter where they would value feedback.e.g.Occupational Therapy Student performs an initial assessment on a patient with an eating disorder in the patients room.
The student must check with their practice assessor if they wish to gain feedback from a service user (patient).e.g.The student asked a practice assessor for permission to complete the assessment and interview the service user (patient) in her room.The assessor agrees but asks the student to have a nurse present.
The student conducts the assessments with the relevant people and also completes a self assessment section.e.g.The student carried out a self assessment and also a service user(patient assessment).
The student then completes an appropriate action plan discussing it with their practice assessor if they wish e.g.The student decides to pre book a room for the next appointment to avoid invading the service users (patients) personal space Once the assessment tool has been completed it is uploaded to an online portfolio area where the student can then discuss further with their university based tutor if they wish This is inline with O'Malley et al.'s (2003) definition of mobile learning as: … any sort of learning that happens when the learner is not at a fixed, predetermined location, or learning that happens when the learner takes advantage of learning opportunities offered by mobile technologies.
Both parts of this definition apply to the ALPS programme as the students were out with the University, rotating around a huge variety of placement locations using a PDA to complete assessments and access learning materials.

2
Self-directed Learning Mobile technology on its own is ideal as a tool to facilitate self-directed learning and, therefore, learner responsibility as it puts the responsibility for learning directly into the hands of the student.However, coupling this digital freedom with formative assessment allows the student a much greater choice in deciding when, how and from whom they gather feedback.The purpose of formative assessment is to use the feedback gathered to direct future learning, i.e. to identify and plug gaps in knowledge and skills (Black and Wiliam, 1998).There is evidence that formative assessment encourages learners to take responsibility for their own learning (Nicol and Macfarlane-Dick, 2006).Therefore, we hypothesised that providing students with mobile devices upon which to complete assessments and access learning material would encourage them to become more self-directed in their learning on placement.There is evidence that self-directed learners achieve higher grades within formal education (Shin, Haynes and Johnston, 1993) and are more able to cope with the demands of continuing professional development in later life (Candy, 1995).By implementing this initiative we were effectively asking students to take on the responsibility of becoming self-directed learners, to identify their own areas for development and assessment opportunities for the purpose of increasing their confidence and competence without any summative requirement to do so.Defining self-directed learning is complex; in this paper we will be using Malcolm Knowles definition of self-directed learning as a process: ... in which individuals take the initiative, with or without the help of others, in diagnosing their learning needs, formulating learning goals, identifying human and material resources for learning, choosing and implementing appropriate learning strategies, and evaluating learning outcomes.(Knowles, 1975, p. 18) The process outlined by Knowles has its critics; it is approached in a linear way (Merriam and Cafferella, 1991) which does not take into account the different contexts in which learning can occur.For example, learning can result from experiences in action (Schön, 1983) that may trigger a learning project (Dewey, 1938) rather than be a pre-planned activity as in Knowles' work.As our students are reliant on what happens in the workplace to provide learning opportunities through which to complete assessments from a limited range that we have set, we seem to fall short of Knowles' definition of self-directed learning; however, no learning takes place outside of a context.Our experiences are what drive us to want to learn new skills and we cannot separate our life-worlds (Pachler, Bachmair and Cook, 2010) from learning.The second issue with Knowles' approach is that it does not acknowledge the level of environmental control the learner has over implementing a solution to their self determined learning needs.In our case, students are subject to the politics, policies and practices of the workplace, where access to learning opportunity is considerably variable (Fuller and Unwin, 2004).Despite these criticisms, the process outlined by Knowles of learners taking the initiative to identify, implement and evaluate their own learning matches the ethos of preparedness for professional practice that the ALPS programme promotes, and, within our specific context of formative assessment of core competences in the workplace, this is exactly what we want the students to do.The long-term benefit of becoming self-directed, and therefore lifelong learners, is evident for health and social care students who are required to take personal responsibility for the updating of their skills and knowledge throughout their professional lives; in the short term, student responses vary significantly.Therefore, why do some students engage with self-directed learning while others do not?

Case study -Training students in self-directed mobile learning
The formative assessments ALPS asked the students to complete were: knowing when to consult or refer, working interprofessionally, gaining consent, demonstrating respect for service users and carers, and providing information to a colleague; these are all scenarios encountered by the 16 health and social care professions involved in ALPS.Each of these assessments had 4 different sections that the student could choose to complete.A self assessment, peer assessment, practice assessor assessment and service user (patient) assessment.125 students from Medicine, 32 from Diagnostic Radiography, 40 from Occupational Therapy, 50 from Speech and Language Therapy, 22 from Dental Hygiene and Therapy and 40 from Physiotherapy, 3 from Social Work were given ALPS assessments on mobile devices.
Due to the enormous number of potential assessors involved in the completion of ALPS assessments (service users/patients, carers, peers and practice assessors) we realised that the only practical way to ensure that the process was a success was to make it student-led.This meant that the students were completely responsible for the entire learning process, from identification of an assessment opportunity to evaluation and action planning.Thus, the students were taking on a significant part of the role of the teacher.This brings with it pedagogical, ethical and practical considerations and requires a fully integrated teaching and training solution.We called this the students as teachers programme.
Each student attended a training session lasting approximately one and a half hours, though this varied dependent on timetabling and course structure.
The session was devised to present both the pedagogy and the technology as seamlessly combined.We could not assume a level of competence or awareness of reflective practice or technical ability so we designed a training programme that incorporated not only how to use the technology, but a protocol for its use in clinical settings and an overview of learning preferences to consider when demonstrating the use of the mobile device (see Figure 2).The students were required to roleplay an assessment scenario and demonstrate the use of the device prior to the end of the session.
During the session students were asked to attend a focus group following their placement to report their experiences of using self-directed mobile assessment in a work-based setting.• e-portfolio guide -for both mobile and online assessments (video, handbook and/or live online demonstration/practical if in computer cluster) • Guide to using the PDA in practice (handout and/or slides) • There is also a comprehensive e-learning module that can be used to cover all aspects of the ALPS training.This can either be completed in the classroom setting in an IT cluster, with tutor or lead present to answer any questions, or at the student's convenience.
A typical ALPS training session should consist of:

Students' response to training
Following training students were asked to volunteer to attend focus groups to share their experiences of mobile learning and completing assessments in practice.Focus groups were held following the completion of each placement with representatives from each profession.On the whole most students liked the training that was given and the accompanying documentation Oh I thought it was really useful, it was quite clear, really clear and going through everything.And the instructions were quite clear and didn't -even going through it once I had a good idea of what to do so, I tried it (Student 1) … all the instructions and stuff like that are good [group agreement] … and the information (Student 2) However, for some students there was a variance in how they felt about the training, even within the same training session!These two students attended the same training session and these are their responses: ...had the device just chucked at me … (student 3) ...handouts are clear and explained everything well … (student 4) What we must ask is why did two students have such different responses to the same experience?As both students were treated in the same way and received the same level of training this difference in response can only be due to the individual student's attitude toward the concept of some aspect of self-directed mobile delivered formative assessment or their understanding of this process.Whilst we acknowledge that other possible influences effect student response; for example the role of technology in the student's life, their digital confidence and beliefs regarding appropriate use of mobile technology (Sharples, 2006) we believe that deeper explanations lie in the student's attitude towards learning and responsibility; their «readiness» or ability and motivation to be self-directed (Guglielmino, 1997).
Whilst Knowles (1984) has a defined notion of the self-directed learners attributes, how a learner achieves these characteristics is less well defined, there is an assumption that as young people mature into adults their ability to learn independently and become self-directed learners matures along with them.However, it can be argued that learning is situational (Lave and Wenger, 1991).
For example a student can be self-directed when gathering and selecting relevant information to craft an essay, but very dependent when on work-based placement.Grow (1991) offers the notion of stages of self-direction in his staged self-directed learning model (SSDLM) (see Figure 3).The learner moves (or does not move) through the stages as they develop knowledge and skills.It is possible to be a stage 1 learner in one subject and a stage 4 learner in another due to the situated nature of learning.Grow states that he makes certain assumptions in his model, for example that there is nothing wrong with being a dependent learner (though he does note it can be a serious limitation) and that self-direction can be taught.Is this in fact the case?Are there learners that remain dependent learners all their lives?Surely, if learning is context specific no-one can remain dependent all their lives in all areas?Can self-direction be taught, or is it in fact reliant on personal characteristics?(Brockett and Hiemstra, 1991).Additionally there must be some underpinning conditions required for learners to learn at all, infrastructures and minimum requirements, which are not acknowledged in the model.Spears and Mocker (1984) in particular emphasize the importance of learner environment as a condition affecting self-directed learning.
On the face of it the model can be seen as an over simplification of what is in fact an extremely complex combination of variables including learner dispositions, environmental circumstances and politics, however it sits well with the literature on apprenticeship where a staged approach to the development of professional skills and attitudes is often used (Lave and Wenger, 1991;Billet, 2003;Collins 1989).
Additionally the potentially linear appearance of the model is counteracted as in fact Grow acknowledges that learners can move between stages and regress under certain circumstances.It is also important to note that stage 4 learners do not necessarily have complete control of their learning; in formal education frameworks and choice are provided (Guthrie et al, 1997, Temple & Rodero, 1995).
It is possible that the student's stage of self-directedness (for example stage 1) is one variable that dictates the student's response to the concept of conducting ALPS mobile assessments.This and other possible explanations are explored in the results and conclusion section of the paper.

Student responses to using self-directed mobile formative assessment during work-based placement
We have established that the students' perception of training can vary considerably.However, it wasn't until students were actively engaging in this process that their ultimate experiences and attitudes were realised.
The focus groups we held following student placements were audio recorded and transcribed.The transcripts were analysed using thematic analysis.Whilst the majority of feedback was concerned with practical issues, there were always extremes of experience-students within the same profession and learner group  In this stage, learners have skill and knowledge, and they see themselves as participants in their own education.They are ready to explore a subject with a good guide.They will even explore some of it on their own.But they may need to develop a deeper self-concept, more confidence, more sense of direction, and a greater ability to work with (and learn from) others.Stage 3 learners will benefit from learning more about how they learn, such as making conscious use of learning strategies.
Stage 4: Learners of High Self-Direction Self-directed learners set their own goals and standards-with or without help from experts.They use experts, institutions, and other resources to pursue these goals.Being independent does not mean being a loner; many independent learners are highly social and belong to clubs or other informal learning groups.
loved the assessments, or hated them; they liked the mobiles, or thought they were a waste of time.Due to this wide variety of responses it became apparent that the individual learner and their personal experiences were key variables within the acceptance or otherwise of self-directed mobile assessment.Therefore, we examined this further and determined an overarching set of concepts that influenced the take up of self-directed mobile assessment.These are the themes we identified as relevant to this article.
• The effect of learner goal on engagement with self-directed mobile assessment.• The effect of SSDL stage on engagement with self-directed mobile assessment.
• The relationship between self-directed mobile assessment and student confidence.

The effect of learner goal on engagement with self-directed mobile assessment
A potential factor affecting the uptake of self-directed mobile assessment is the goal of the individual student.We were just really busy all the time there was no time for that and it didn't really serve any purpose, any meaningful purpose to me or to them.(Student A) This student's goal is to pass the placement, a basic, but imperative goal for all students.The student has made a value judgement regarding the process, despite tutor advice.This is probably because the benefits of completing the assessments are not immediately obvious.Student B's goal is to improve their skills and working practices.They have seen the value in the assessment process.This does not mean that the student is not also concerned with passing the placement; but they aim to achieve a wider set of goals compared with student A.
… we have to compile a CPD folder I think it would be really good to add to your reflective pieces erm to show how service users view you as a t* (profession)(Student B) Student B is considering themselves as a professional, looking beyond the placement experience to a time when they will be personally accountable to service users (patients).Personal dispositions (Perkins, Jay and Tishman, 1993) play a role in learner goals, their educational background, cultural norms and values and the life-worlds (Pachler, Bachmair and Cook, 2010) to which they belong to all shape how they feel about learning and their pre-disposition to self-direction (Brockett and Hiemstra, 1991).Student A is not necessarily at a lower stage of the SSDL model than student B. It is possible to be at stage 4 of the SSDL model and actively choose not to engage in formative assessment but to focus on summative assessment.Both types of learner in this example have different ultimate goals; and these goals clearly have an influence on the uptake of self-directed mobile assessment.Whilst this is not an issue in itself, potentially student A may feel less prepared for practice on graduation due to their linear approach to their studies.

The effect of the SSDL stage on engagement with self-directed mobile assessment
The student's stage of maturity within the SSDL model has a direct effect on their engagement with the mobile formative assessments.The focus of the assessment is to gather feedback, helping the student improve their skills in a safe environment.
However, here we see a student that clearly sees the assessments as just something else to do on placement, a tick box exercise.
I was doing the assessments I was getting into the swing of it but the questions there's too many that needs shortening because I think that will probably win people over as well because then it's not as time consuming because I did quite a few and I was knowing what was coming next but it's still that seem to be asking the ins and outs of things that I don't know whether a tick box would be quicker just to do you know what I mean (Student C) If a student does not engage properly with the assessment, it negates the purpose of the assessment.This is an example of a stage 1 learner from Grow's SSDL model: dependent on the teacher's instruction.They have completed the assessment, because they were told to, without actually understanding the aim of the process.The student has highlighted physical obstacles i. e. number of questions and time taken using the device, as a barrier to use but have failed to see any value in the assessments above and beyond the practical issues.A better understanding of the importance and value of the assessment may have altered their perspective on and patience towards the physical issues of using the tools as demonstrated in the MEDS evaluation of using mobile devices with students with disabilities.(Dearnley, Walker and Fairhall, in press) This type of student is likely to experience the most difficulty and anxiety in completing assessments as the assessment process involved identifying a learning opportunity.Identifying appropriate assessors and reflecting on feedback critically require a stage 3, intermediate level of self-direction.Without a supportive and directive assessor, these students cannot hope to benefit from the assessment in the way it is designed to be used, and the assessments just become another task that the student has to complete on an already busy placement, potentially jeopardising the student's ability to learn the key lessons from experiences.
In this way, the assessments designed to promote self-direction and ultimately confidence and competence could actually prove detrimental, and result in the exact opposite-unconfident, incompetent students.
In contrast here we can see a student at stage 4 respectively of the SSDL Model.This student has recognised the personal responsibility for learning required from them, they have identified that they have a choice in how they use the assessments to best aid their development.
The process of reflecting all the time is important; it is good to have something which makes you do it more often.(Student D) Whilst it is right that young people take more responsibility for all aspects of their lives as they mature, the level of responsibility we can reasonably expect from all students is less straightforward.In this example we can see that the stage 4 learner is planning for the future and will likely achieve their goals.But what of the more dependent stage 1 learner who can't see the point of formative assessment?In the conclusion we will consider how we can support these early stage learners to engage with the process and ensure they have as equitable a student experience as their more mature self-directed colleagues.

The relationship between self-directed mobile assessment and student confidence
A degree of confidence is needed to try the self-directed mobile assessments initially, as the student is required to pro-actively approach others for feedback.Not all students felt comfortable doing this.
… anything that takes longer than filling out forms is just going to annoy them.You feel like you are just tip-toeing around.(Student E) Some students took what they saw as a less threatening route by approaching peers for feedback initially.Even though the assessments were formative and would not, therefore, affect their ultimate summative grades, the students perception of their practice assessor was tied to this summative grading.Therefore, getting feedback from peers was seen as preferable.
I think a benefit is probably feedback is less threatening in a sense like when if your educator sort of says you kind of think is that going to affect my grade is that where as the peer assessment they can always pull you on the side and go I did noticed you know what I mean so I think there is a lot of benefits to it (Student F) This student feels more in control of their learning when they have a choice of who to approach and when.For this student, assessor-led assessment is more daunting than student-led.Is this to do with timing?In this case the student chooses whether they are «ready» to complete an assessment; it is not at a dictated time.
Alternatively, is it because the assessment is with a peer and, therefore, a «safe» assessment?

Other factors affecting up take of self-directed mobile assessment
We cannot, of course, ignore the external factors influencing the student uptake of self-directed mobile assessment.Several students encountered resistance to the use of mobile phones in healthcare settings and some were flatly refused when they asked to use them.This could be a historical cultural resistance based on past legislation where mobile phones were banned in hospitals for fear of interference with electrical equipment (a fear largely disproved by the Department of Health report of mobile phones in hospitals (DoH, 2009) or due to the more patient centred concerns such as dignity, confidentiality and ethics which still remain policy concerns.The resistance could also be due to concerns regarding professional image: health professionals may feel that patients would perceive them to be rude or not concentrating on them if using a mobile device during a consultation.Whatever the reason, the powerful influence of the learning environment and practice assessors therein on the learner cannot be underestimated.Whilst out of the scope of this article we intend to consider the external influences on learner take up of self-directed mobile assessment in future work.
5 Conclusion Does mobile technology empower students to become more self-directed in their learning?We would argue that mobile technology provides the opportunity to facilitate self-directed learning; but that take up is dependent upon several factors.Underpinning everything is the student's understanding of their role within learning: do they accept they have an active and influencing role in the process?(Taylor, 1995) In addition, the student's individual goals, their stage of development on the SSDL model and confidence to approach others for feedback initially all affect student uptake and engagement.In this way more confident and developed learners seem to get far more out of the self-directed mobile assessment process than their less confident counterparts.
What we must ask ourselves is, is this ethically sound?By introducing such innovative processes are we creating a digital and pedagogical divide?Training is a possible solution with which to address this potential divide; but the content is vital.It is not enough to train students to use mobile technology and provide them with pedagogical reasons for doing so.We must also consider the implications of mobile technology: it necessitates a degree of self-directedness.Lowry (2006) makes several suggestions on how to facilitate self-directed learning including helping the learner to recognise the initiating point for a learning lesson and teaching needs assessment techniques.These two suggestions would add significant value to our future training programmes.Lowry also identifies the need for students to appreciate that they can act on the world and change it, which is a key paradigm shift for many learners.Another potential solution is peer support.When training students it was clear that there was greater group engagement in the process when those who were less mature or confident had the opportunity to learn from and alongside those students who ‹got› the process.This reflects Assinder's (1991) findings that a range of ability in class increases understanding among the group as a whole.The sessions where there was an opportunity to properly role-play and discuss the implications of the assessment resulted in a better, more holistic group participation in the training.By combining Lowry's suggestions to facilitate self-directedness among learners with a programme of peer support into training we can improve students' preparedness for practice, and encourage all learners to make the most of self-directed mobile learning.However, ultimately the student enters the work placement alone and we must be realistic regarding the degree to which we expect students to challenge a risk averse environment like the NHS when they inhabit a relatively insignificant place in it.Over time we hope that the acceptability of mobile technology for education purposes will help to change attitudes among healthcare institutions and workers, making this a less challenging process for students.Shin, John H.;Haynes, Brian;Johnston, Mary E. (1993)

Figure 2 :
Figure 2: Outline of training available from the ALPS Trainer Handbook Training packages available include: • An introduction to the purpose of ALPS (slides) • Basic PDA functionality training-Quick Guide to the PDA (slides) • More advanced PDA functionality training-Getting the most out of your PDA (slides) Assessment Suite training including: • Assessment tool guide (video and/or handout)• e-portfolio guide -for both mobile and online assessments (video, handbook and/or live online demonstration/practical if in computer cluster) • Guide to using the PDA in practice (handout and/or slides) • There is also a comprehensive e-learning module that can be used to cover all aspects of the ALPS training.This can either be completed in the classroom setting in an IT cluster, with tutor or lead present to answer any questions, or at the student's convenience.
1.An introduction to the purpose of ALPS 2. How and when the mobile devices and/or tools will be used in the context of their placement/programme of study 3.A clear expectation of how many assessments students are expected to complete and how often tutors will check and respond to assessments within the e-portfolio 4. One of the PDA training packages if using mobile devices 5. Assessment suite training 6. Explanation of protocol for use of PDA and assessments in practice settings 7. Questions www.medienpaed.com> 30.6.2011

Figure 3 :
Figure3: The staged Self-directed Learning Model-Grow, 1991 p. 129 Stage 2 learners are «available».They are interested or interestable.They respond to motivational techniques.They are willing to do assignments they can see the purpose of.They are confident but may be largely ignorant of the subject of instruction.These are what most school teachers know as «good students».