Evaluating Mobile Learning Practice Towards a framework for analysis of user-generated contexts with reference to the socio-cultural ecology of mobile learning

Against the conceptual and theoretical background of a socio-culturally orientated approach to mobile learning (Pachler, Bachmair and Cook, 2010), this paper examines the evaluation of user-generated contexts by referring to an example from the use of mobile phones in schools. We discuss how mobile device-related, user-generated contexts around structures, agency and cultural practices might be brought into a fruitful relationship with institution-based learning. And, we provide categories for evaluating the use of mobile devices to generate meaning from and with fragmented and discontinuous media and modes at the interface of learning in formal, institutionalised and informal, self-directed settings. The evaluation criteria build on the framework of a socio-cultural ecology of mobile learning developed by the London Mobile Learning Group.


1
Introduction In our society characterised by fragmentation (Schulze, 2000) and individualisation (Beck, 1986), consumption as well as information and knowledge delivery and production (see e. g.Pachler, Bachmair and Cook, 2010) are «‹chunked›, structured and connected in very different ways from the lecture, the web and the book» (Traxler, 2010, p. 108; see also Hug, 2007).Yet, information and knowledge cannot just be ‹consumed› but have to be filtered and transformed in (individually) meaningful ways.Mobile devices as multimedia computers provide functionalities that allow users to deal with such distributed and fragmented information and knowledge -be they related to society or to media structures.Mobile devices have become more and more embedded in the life-worlds of users; they have become normalized in everyday life.Because of the degree of integration they have achieved, they can be considered as important cultural resources; resources, which embody social values and carry ideological baggage (see Bruce and Hogan, 1998) and resources which are governed by social structures, cultural practices and user agency (Pachler, Bachmair and Cook, 2010).As convergent tools, they also allow for complex networking activities distributed across time and place.Mobile devices are attractive to us from an educational perspective Themenheft Nr. 19: Mobile Learning in Widening Contexts: Concepts and Cases

 
given the affordances they provide for meaning-making, for engaging with, and for mediating the world around us as well as for communicating with it (see Kress and Pachler, 2007).What is particularly striking, we find, is their increasing portability, functional convergence and connectivity, which affords a potential for social interactivity, ubiquitous information retrieval, processing and exchange as well as context sensitivity and location awareness.And, there is a noticeable trend in the reduction of cost of hardware and services as well as in personal ownership and attendant familiarity with, and personalization of them.Related to this trend is an increasing level of expertise of young people in mobile device use in the context of their participation in mass communication and their media use.In recent publications, we have outlined not only the concept of a socio-cultural ecology of mobile learning in order to frame phenomena around the use of mobile devices from an educational perspective, but we have also described the different aspects of this concept against the background of the appropriation mechanisms of users and learners (see e. g.Pachler, Bachmair andCook, 2010 andPachler, Cook andBachmair, 2010).In this paper, we focus on a dimension that we believe remains in need of more research, namely the evaluation of user-generated contexts.By using mobile technologies and convergent media, and by working with pieces and fragments of (distributed) contents and information, learners generate contexts.Such contexts are situations in which learners use and ‹appropriate› resources (see Pachler, Cook and Bachmair, 2010 for a detailed discussion of the notion of appropriation) that are available to them at the time of use in order to make meaning.Of particular relevance in this regard is the ‹chunked› nature of content (Traxler, 2010, p. 108).Content, be it ‹continuous› or ‹discontinuous› in its mode of representation, can be seen to provide pieces of information and embedded links to other content and contexts that are ‹filtered› and picked up by media users for the purpose of construction of individualised meaning -irrespective of the intention of the content makers and providers.We argue here that the notion of discontinuity has a key function in the appropriation process through reading and the production of contemporary, i. e. multimodal, texts as well as in meaningmaking and learning processes, especially with mobile multimedia and multimodal devices.Text for us is not only written or spoken text, but also media, modes, genres, and -referring to Fiske (Fiske, 1989) -any other material, object or concept available.We view discontinuity as a cultural phenomenon that is particularly prevalent in the media and to have considerable implications for meaning-making and cultural practices such as learning and literacy, particularly in the context of mobile device use.Discontinuity in the context of learning with mobile devices we consider to be a certain break in consistency.This refers to sequencing and time line but also to aspects such as (distributed) authors, plot, and mainly in location of resources etc. Discontinuity is a characteristic of learning and meaningmaking in our information society where learners are confronted with the need to construct coherent and ‹continuous› meanings from discontinuous resources.These resources are then objectified in the form of different kinds of texts.In the following, we briefly explore the key components of the socio-cultural ecology of mobile learning.After introducing an example of the production of micro-content by using mobile devices, we discuss the ‹chunked› and fragmented nature of information that is available to learners from different contexts as well as learners› ability to use these resources for the construction of (individualised and meaningful) contexts for learning.And, we provide criteria for the evaluation of mobile learning practice that refer to the socio-cultural ecology of mobile learning as meta-level as well as to user-generated contexts that can be described -in relation to the socio-cultural ecology -micro-level.

1
Theoretical background

A socio-cultural approach to learning with mobile devices
We take the view that ‹mobile› learning, or for that matter ‹e› learning or ‹ubiquitous› learning etc., do not point to a different kind of learning but, instead, to different conditions and environments of learning (Kress and Pachler, 2007).Mobile devices are part of users› creation of contexts and enable them not only to connect different contexts but also to create their own contexts for learning (see e. g.Pachler, Bachmair and Cook, 2010;Cook, Bradley, Lance, Smith and Haynes, 2007;Luckin et al., 2010).Also, we consider mobile learning to be about understanding and knowing how to utilize our everyday life-worlds as learning spaces.Given this framework, we do not subscribe to productivity-orientated and transmission-based notions of mobile learning (see also Pachler, Bachmair and Cook, 2010).Instead, we view it as concerning the processes of coming to know, and of being able to operate successfully in, and across, new and ever changing contexts and learning spaces with and through the use of mobile devices (which also gives evidence to links between mobile learning and concepts such as formation (Bildung) (see e. g. (Pachler, Bachmair and Cook, 2010;Bachmair, 2009Bachmair, , 2010)).Therefore, from our perspective mobile learning is not primarily about technology.Instead, we focus on the process of meaning-making, particularly in the context of media use in everyday life, in semiotic terms between the making of signs and the making of concepts (Kress and Pachler, 2007).User-generated content and contexts for us are two significant dimensions of the mobilization of mass media.Both are the result of semiotic work: that is, purposive work with meaning resources (see Kress and Pachler, 2007).In our work, we focus on aspects of a socio-cultural ecology (see Pachler, Bachmair and Cook, 2010), with the learner at the intersection of: • structures (convergence, milieus, fragmentation, provisionality, discontinuity), • agency (appropriation, meaning-making, a new habitus of learning, naïve, native, expertise) and • cultural practices (normalization, self-expression, communication, user-generated content and contexts).

Focus on appropriation and meaning-making
From a socio-cultural perspective on mobile learning, the question about content delivery turns into a question about appropriation.This change in the understanding of media reception was described by Stuart Hall's ‹encodingdecoding model› (Hall, 1980).This model helps us understand the dynamics of production and reception of media and content in mass-communication.It views media reception as a culturally situated and subjectively meaningful activity.Also, it relieves the recipients from passivity and affords them the role of active and constructive individuals who engage in meaningful media reception and the production of meaningful individualised contexts.Finally, the model affords media users an important role in the construction of contemporary landscapes of masscommunication.
Adopting a contemporary understanding of the structuration of society and participation mechanisms, the socio-cultural ecology draws on Anthony Giddens who, like Stuart Hall, is a key player in the field of Cultural Studies.According to Giddens (Giddens, 1997) structures leave their marks in people's agency and, conversely, people's agencies produce structures.This ‹dualism› seeks to express the interdependence of agency and structures and their interrelationship; the two dimensions -agency and structure -can be separated for analytical purposes only.
Drawing on Hall as well as Giddens, we view media users as ‹consumers› who engage in appropriation -which can be seen as an instantiation of learning.We consider appropriation as processes of «interaction, assimilation and accommodation as well as change».Thus, appropriation is «closely linked to learning […] understood as a process of meaning-making within social structures, cultural practices and agency» (Pachler, 2010, p. 162).Appropriation relates to all kinds of activities and interactions of people that aim to deal with information and media: «appropriation is a theoretical response to a society of individualised risks, which no longer has school learning as its preferred process of appropriation.»(Pachler, Bachmair and Cook, 2010, p. 224) In the process of appropriation, users select those resources that are relevant for them in their meaning-making process against the background of their individual history with reference to their agency and cultural practices.In so doing, they construct new structures and contexts which are, in turn, meaningful in relation to already existing structures and in relation to the users' life-worlds.As mentioned already, media users are confronted with pieces and fragments of information.Discontinuous text structures find their reflection in discontinuous text reading as appropriate agency and cultural practices.‹Multi-tasking› as one example is an attendant pattern of media use that combines activities such as communication in a chat on Facebook whilst listening to music and doing one's homework (see Moser, 2008a, p. 290).Phenomena of dealing with information or, using the language of a socio-cultural ecology, cultural practices (i.e. routines in relatively stable situations and contexts) such as ‹multi-tasking›, are evidence of orientation strategies which address the requirement for ‹selection› and ‹choice› in the process of dealing with fragmented pieces of content and information in a subjectively meaningful way.Through these selection and navigation strategies users express their agency and, through them, cultural practices become ‹materialised› or objectified -either as objects or as ‹spaces› such as contexts.

User-generated contexts: a micro-framework for analysing mobile learning
Discussions around user-or learner-generated contexts are key within the field of mobile learning (see e. g.Brown, 2010).Referring to Paul Dourish (Dourish, 2004) we understand context as dynamic and created in (social) interaction.Situative and appropriate uses of resources to establish contexts and the dynamic nature of such contexts imply challenges for their evaluation and have consequences for replicability and transferability.Firstly, contexts are often perceived as limited in space and time, and thus ephemeral, as well as, at the same time, implying subsequent activities.Secondly, contexts refer to resources such as tools, structures, agencies etc. as well as to themselves.Thirdly, the subjectivity as well as skills, knowledge and understanding of the learners generating contexts are central components of contexts.In a recent paper, Rosemary Luckin widens this view up to «across multiple physical spaces and times» (Luckin et al., 2010).Together with Luckin, John Cook (Cook, 2010) considers context not only limited to one specific place or a specific time frame, but the opportunity to learn across contexts and to create contexts within contexts.
In relation to mobile learning, concepts of, and theories around user-generated contexts are relevant for several reasons: • they move the focus away from user-generated content and the idea that learning tools or given learning content are central within the learning process; • traditional learning materials such as books are no longer the only resources for learning.Agency as well as technologies, structures, cultural practices, networks, information etc. become relevant as resources for learning; • context models meet contemporary developments in mass communication as well as an understanding of learning as meaning-making within formal and informal structures, as both aspects focus on the user and the content and contexts s/he selects and produces; • because contexts are not necessarily related to specific places, spaces outside the classroom become learning spaces; • users/learners act flexibly and are able to adopt available resources and their affordances to specific situations of contexts; and • the concept of user-generated contexts opens up the perspective on the learners' everyday life and to ‹informal› contexts and activities.

Learners generating contexts ‹on demand› by using mobile technologies and text fragments
One example from a mobile learning project in a Swiss secondary school, which is described in detail in Pachler et al. (Pachler, Bachmair and Cook, 2010) as well as in (Seipold and the London Mobile Learning Group, LMLG, 2008-2010), shows how information from informal (learning) contexts can be combined with learning resources traditionally used in schools and how these learning resources combine different contexts and create new contexts.
The ‹Handy› project was realised by a teacher, Rolf Deubelbeiss, in a Swiss private secondary school with a specialism in sports (Nationale Elitesportschule Thurgau).
The mobile phone was used as a learning tool as well as a topic of inquiry.The aims of the project were to educate students about and to support them in the use of their mobile phones.For the practical part of the project the students used their own mobile phones.There was no external financial or technical support.Students were asked to work on a topic with a view to producing ‹microlearning contents›.
Students were free to choose the school subject, as well as the media format (film, picture, sound or text).The micro-content in the form of MMS (Multimedia Messaging Service) was saved as draft and distributed via Bluetooth to the project coordinator's mobile phone.He revised language and orthography, and uploaded the small units with his mobile phone to a public weblog.Some of the pictures used by the students were already available on their phones and thus not produced explicitly for the project.Other pictures were taken from a textbook.In most cases the project coordinator had to take a lead role and direct students towards being ‹creative› in composing the micro-content units.The weblog, as public place to store students' learning units, was produced by the project coordinator.Students could access the material, download it and use it as small learning units.Due to lack of space we can give only one example here of the learning material produced by students.This example consist of different sections which are subsequent and which belong together.The single pieces of information -a picture and written text -are from different contexts of the learners' every-life: the picture is strongly related to aspects of life outside the curriculum and it refers to specific social situations as well as to consumption.Looking at the picture at face value it is difficult to bring it into a direct relationship with school, the curriculum and learning.The narrative that emerges around the image is at a distance to the curriculum -but of meaning and importance for the pupils against the background of their individual lives.In order to fulfil the learning task set by the teacher, the learner reverts to a mode of representation that is typical of, and characteristic for institutionalised learning: continuous written text.It is the text that carries much of the meaning.The content is obviously linked to school and learning: the construction of this short narrative, which tells about a boy at the station makes use of features such as example, explanation, question & answer.
The ‹reader› has to construct a relationship between the image and the written text but the framing as traditional grammar exercise provides coherence and continuity related to fragments that are originally from different contexts, i. e. school and everyday life.As a consequence, the learning unit as a whole tells a totally different story than its constituent parts: the learner tells us something about himself in relation to school learning.He is able to deal with learning material and content in terms of the curriculum and in meaningful ways.He is an expert in a specific subject domain and is able to provide this knowledge in a structured way to others.
He is an expert in modes of representation prevalent in his everyday life as well as in modes that are characteristic for school and school-based learning.He knows how to combine text-fragments and pieces of information in order to produce a coherent and meaningful learning object.Also, he is able to transform subject experiences, knowledge and meanings into objective and meaningful products.By doing so, his knowledge is reconfigured and fits into the conceptual frame of the school and the curriculum.He is constructing user-generated contexts.
3 Criteria for the evaluation of mobile learning practice Aspects that emerge from this example need a framework that allows systematic evaluation.So far, we have outlined a model for the description and analysis of mobile learning practice on the basis of the socio-cultural ecology of mobile learning (see e. g.Pachler;Bachmair and Cook, 2010;Seipold, Pachler and Cook, 2009;Seipold and Pachler, 2009;Seipold and Pachler 2010,).This framework covers some of the aspects that we consider to be relevant to describe learning that emerges through the use of mobile devices.At the time of developing the following categories, we aimed to find a framework that allowed us to (a) describe learning with mobile technologies according to our conceptual framework and to (b) find areas that would allow us to widen our perspective on learning with mobile devices as well as (c) to enrich our conceptual framework by aspects that emerge from the practical use of mobile devices for learning.The concepts, theories and questions covered in the following we considered as useful to approach these our aims.In its original iteration, this framework refers to the following five categories: Also, we referred in an early stage of our research to the formal aspects ‹place› (inside school -outside school) and ‹didactic design› (‹closed›, i. e. teacher centred -‹open›, i. e. learner centred and constructivist).These two areas are not meant to be criteria for evaluation, rather initial ideas about the didactic design of mobile learning practice.However, even if this framework, which affords descriptive and hermeneutic analysis, was useful for our initial attempts to shape the socio-cultural ecology by mobile learning practice, we recognise that it lacks a clear operationalisation which allows for scalability.Another scheme that attempts to allow a systematic approach to analyse learning with mobile devices are the ‹four parameters› (Pachler, Bachmair and Cook, 2010, p. 298).Each of these parameters has two poles which represent school and everyday life and attendant practices, activities, assumptions and modes: • Combined, the ‹framework› and the ‹parameters› provide orientation in the form of a conceptual framework that considers any kind of resource available for meaning-making, i. e. learning, and tries to elicit tensions that emerge within societal and technological structures but which can, at the same time, be taken as opposite poles between which mobile learning practice can be focused.The categories proposed above allow for the analysis of mobile learning practice on the basis of a socio-cultural ecology of mobile learning.However, they don't answer the question how to evaluate user-generated contexts.In his recent research, Cook describes contexts as a micro-dimension of a socio-cultural ecology of mobile learning (Cook, 2010).With his ‹typology-grid› for analysing mobile learning practice, Cook combines the key components of the socio-cultural ecology of mobile learning cultural practices, structures and agency with a formal description that attempts to cover the ‹innovative› potential inherent in the learning design and ‹micro dimensions› which cover ‹user generated contexts› with aspects that relate to categories such as ‹active learning›, ‹reflection› and ‹attention›.By providing some key questions, the focus is on the interplay between ‹intervention and innovation› and ‹cultural practices›, between ‹intervention and innovation› and ‹structures› and between ‹intervention and innovation› and ‹agency›.
Cook also suggests the following questions: • Which cultural practices does this intervention or innovations relate to, build upon, challenge etc? • What Structures does it utilise?Are these ‹standard› or ‹bespoke›?• How does Agency (human capacities to act in the world) affect the intervention, or how is the intervention dependent on Agency?
Our attempts to develop a framework for analysis and planning of mobile learning practice on the basis of a socio-cultural ecology of mobile learning, therefore, cover the following dimensions and concepts: • a conceptual and theoretical framework: structures, agency and cultural practices; • an analysis/planning tool with a focus on the interplay between school and everyday life: the four parameters; and  (Cook, 2010) • an analysis/planning tool with a focus on the interplay between intended and actual use of mobile devices and the key components of a socio-cultural ecology of mobile learning and contexts.
Cook suggests that the ‹micro dimension› around user-generated contexts should focus on the appropriation mechanisms of learners.Accordingly, as our example has shown, a further step towards a socio-culturally informed evaluation scheme for mobile learning practice should be centred around appropriation as key reference point in the dynamic around structures, agency and cultural practices initialised and realised by the learner.This would also strengthen the link to concepts of contexts such as Dourish's notion of context as constructed through social interaction.

Conclusion
In this paper we explored categories for the evaluation of mobile learning practice across formal and informal settings which are based on our socio-cultural ecology of mobile learning.We proposed a perspective on the engagement with discontinuous (text) resources as context generation.In particular, we examined some characteristics of multimodal text production by learners and their potential for continuity and convergence by means of user-generated contexts.And we discussed some criteria for evaluation of mobile learning practice.We are aware that the categories provided are all but first steps towards a systematisation with a focus on appropriation by, as well as agency and cultural practices of learners.

Figure 1 :
Figure1: Key components of a socio-cultural ecological approach to mobile learning(Pachler, Bachmair and Cook, 2010, p. 25) The first part of the featured example consists of a photo of a boy eating.Below it a commentary reads: «Here is my picture of Fabian :-) and here a sentence to go with it, in which you have to determine the parts of the sentence».The pupil poses the question: «Can you determine the parts of the sentence?»and writes the following sentence, structured 1 to 4: «1.Fabian 2. eats 3. a Hamburger 4. at the station.»Below the picture, in the final section, the answer is provided: «Solution: 1. Subject 2. Verb 3. Object 4. Prepositional object» (Translated fromDeubelbeiss, 2007).Satzglieder 3. SekHier mein Bild von Fabian :-) und hier einen Satz dazu, bei dem du die Satzglieder bestimmen musst:

Figure 2 :
Figure 2: Syntax(Deubelbeiss, 2007) Parameter A: Learning sets Pole: Practice of the school -Pole: Practices of mobile media • Parameter B: Relationship to the object of learning Pole: Mimetic reproduction -Pole: Personal reconstruction • Parameter C: Institutional emphasis on expertise Pole: School curriculum -Pole: Personal expertise • Parameter D: Modes of representation Pole: Discrete (mono media, mono modal) -Pole: Convergent

Table 1 :
Typology-grid for analysing mobile learning