Wiki-Skills » WP8 Evaluation » Synthesis of test courses evaluations

Synthesis of test courses evaluations

Last modified by yves boisselier on 2013/12/17 22:19

Summary

All training sessions realised in the six countries (France, Germany, Greece, Spain, Sweden and Switzerland) confirmed some key aspects in relation to the Wiki Competences listed in the Pedagogical Framework. In addition, the scope of the evaluation also covered the methodology on seting-up, executing and evaluating training activities/projects related to the wiki-culture and collaborative approaches. The feedbacks provided constructive elements for the improvement with a clear contribution towards the sustainability of the WikiSkills training approach for future deployments and business models supporting a wider community of practice.

This document summarises :

  • the key Wiki Competences the trainees have acquired,
  • the impact on the pedagogical approach of the trainees as teacher,
  • the impact on the way of working within their community and their beneficiaries/colleagues/ecosystem actors.

The WikiSkills methodology and tools, the evaluation approach and tools were evaluated to identify strengths and weaknesses. These were pinpointed during the test and pilot phase realised in 2014. The purpose of that evaluation process was to identify what needed to be improved for a future exploitation and dissemination phase.

1 - Introduction

This analysis was elaborated based on each partner's evaluation reports available in the D8.2 section. These reports describe the conditions and contexts of the test courses, of the target groups involved and of each evaluation in terms of related wiki competencies, pedagogical impacts. The WikiSkills material was also evaluated for improvement.

An extended online questionnaire to evaluate this test/pilot phase was developed using the quiz tools provided by the Chamilo platform. One of the partners made some modifications on the questionnaire. In any case, the application of the standard questionnaires impacted only 35 responses on 66 participants.

 Nb of trainees in the target groupNb of people who answer
to the pre-questionnaire
Nb of people who answer
to the post-questionnaire
 nb of interviews Target group 
                  71                     52             12Primary and secondary school teachers
            80                 48                     44               3Teachers for adults
              8                  7                       6               4Teachers for higher education
            15
+ 125 students as a result of 3 cascading deployment scenarios trainees and teachers who followed WikiSkills courses and applied it in real situation.
                  7                          3             15 High-school students

2 - Dimensions evaluated

During the pilot and test courses, several dimensions were evaluated using the Evaluation Framework: the WikiSkills competences acquired through the training; Behavioural evolutions such as the collaboration in a virtual community; and the impact of the WikiSkills project on the pedagogical approaches.

2.1 -WikiSkills Key Competences

2.1.1- Skills or knowledge developed by teachers / trainers during the WikiSkills training

The participants demonstrated very various level of experience on wikis. For example, in Switzerland, more than 70% had no concrete experience on Wiki tools, and 22% were not regular users. But in general, the participants were not regular users and active contributors to wikis communities (which is then very interesting as it demonstrates that the WikiSkills results and outputs can be effectively used to actively develop this wiki culture dimension into various educational, pedagogical or knowledge sharing environments and target groups). 

Those who had no prior knowledge were expecting a general overview of the capabilities of wikis, "best practice" examples and an information on how to register on Wikipedia. During the interviews, these participants stated that they received a very good understanding of how wikis work and what they can be used for and what they can bring them.

Teachers reported that the WikiSkills training enabled them acquire new collaborative and ICT skills; and that they understood how to apply those skills and tools in their particular educational settings.  Besides of getting to know how to use the functionalities of wiki environments,the teachers also confirmed that they learnt how to manage students and how to organize team work in the context of wiki-based and collaborative learning activities.

By developing their own wiki-based learning scenario (practical hands-on training of teachers), the teachers directly understood the concept and the efficiency of collaborative learning.   
When sessions included the participants who came from different fields of working, they always mentioned they have learnt from each other during the course (i.e. team work, questions raised from participants). It was very helpful for them and brought them new perspective.

2.1.2- Wiki key competences

Globaly each partner reported that the participants were collaborative and worked as a team, contributing to the course and contributing to questions and answers. The spirit of contribution to other wikis was present in the training sessions, but, individually, many trainees didn't know if they were to contribute to other wiki pages related with their interest. All students/teachers involved in a course developed the majority of the following WikiSkills key competences through the scenario the were participating to:

  • Creativity and innovation: participants brought out new ideas and evaluated them in the context of the wiki activities.
  • Critical thinking, problem solving, decision making: while working collaboratively, participants had to constantly evaluate each other’s contributions and to make decisions in case of conflicts. To do so, they had searched for more resources in order to find the adequate solution to reach their objective.
  • Learning to learn: participants became active actors of the learning activity.
         "They distributed roles among them, searched for resources, elaborated content, self-evaluated their work, and chose by themselves the relevant solutions depending on the context."
        " Teachers did not have to intervene", "it was a constant self-evaluation"
  • Communication: In some case, and/or at the end of a session, when the participants accepted the transparency of wiki workspace, they brought attention to the way they expressed themselves in the wiki, and corrected the mistakes made by other trainees without waiting for the teacher's intervention. The visibility of their contribution towards their teachers and the other students led to develop their awareness on the direct impact of their contributions. This awareness also led to develop a more responsible way of communicating.
  • Collaboration: for teachers, the activity enhanced the collaboration skills of the participants. Indeed, in usual collective projects, students distributed tasks among them, worked separately, and then compiled all the parts. In contrast, wiki and collaborative culture, which allowed simultaneous work, disturbed their way of working; except in the case of the deployment scenario at FBS in France, the training periods (form the first to the final sessions) were too short in general to ensure the full benefit of the training in supporting the new collaborative approach. One teacher said "the learning concept behind wikis “seems more active than the -one way- learning”". Trainings sessions implemented in several working phase separated by individual working phase have given better results for collaborative works; then the trainees had the opportunity to have a more concrete experience. (i.e : student's interview abstract  : "We better understand others. We see how they work; this is the interesting side with collaborative and sharing tools" source MAC-Team report FBS scenario for ebook realisation).
  • Information Literacy: students learnt how to evaluate the relevance of the information found by the group in the Internet to their particular objectives.
  • ICT Literacy: they all learnt to use wiki easily, how to integrate images, ... all trainees had openness regarding ICT tools. In the FBS deployment scenario, the students also recognised that there is a need to keep open mind set to new tools in order to always adapt the best collaborative tool to the context and to the different collaborative groups they may need to run in parallel. The students also acknowledged that despite they are still "young" and belonging to the Y-Generation, they need to make efforts to always adapt instead of still using the same tools. They got aware about the need for agility in their future life and to cultivate that attitude. The Swedish pilot training illustrated some technical reasons which has threatened some teachers in using wikis for courses:
    • "We have no infrastructure support for our wiki like we have for our learning platform. In the learning platform, there are other people who ensure that all participants are included. On a wiki, I have to create a number of pages where students get their tasks, build a framework for the tasks and archive files from last year. In my teaching this means dozens of pages. It is about both formatting the text on the pages and managing links. Managing links is not trivial. "Almost half of the students have not succeeded in creating a link back to their own article, despite instructions".
    • NB : The technical support provided by Information infrastructure department of schools/universities may be involved in the process to support pedagogical teams and students.
  • Social responsibility: the participants became aware of the importance of naming sources of information for better and common trust on information included.

The post session questionnaires by the adult trainees pinpointed the level of actor's contributions in the wiki was quite similar to the existing usage identified as current usage by wiki communities (source Heig - adult session post questionnaire) :

  • from 2.8% to 5.2% of trainees acted as creator
  • from 22% to 31% of trainees read, and corrected mistakes,
  • and an average of 30% did not act at all in the wiki, and stay as observers.

For most of the trainees, being an actor of the wiki-culture was a very new situation. They have needed a longer training experience to become a real active actor of a wiki community (see following section "Community of Practices").

2.2 - Learning scenarios

Each training pilot session has applied the relevant scenarios depending on the target groups. The trainee's feedback is always positive and the participants were very active. The roles (creator, corrector, reader, moderator, ...) accepted by trainees within the groups are quite similar to the best practices of wiki communities. Even if the evaluation of training could lead to update some scenario, the WikiSkills scenario database was shared as "good practices for training on Wiki Culture". (see section 2.1-2.2 in D3.1 Definition of training scenarios and training curriculum). Partners could possibly think of a third step to go further for more concrete Wikis contribution to be delivered by students (ie "because the one duration for the two steps may not be enough to confront them with all the conditions they will meet in their future working life") and a longer duration scenario as the one built with FBS working at teachers and students levels at the same time and part of a full curricula could be used as a reference case in that respect.

2.3 - Virtual community of practices

The training sessions gave the opportunity to perceive the value of the efficiency of collaborative work. Many trainees declared their interests to collaborate further in a community. But it was not easy to have proof of these trainee's contributions in any virtual community of practices. Some direct experience was implemented just after some WikiSkills training sessions; and the FBS Scenario also integrated the monitoring of contributions done by students in public and well-known wikis (these contributions were also assessed by MAC-Team and included into the final score received by the student for its Master diploma).

In Spain, one teacher who followed the course presented the project to her colleagues, they showed a high level of interest in participating. As a result, they created a scenario of interest for their different disciplines and teaching objectives. On this basis, other teachers of the centre got interested. Furthermore, it is planned that teachers will present their project to all the teachers of the centre. Teachers are also planning to create a scenario which would be open to other educational centres. In summary, this experience pointed the impact of a "creator" profile to start and impulse initiatives through a common shared project with an added-value for the community, it is mandatory to impulse the energy to potential contributors to act. A similar example in France, through a Master Course session to write an ebook collaboratively on micro-finance (one topic in relation with the Master studies and on which the students also took note collaboratively), revealed the importance to put trainees in situation of living the situation for a concrete benefit.

In conclusion, these findings represent key aspects to help secure this important expectation of WikiSkills project. A follow-up process of training courses during several weeks may be very relevant to be added in the Pedagogical Framework. In addition, it also revealed the need for a supporting and accompanying process which could be very well supported by a community of practice on wikis being used as a resource pool. Mainly trainees were very new with the wiki subject (almost 90% of trainees were not currently active on any wiki platform). The acquisition of this new culture and new usages is not really 100% effective after such short session. The question of collaboration in virtual practices is becoming more relevant to users and organisations after several months of effective usage of wikis.

3 - WikiSkills project Impacts

3.1 - Usefulness of the training sessions

All interviewees mentioned that the outcomes of the WikiSkills training sessions were well beyond their initial expectations, and that they did not only learnt how to use wikis, but also how wiki and collaborative tools/culture can be integrated into very different fields (even beyond educational settings).

The interviewees reported a good interest for the tools, and that they have understood the need to remain agile and open to various technologies and behaviours depending on the context. On top of that, they clearly indicated that they have to use these tools and agility in their future working life. But it seems clear that for students (clearer than for adults) that trainers could be closer during the second step of the training for more intensive and interactive support to enrich the level of the working outcomes. (learning by doing, mixing roles and evolution of the pedagogical function).

3.2 - Fulfillment of user's expectations

The participants mainly expected to learn about one tool, but had the opportunity to get to know how to use a wide range of wiki-alike applications, such as PADs, wiki-spaces, and Mediawiki. Thus, they have discovered the larger impact of practices on the individual usage and interaction with others and with their environment (not only school/professional, but even with their family, friends or private circles).

They learned more collaboratively rather than being alone to do the corresponding effort. They have identified that motivation triggers and information anchorage are not the same; they expect to be much more efficient with that new environment. They clearly indicate that they perceived and understood the induced social changes. They became aware about the necessity to develop a new cultural context, a new thinking model and not only new tools. After that step, they also perceived the generated impacts on the professional relationship, which leads more towards project creativity and project management oriented.

3.2 - Challenge faced

The participants with no experience mentioned that they were quite afraid at the beginning when they noticed that some people in the class have experience with wikis and that they were only a “reader” on Wikipedia. They often mentioned that they were afraid of making mistakes and not being able to understand what is asked from them. But that fear was gone soon when the trainers listed needs and addressed all questions. Their feeling on insecurity vanished out.

Teachers trained were not sure of being able to solve the potential technical problems they could encountered during the usage with their student, and had some doubts about the best way to organize the activity. The support provided by the WikiSkills trainers during the pilot sessions handled that issue, but the quality of the technical support they could benefit from their institutions for a wider deployment has to be clearly identified. To overcome that last aspect and help wider deployment of collaborative culture, the identified need in chapter 2 above for a supporting and accompanying process could effectively be addressed by a community of practice on wikis being used as a resource pool.

Some students did not always feel comfortable working with ICT tools, as each of their contributions appeared clearly and in not finalised version. They felt permanently evaluated while working, and negatively valued this transparency, which highlights who does not appear working "enough", and who edits without telling. As expressed by one of the teachers, “for them, it’s a tool which is constantly judging”. The teachers/trainers have to manage that issue by promoting the collaborative way of working results efficiency by putting the students in an acting position.

3.3 - Impact on the Educational Community

The WikiSkills project through the Learning Scenario for wiki culture raised the interest of other teachers who participated in training sessions and who discovered a methodology which allowed them to work in cooperation, and to establish connections between their different disciplines. At the end of the training courses, directly one third of them mentioned their intention in conducting their scenario by themselves (and effectively did). Some participants decided to also promote the wiki culture approach in their direct working community, demonstrating a lot of interest and motivation for this new collaborative way of training.

Furthermore, some projects were implemented immediately after the courses to use the positive impact immediately in primary schools and high school. They identified that the wiki methodologies can be applied to various disciplines, and allow for visibility. Hence, students were able to externalize their work and show it to others. As a result, they felt responsible and recognised.

- Sant Josep Teresianes school (ES): they have applied a wiki approach for collaborative works and communicate though the School blog to be accessed by the families and the other educational institutions.

- France Business School (FR) with few days sessions to a Master Class (including ECTS points allocation): students had to collaborative take note of courses; then to elaborate an ebook in 3 weeks to summarize the yearly knowledge they acquired on micro-finance; and also to contribute to public wikis or to create wiki pages in relation to the topic of their Master studies (mainly on Wikipedia). The final e-book will be to published by end of 2013 (which also means that the students pursued collaborative work beyond their own studies and obligation for the Master diploma itself).

3.4 - Performance / Adaptation of tools used within the project

WikiSkills Platform: The experts (for the external evaluation of the WikiSkills outputs and outcomes) appreciated the many resources already available online. Sustainability of the scenario, they rated the sustainability of the scenarios and scenario approach is very high since it covers many different thematic areas.

Chamilo platform: some partners used the Chamilo platform less than other; this is rather due to the exploitation of the online questionnaires, which were heavy and exhaustive for the sake of this pilot phase. The integration (or regrouping) of all potential tools usable by trainees on one platform can simplify the access of tools for users with low experience on the wiki-culture tools. That aspect has to be analysed on a case by case depending on the context of the scenario (and the growing maturity of participants) for the future exploitation of WikiSkills results.

Training Evaluation: (Pre-questionnaire & Post-questionnaire): the efficiency of the questionnaire was discussed. The number of answers were not very numerous (mainly due to the pilot phase status and the exhaustive size of the questionnaires), which made quite difficult to synthesise the evaluation of pre-post questionnaire, but which provided very interesting feedback. These questionnaire will be simplified for the WikiSkills training activities when going for a wider exploitation phase (beyond the pilot/test phase now that the training materials, methodologies and teams have been tested, which means towards more exploitation and market-driven concerns).

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

pdf version of the corresponding report (v02 as of 28/11/2013)

Created by Pascal Echardour on 2013/11/05 11:13

Copyright 2004-2015 XWiki
3.4