All training sessions realised in the six countries (France, Germany, Greece, Spain, Sweden and Switzerland) confirmed some key aspects in relation to the Wiki Competences listed in the. In addition, the scope of the evaluation also covered the methodology on seting-up, executing and evaluating training activities/projects related to the wiki-culture and collaborative approaches. The feedbacks provided constructive elements for the improvement with a clear contribution towards the sustainability of the WikiSkills training approach for future deployments and business models supporting a wider community of practice.
This document summarises :
The WikiSkills methodology and tools, the evaluation approach and tools were evaluated to identify strengths and weaknesses. These were pinpointed during the test and pilot phase realised in 2014. The purpose of that evaluation process was to identify what needed to be improved for a future exploitation and dissemination phase.
This analysis was elaborated based on each partner's evaluation reports available in the. These reports describe the conditions and contexts of the test courses, of the target groups involved and of each evaluation in terms of related wiki competencies, pedagogical impacts. The WikiSkills material was also evaluated for improvement.
An extended online questionnaire to evaluate this test/pilot phase was developed using the quiz tools provided by the. One of the partners made some modifications on the questionnaire. In any case, the application of the standard questionnaires impacted only 35 responses on 66 participants.
|Nb of trainees in the target group||Nb of people who answer |
to the pre-questionnaire
|Nb of people who answer |
to the post-questionnaire
|nb of interviews||Target group|
|71||52||12||Primary and secondary school teachers|
|80||48||44||3||Teachers for adults|
|8||7||6||4||Teachers for higher education|
+ 125 students as a result of 3 cascading deployment scenarios trainees and teachers who followed WikiSkills courses and applied it in real situation.
During the pilot and test courses, several dimensions were evaluated using the Evaluation Framework: the WikiSkills competences acquired through the training; Behavioural evolutions such as the collaboration in a virtual community; and the impact of the WikiSkills project on the pedagogical approaches.
The participants demonstrated very various level of experience on wikis. For example, in Switzerland, more than 70% had no concrete experience on Wiki tools, and 22% were not regular users. But in general, the participants were not regular users and active contributors to wikis communities (which is then very interesting as it demonstrates that the WikiSkills results and outputs can be effectively used to actively develop this wiki culture dimension into various educational, pedagogical or knowledge sharing environments and target groups).
Those who had no prior knowledge were expecting a general overview of the capabilities of wikis, "best practice" examples and an information on how to register on Wikipedia. During the interviews, these participants stated that they received a very good understanding of how wikis work and what they can be used for and what they can bring them.
Teachers reported that the WikiSkills training enabled them acquire new collaborative and ICT skills; and that they understood how to apply those skills and tools in their particular educational settings. Besides of getting to know how to use the functionalities of wiki environments,the teachers also confirmed that they learnt how to manage students and how to organize team work in the context of wiki-based and collaborative learning activities.
By developing their own wiki-based learning scenario (practical hands-on training of teachers), the teachers directly understood the concept and the efficiency of collaborative learning.
When sessions included the participants who came from different fields of working, they always mentioned they have learnt from each other during the course (i.e. team work, questions raised from participants). It was very helpful for them and brought them new perspective.
Globaly each partner reported that the participants were collaborative and worked as a team, contributing to the course and contributing to questions and answers. The spirit of contribution to other wikis was present in the training sessions, but, individually, many trainees didn't know if they were to contribute to other wiki pages related with their interest. All students/teachers involved in a course developed the majority of the following WikiSkills key competences through the scenario the were participating to:
The post session questionnaires by the adult trainees pinpointed the level of actor's contributions in the wiki was quite similar to the existing usage identified as current usage by wiki communities (source Heig - adult session post questionnaire) :
For most of the trainees, being an actor of the wiki-culture was a very new situation. They have needed a longer training experience to become a real active actor of a wiki community (see following section "Community of Practices").
Each training pilot session has applied the relevant scenarios depending on the target groups. The trainee's feedback is always positive and the participants were very active. The roles (creator, corrector, reader, moderator, ...) accepted by trainees within the groups are quite similar to the best practices of wiki communities. Even if the evaluation of training could lead to update some scenario, the WikiSkills scenario database was shared as "good practices for training on Wiki Culture". (see section 2.1-2.2 in). Partners could possibly think of a third step to go further for more concrete Wikis contribution to be delivered by students (ie "because the one duration for the two steps may not be enough to confront them with all the conditions they will meet in their future working life") and a longer duration scenario as the one built with could be used as a reference case in that respect.
The training sessions gave the opportunity to perceive the value of the efficiency of collaborative work. Many trainees declared their interests to collaborate further in a community. But it was not easy to have proof of these trainee's contributions in any virtual community of practices. Some direct experience was implemented just after some WikiSkills training sessions; and the FBS Scenario also integrated the monitoring of contributions done by students in public and well-known wikis (these contributions were also assessed by MAC-Team and included into the final score received by the student for its Master diploma).
In Spain, one teacher who followed the course presented the project to her colleagues, they showed a high level of interest in participating. As a result, they created a scenario of interest for their different disciplines and teaching objectives. On this basis, other teachers of the centre got interested. Furthermore, it is planned that teachers will present their project to all the teachers of the centre. Teachers are also planning to create a scenario which would be open to other educational centres. In summary, this experience pointed the impact of a "creator" profile to start and impulse initiatives through a common shared project with an added-value for the community, it is mandatory to impulse the energy to potential contributors to act. A similar example in France, through a Master Course session to write an ebook collaboratively on micro-finance (one topic in relation with the Master studies and on which the students also took note collaboratively), revealed the importance to put trainees in situation of living the situation for a concrete benefit.
In conclusion, these findings represent key aspects to help secure this important expectation of WikiSkills project. A follow-up process of training courses during several weeks may be very relevant to be added in the Pedagogical Framework. In addition, it also revealed the need for a supporting and accompanying process which could be very well supported by a community of practice on wikis being used as a resource pool. Mainly trainees were very new with the wiki subject (almost 90% of trainees were not currently active on any wiki platform). The acquisition of this new culture and new usages is not really 100% effective after such short session. The question of collaboration in virtual practices is becoming more relevant to users and organisations after several months of effective usage of wikis.
All interviewees mentioned that the outcomes of the WikiSkills training sessions were well beyond their initial expectations, and that they did not only learnt how to use wikis, but also how wiki and collaborative tools/culture can be integrated into very different fields (even beyond educational settings).
The interviewees reported a good interest for the tools, and that they have understood the need to remain agile and open to various technologies and behaviours depending on the context. On top of that, they clearly indicated that they have to use these tools and agility in their future working life. But it seems clear that for students (clearer than for adults) that trainers could be closer during the second step of the training for more intensive and interactive support to enrich the level of the working outcomes. (learning by doing, mixing roles and evolution of the pedagogical function).
The participants mainly expected to learn about one tool, but had the opportunity to get to know how to use a wide range of wiki-alike applications, such as PADs, wiki-spaces, and Mediawiki. Thus, they have discovered the larger impact of practices on the individual usage and interaction with others and with their environment (not only school/professional, but even with their family, friends or private circles).
They learned more collaboratively rather than being alone to do the corresponding effort. They have identified that motivation triggers and information anchorage are not the same; they expect to be much more efficient with that new environment. They clearly indicate that they perceived and understood the induced social changes. They became aware about the necessity to develop a new cultural context, a new thinking model and not only new tools. After that step, they also perceived the generated impacts on the professional relationship, which leads more towards project creativity and project management oriented.
The participants with no experience mentioned that they were quite afraid at the beginning when they noticed that some people in the class have experience with wikis and that they were only a “reader” on Wikipedia. They often mentioned that they were afraid of making mistakes and not being able to understand what is asked from them. But that fear was gone soon when the trainers listed needs and addressed all questions. Their feeling on insecurity vanished out.
Teachers trained were not sure of being able to solve the potential technical problems they could encountered during the usage with their student, and had some doubts about the best way to organize the activity. The support provided by the WikiSkills trainers during the pilot sessions handled that issue, but the quality of the technical support they could benefit from their institutions for a wider deployment has to be clearly identified. To overcome that last aspect and help wider deployment of collaborative culture, the identified need in chapter 2 above for a supporting and accompanying process could effectively be addressed by a community of practice on wikis being used as a resource pool.
Some students did not always feel comfortable working with ICT tools, as each of their contributions appeared clearly and in not finalised version. They felt permanently evaluated while working, and negatively valued this transparency, which highlights who does not appear working "enough", and who edits without telling. As expressed by one of the teachers, “for them, it’s a tool which is constantly judging”. The teachers/trainers have to manage that issue by promoting the collaborative way of working results efficiency by putting the students in an acting position.
The WikiSkills project through the Learning Scenario for wiki culture raised the interest of other teachers who participated in training sessions and who discovered a methodology which allowed them to work in cooperation, and to establish connections between their different disciplines. At the end of the training courses, directly one third of them mentioned their intention in conducting their scenario by themselves (and effectively did). Some participants decided to also promote the wiki culture approach in their direct working community, demonstrating a lot of interest and motivation for this new collaborative way of training.
Furthermore, some projects were implemented immediately after the courses to use the positive impact immediately in primary schools and high school. They identified that the wiki methodologies can be applied to various disciplines, and allow for visibility. Hence, students were able to externalize their work and show it to others. As a result, they felt responsible and recognised.
- Sant Josep Teresianes school (ES): they have applied a wiki approach for collaborative works and communicate though the School blog to be accessed by the families and the other educational institutions.
- France Business School (FR) with few days sessions to a Master Class (including ECTS points allocation): students had to collaborative take note of courses; then to elaborate an ebook in 3 weeks to summarize the yearly knowledge they acquired on micro-finance; and also to contribute to public wikis or to create wiki pages in relation to the topic of their Master studies (mainly on Wikipedia). The final e-book will be to published by end of 2013 (which also means that the students pursued collaborative work beyond their own studies and obligation for the Master diploma itself).
: The experts (for the external evaluation of the WikiSkills outputs and outcomes) appreciated the many resources already available online. Sustainability of the scenario, they rated the sustainability of the scenarios and scenario approach is very high since it covers many different thematic areas.
: some partners used the Chamilo platform less than other; this is rather due to the exploitation of the online questionnaires, which were heavy and exhaustive for the sake of this pilot phase. The integration (or regrouping) of all potential tools usable by trainees on one platform can simplify the access of tools for users with low experience on the wiki-culture tools. That aspect has to be analysed on a case by case depending on the context of the scenario (and the growing maturity of participants) for the future exploitation of WikiSkills results.
: (Pre-questionnaire & Post-questionnaire): the efficiency of the questionnaire was discussed. The number of answers were not very numerous (mainly due to the pilot phase status and the exhaustive size of the questionnaires), which made quite difficult to synthesise the evaluation of pre-post questionnaire, but which provided very interesting feedback. These questionnaire will be simplified for the WikiSkills training activities when going for a wider exploitation phase (beyond the pilot/test phase now that the training materials, methodologies and teams have been tested, which means towards more exploitation and market-driven concerns).
pdf version of the