Measuring the quality of an e-learning course. Case OLAREX project

The EU Commission's learning action plan stresses the importance of ICT in education and quality assurance is part of the Bologna Declaration. 
General approaches to quality and to measuring quality are ISO, TQM and EFQM and Kirkpatrick for education. 2 popular quality measuring models are SEVAQ and SEEQUEL which are both based on EFQM and Kirkpatrick. 
In the project OLAREX a set of e-learning courses are developed to fulfill the project objectives: to enhance and modernize STEM curricula, foster students' creativity and motivation and develop professional e-didactic and technology competences and skills. 
The following quality assurance methods and tools are used: external and internal evaluation, comparative assessments, learners evaluation using questionnaires, informal discussions, observations and networking. 
A  SEEQUEL  revised  model  is  developed  by  the  external  experts  for  the external evaluation. The model is described in the paper. Part of the evaluation report is presented. In this report strong and weak points are formulated and needed improvements of the e-learning  courses are advised.

The EU Commission's Learning Action Plan [1] stresses the importance of ICT in (Higher) education when it defines e-learning as "the use of new multimedia technologies and the Internet to improve the quality of learning, by facilitating access to resources and services as well as remote exchanges and collaboration". Linked with the Bologna Declaration, EU Commission promoted already the development of a quality assessing methodology and a set of quality criteria to be used when measuring the quality. The culture of quality in e-learning is frequently weak, and, when present , is focusing on the didactics of e-learning and not so much on the more general impact that ICT is having at different levels (management, funding, international academic collaboration) of the education institute. Measuring the quality of the e-learning solution is even often limited to measuring learner's satisfaction. There have been already several initiatives to address quality of e-learning.
They are based on quality assurance models and systems that are used in general quality management in organisations.

General approaches to quality and measuring quality 2.1. ISO9000
ISO9000 [2] is a family of standards for quality management systems. ISO is the International Organization for Standardization and is administered by accreditation and certification bodies. ISO9001 is inspecting the final product but it also introduces the optimization of a companies processes. The effectiveness of tasks and activities is measured via process performance metrics, focusing on continual process improvement and tracking customer satisfaction.
Since the ISO standard defines the expected levels in terms of efficiency, compatibility and maintenance, evaluation consists almost exclusively of quantitative measurements. In the frame of quality of e-learning, very little information is retrieved to measure the degree of success of a training activity from a pedagogical point of view, that cannot be expressed as a quantity.

TQM and EFQM
TQM (total quality management) [3][4] is a management approach for an organization, based on the participation of all its members and aiming at long-term success through customer satisfaction, and benefits to all the stakeholders of the organization; It is a management strategy aimed at embedding awareness of quality in all organizational processes, and not only in those processes linked with the intended end-product or -service; EFQM (European Foundation for Quality management) [5] is the excellence model of TQM. It is guiding the organizations to TQM, by helping them understand the gaps by measuring where they are on the path to excellence, and to guide them in initiating remedial actions. The EFQM excellence model is the most widely used organizational framework in Europe as a basis for several national and regional quality awards and quality certificates. The model can also be used as a self-assessment model for all kind of organizations and it can be used as a benchmarking tool .
The model consists of 9 main criteria against which to assess an organization's progress towards excellence.
[6] Five of them are "enablers" : leadership, policy and strategy, people, partnership and resources, processes. Four of them are "results" : customer acceptance, functioning of employees, position in society, company results.
Sub-criteria have to be defined for all 9 main criteria. Self-assessment is organized by developing a questionnaire consisting of questions/ statements for all sub-criteria.

Kirkpatrick model for measuring quality of training [7]
Kirkpatrick model is focusing on the training process in the organization. The model of Kirkpatrick is focusing on the return of the training.
If you deliver training it is important to measure its effectiveness, you want to see a good return. Kirkpatrick's model can be used to measure and analyse the effectiveness and the impact of training, so that it can be improved in the future.
The model consists of 4 levels: how well did the learners like the learning process, what did people actually learn, what changes in job behavior resulted, what are the results in terms of reduced cost, improved quality, increased production …

Intro
In EU funded projects quality measuring models were developed. Two popular models are: SEVAQ (Self Evaluation of Quality in e-learning) and SEQUEL (Sustainable Environment for the Evaluation of Quality in e-learning) Both models are based on the EFQM model. Both models also integrating in their model the Kirkpatrick model.

SEVAQ [8][9]
An e-learning self-assessment model has been developed. To measure the overall quality of the e-learning organization, a set of main criteria, criteria and sub-criteria are identified that cover the organization wide aspects of the organization of training. The overall quality is measured by self-assessment by the internal stakeholders. They are asked to express their level of agreement with the formulated statements of the questionnaire.
A first model is used for the self-assessment by the learners. It is covering three main criteria: learning resources, learning processes and learning results.
A second model has been developed afterwards to include also management and teachers and tutors of the organisation in the measurement process. It is covering two main criteria: management of learning process and resources, management of people.

SEEQUEL [10], [11]
The SEEQUEL -Sustainable Environment for the Evaluation of Quality in E-Learningproject co-ordinated by the MENON Network [10]. SEEQUEL produced the SEEQUEL core quality Framework, an integrated set of quality criteria. The quality is structured as 3 main criteria of the "learning experience", being the key focus of quality: learning sources, core learning processes and learning context. The SEEQUEL Core Framework, empowers the elearning users to evaluate the eLearning quality by comparing what they experience with their personal expectations. The third element is the most innovative aspect of the SEEQUEL framework, and allows justifying the fact that not all the teachers have the same understanding of eLearning quality, as not all the university managers do, and so on. Every view of eLearning quality, as well as every approach, is legitimate because it is grounded on visions and values that cannot be homologated. For this reason, only a holistic approach to quality, able to "manage" all these views and the entire related values and criteria can guarantee a correct picture.

Objectives of OLAREX e-learning project. [12]
The project aims are to enhance and modernize STEM curricula, foster students' creativity and motivation, and develop professional e-didactic and technology competences and skills.
To accomplish these aims, the project partners will  Develop multimedia interactive e-learning contents  Implement multilingual e-learning portal in order to deliver e-materials and learning tools with remote experiments to the target groups and to provide the external tools and social networking (Facebook, TeacherTube, …).
 Improve e-didactic competences for 3 different target groups: for teachers, future developers of learning materials and for museum employees.

Development activity.
11 e-learning courses are developed for secondary school teachers OLAREX partners have developed the Module Specification e-content with eXeLearning.
[13] eXeLearning is an Open Source authoring application dedicated to the teachers and academics for publishing a web content without the need to become proficient in HTML or XML makeup. The content developed in this application can be exported in IMS Concept Package, SCORM 1.2, IMS Common Cartridge formats or as a simple self-contained web page.

Quality assurance frameworks
(1) Stakeholders (two main target groups highlighted in italics)

 Teachers in secondary and higher educational level
 Museum or other cultural/memory institutions' employees  Students in secondary and higher educational level  Students in non-formal education  Influential educational practitioners  Policy makers and educational networks  Regional and local education authorities and other bodies responsible for strategic developments concerned including those agencies which promote ICT in schools (2) quality frameworks The evaluation methodology developed for OLAREX was based on the SEEQUEL framework supplemented by user evaluation tools for remote laboratories and web portals.
As OLAREX was primarily developed to support quality assurance in non-formal and informal education, the SEEQUEL framework is supplemented with other tools used for quality assurance. The tools used in the OLAREX project were the following: The comparative assessment: The comparative assessment is needed to ensure that the developed products and solutions correspond to the results of the needs analysis. The analysis will be based on the results of the external and user evaluation results compared with the results of the target group knowledge needs analysis and the target group educational needs analysis.

content development instruction:
Partners involved in content development received instructions on learning module development (structure, length, style, multimedia, etc). All partners developing modules had to comply with the instructions and quality measures.

External evaluators
Two external experts were invited to conduct an external evaluation based on the SEEQUEL framework and share an opinion about the developed products, services. The two evaluators have teaching experience in secondary and higher/adult education, have been involved in formal and informal/non-formal education learning process, know the SEEQUEL core quality framework and have experience in web portal and e-learning quality assurance and evaluation. They received access to the materials, mostly through the wiki and the Moodle online platform.
The external evaluators have modified and tailored the original matrix to suit the needs of the project and by taking into consideration the information sources available for them (they were not part of the developing group and did not access the filled user questionnaires). The evaluators submitted a full evaluation report consisting of the filled matrix (evaluation criteria) and a descriptive evaluation with recommendations for improvement. After receiving the draft evaluation report a virtual discussion was organised with the participation of the evaluators and the project partners to draw the main conclusions and summarise suggestions for the future use of OLAREX products and services. The final evaluation report has been submitted in June 2013.

The OLAREX quality measurement instrument: the revised SEEQUEL model
The Olarex quality measurement instrument is developed based on the SEEQUEL model. The idea of learning experiences is copied and the general structure in 3 levels is also copied. We identify learning sources, core learning processes and learning context. The following figure presents the improved SEEQUEL model.

Customized model to be used by the external experts
Because the SEEQUEL instrument covers all organizational aspects around the development of e-learning courses, and because the external evaluators are not part of the project team, only a subset of criteria was selected, those that can be measured by the external experts. The following criteria and subcriteria are selected.
What has been measured or evaluated in these 13 groups?

1.2A.2.general info for the learner
Is the learner informed about target groups, studyload of the course and about learning outcomes/ competencies of the course?

1.2A.3. pre-requisites for learning
Is the learner informed about the pre-requisites on point of learning content and about experience of using ICT, and how he can prepare himself for the course?

1.2B.1. Structure of the content (and appropriateness)
Is the content structured in modules and differentiating for different types of target learners. Can the learners work in an easy way and active with the content?

1.2B.2. Integration of external content in the course
Are examples shown to the learners to understand the content , to explore applications of the content and to support the assignments? Are links and references of the sources provided?

assessments criteria, methods and tools
Clear assessments objectives? and are the criteria, the methods and the tools aligned with them? Are formative and summative asessments used and are they encourage creative and critical thinking? Are assessments organized at regular intervals and at the end? Are the procedures and the assessment criteria communicated with the learners?

2.B4.2.Assessment feedback
Are learners informed about the results of the assessments and are they advised about evt required remedial actions?

External evaluation report
The individual courses have been evaluated and scores have been given for all statements of the criteria. Per subcategory a mean score has been calculated. Based on these scores, the evaluation has been formulated as strengths, weaknesses and suggestions for improvement.
The evaluated courses comprise of e-learning courses developed for secondary school teachers to train them for using remote experiments in an online learning environment. This interesting concept provides hands-on experience for teachers, the primary target group in the evaluated courses, related to remote labs, and it is a good way for them to experience e-learning from a learner perspective. Successful combination of theory and practice is another distinction of the evaluated courses. As has been already mentioned, this evaluation report is based only on the evaluation of the end products as the experts were not included in the development process, and due to their professional commitment to the task they did not take into consideration results questionnaires of the learners having undergone the training.

An example part of the report
In figure 2 an example part of the end report is presented.
For each of the sub criteria, the strong and the weak points are defined based on the mean scores of the individual scores per question/ statement belonging to that subcriteria and that was given by the two experts.
At the end of the report some more general recommendations have been formulated for further actions. Before the completion of the project the OLAREX project management team may want to consider:  Using the entire SEEQUEL framework and including all relevant stakeholder in the evaluation of the project  Implementing suggestions for improvement identified by the external evaluation as well as self-evaluation See section 9  Using peer-assessment for evaluating implemented suggestions for improvement  Using another authoring tool or platform if the current one has a functions for implementing suggestions for improvement successfully limited tools and/or Besides the suggestions for improvement which have been identified by the experts for a particular course, development teams may also want to take into consideration additional explanations of the evaluation process, general conclusions and observations of the experts presented in this document. It is recommended that the OLAREX project management team also make a plan for implementation of improvements after the completion of the project to ensure sustainability of the project results, and to further disseminate good practice of teaching with remote experiments as conceptualized in this project.

Conclusion
e-learning courses are developed in the OLAREX project by all partners of the project consortium. These are developed to achieve the project objectives: to enhance and modernize STEM curricula, foster students' creativity and motivation and develop professional edidactic and technology competences and skills.
Quality assurance is an important issue in ICT enhanced learning.
Models and instruments to measure the quality of the e-learning development and courses are described. The external experts revised the SEQUEL model, applied it in their external evaluation. They reported about strong and weak points and advised on necessary improvements.