EXTERNAL QUALITY EVALUATION OF NON-UNIVERSITY STUDY PROGRAMMES IN LITHUANIA

. The author analyzes the international nature of study programme evaluation with regard to the assurance of study quality. The organisation of the evaluation process of the non-university study programmes which were developed and submitted for realisation 1 in Lithuania and other countries is also presented and compared. It is being analysed whether it is possible to identify the quality of these programmes based on qualitative and qualitative indicators.

Qualitative indicatorscharacteristics used to reveal content value. Quality assuranceprocess and systematic procedures which help to ensure the quality of teaching/study institution activities, programmes and action-guiding documents. Regular programme evaluationperiodic evaluation defined by the agency, which organises external evaluation, strictly defining when the repeated evaluations of the programme quality will be performed at an educational institution. It is recommended that the educational institution itself performs regular internal study programme self-evaluations to improve the quality assurance system. Subject evaluation is oriented to the quality of the specific content of the subject under evaluation, its realisation opportunities, appropriacy for the study programme and general suitability for similar programmes (e.g., belonging to one study area). Thematic evaluationevaluation of a chosen topic which is significant for the quality of the study programme. For example, evaluation of final work, student support and counselling. Validity of evaluationproper selection and application of the evaluation methodology, which allows to measure what is measurable, research what is researchable and obtain evidence based outcomes, which provides evidence that the outcomes are as they should be according to the purpose of the evaluation.
Introduction. The need to study the quality of programmesi.e., the development of quality assurance systems at the institutional, national and European levels -is an issue at stake highlighted in the Bologna Declaration (Bologna Declaration, 1999) and particularly expanded in the Berlin Communique (Berlin Communique, 2003); it means seeking to include programmed and institutional evaluations into national systems, to foster internal self-evaluation, and to develop accreditation and comparability by 2005. Quality assurance is one of the priority objectives in the Lithuanian higher education system in order to be integrated into the corresponding European system. The transparency of the study quality evaluation policy and the validity of the methodological approaches are urgent and relevant issues in order to offer competitive studies on the European level and to adjust the quality of the studies to the international level. This need becomes particularly relevant given the current insufficiently developed study quality evaluation and assurance system. Rapid development of nonuniversity studies both in Lithuania and in other countries and higher education becoming more important enhance the discussion about study programme evaluation issues on the national as well as on the international levels.
Researchers pay considerable attention to the evaluation of the study quality and on quality assurance issues on the national and international levels. The concept of higher education study quality evaluation and assurance has been broadly discussed recently (Broadfoot M. Patricia, 1996; C. Campbell, Marijk van der Wende, 2000; A. Čiţas 1996, 1999, 2001; G. Stobart and C. Gipps, 1997; E. Stumbrys, 2003;S. Tidikis, 1998;S. Vengris, 1998;A. Vroeijenstijn, 1995). During the past few years the need for study quality internationalisation, evaluation and qualification recognition has been more and more emphasised (Berlin Communique, 2003; C. Campbell, Marijk van der Wende, 2000; The Danish Evaluation Institute, 2003; Ch. Thune, 2002). The issues of non-university study programme quality assurance and evaluation are analysed in the common context of higher education, however, sometimes they are also discerned as separate problems. Study programme development and evaluation problems on a non-university level abroad have been analysed for a long time (J. Losak, N. Burns 1971), but in Lithuania these analyses have just started. The development, evaluation and realisation problems of non-university study programmes were investigated by G. Ţibėnienė (2002, 2003a); the coherence of non-university and university business studies and the development of competency levels were studied by D. Lepaitė (2001Lepaitė ( , 2002; the didactical diagnostics of team work competencies and their development in nurse education and training were researched by V. Ţydţiūnaitė (2003). A critical approach to non-university study programmes and institutional quality assurance evaluation systems with the focus on organisational aspects was reflected in the article by M. Misiūnas and V. Tūtlys (2003). However, neither the problems that arise in performing nonuniversity study programme evaluation nor their causes have been investigated in depth.
The need for the present analysis was highlighted by recent research. According to the data presented by the sociological research into the Lithuanian higher education students and graduates (2003) and the research data presented by G. Ţibėnienė (2002; 2003) and V. Ţydţiūnaitė (2003), college graduates lack practical knowledge and skills. Among others, the above mentioned research emphasises that non-university study programmes have to be oriented to practical activity. However, these analyses have shown that these study programmes are more oriented to theory than practice, although, during the programme development and evaluation the focus on practical activity was taken into account.
The aim of the article is to analyse the problems arising in the evaluation of the quality of the non-university study programmes which were developed and submitted for the realisation on the basis of the Lithuanian experience.
The research object is the evaluation process of the non-university study programmes developed and submitted for realisation.
The research methodology. The following methods were applied: scientific literature analysis, document analysis and questionnaire survey. The expert method was used to evaluate all 294 study programmes developed in Lithuania in 1999 through 2002 according to the qualitative and quantitative indicators determined by the requirements for non-university study programmes, regulated by the Ministry of Education and Science of the Lithuanian Republic (LR MES) and the Government of the Lithuanian Republic. Each study programme was evaluated by 4-5 evaluators: Competent representatives of employers and higher education schools, representing the corresponding study field, evaluated the subject matter content and employees-experts from the Centre for Vocational Education Methodology at the MoES evaluated the adequacy of the programmes with respect to the documents regulating non-university studies. The data were analysed using mathematical-statistical methods applying the computerised programme Paula (B. Bitinas, 1998). The methodological background of the research is based on the following prerequisits:  Study programme evaluation is one part of the of the study quality assurance system;  The realisation of the Curriculum theory is the basis for a qualitative study programme (R. Lauţackas, 2000; F. Kerr, 1967; R.W. Tyler, 1949).
The article consists of four parts. In the first part the justification for the internationalisation presumption of the study programme evaluation with regard to quality assurance is provided. In the second part the comparison of the evaluation characteristics of the programmes which were developed and submitted for realisation in Lithuania and other countries is performed. In the third and fourth parts the evaluation of the non-university study programmes on the basis of qualitative and quantitative indicators is analysed and the organisation of non-university study programme evaluation processes is analysed.

Study programme evaluation as a presumption for study quality assurance.
The demands of the market economy and knowledge society (P. Drucker, 1993), the "maturing" new sector of higher educationa binary system with higher education being more massive -were researched by R. Barnett (1990); D. Gudaitytė (1998Gudaitytė ( , 2000Gudaitytė ( , 2002, P. Jucevičienė (1998), and others. The international situation has exerted impact upon the development of nonuniversity study programmes and on the formulation of their quality evaluation principles as well as upon a change of the attitude towards quality in education. Quality assurance of studies is one of the priority objectives related with the aim of integrating the Lithuanian system of higher education into the world education area and of the evaluation of the study programme quality 2 . Quality evaluation systems of studies have been created, are in effect and are constantly improved in many countries.
Quality evaluation of studies is usually performed on the study programme or institutional level, sometimes applying subject, thematic, academic and other form of evaluation. But on the institutional level the evaluation of the study process (study programme organisation, improvement, management) is always an important part. Regular institutional and programme evaluations are performed in Austria, Bulgaria, Germany, Finland, Hungary, Ireland, Latvia, Holland, and Rumania, among others. The Centre for Quality Assurance and Evaluation of Higher Education, 1998), which are affected, according to C. Spiel (2002), by still inconsistent and changing study programme evaluation strategies. This can be justified by the fact that "the resistance to the application of new didactical ideas is more intensive than to ideas in other fields"(B. Bitinas, 2000, p. 56-57). In the publications and internet pages of various evaluation agencies and organisations and in scientific sources it is possible to find various concepts -even inadequately defined -applied in the study programme evaluation: standard, criterion, indicator, aspect, dimension, and etc. The interpretation of the terms in the educational science and the Lithuanian language has raised a need for discussion. The broadest discussion on this problem was provided by P. Jucevičienė (2003, December) when analysing the impact and problems of the European educational thesaurus. As two concepts will be used in this research article -aspect 3 and indicator 4 -, only the motives underlying the choice of these terms will be presented.  The concept of "aspect" is often used in the literature presented by the study programme evaluation agencies in other countries. This concept, as stated by I. Savickienė, K. , is synonymous to the term "dimension". On the basis of the Contemporary Dictionary of the Lithuanian Language (2000), an aspect is an approach to investigate or discuss any subject or phenomenon. In the evaluation context, this means that an aspect should reflect an evaluative nature of the approach. However, the performed analysis of literature sources and documents of other countries has revealed that "aspect" differs in its content range (in various countries the range and meaning of the aspects of evaluation are different). In some countries not many aspects are used for study programme evaluation, however, these aspects are broadly spread into other evaluative features. For instance, in England only five aspects are applied to evaluate a developed and submitted for realisation programme 1) Curriculum designing, content and organisation; 2) Teaching, learning and evaluation; 3) Student support and counselling; 4) Teaching/learning resources; 5) Quality assurance of studies and enhancement perspectives (The Quality Assessment Method from April 1995). In some countries several separate aspects are used for the evaluation of the features revealing an aspect treated as one in another country (e.g. in England and Finland). Therefore, in the development of an external study programme evaluation strategy in Lithuania it is necessary to define the concepts used in the study programme evaluation, their use, content and range. It is popular to broadly use the concept "aspect" to reveal the characteristics of evaluation; the term "aspect" is found in the international documents and the documents of the national evaluation agencies in other countries and in their informational material presenting the background of evaluation. Taking into account the performed comparative literature analysis, the verbatim translation from the English language of the concept "aspect", and the frequency of the use of this term in the analysed literature (K. Pukelis, I. Savickienė, 2003; The Centre for Quality Assurance and Evaluation of Higher Education, 1998 and others), we choose to use this concept.  To concretise the chosen aspect level, the evaluation according to other features, such as indicators, can be applied. In English the concepts indicator and indicators 5 are identical, but in the Lithuanian explanation these concepts differ. Therefore, the term indicator was chosen as it sounds more Lithuanian and is more suitable for our understanding. According to C. Spiel (2002), study programme evaluation originated from the need to provide evidence or show the need to improve the value of a programme, its efficiency and usefulness, as well as the purposefulness, clarity and achievability of the study goals. The necessity of the need for teaching programme evaluations was grounded by curriculum theory researchers (R. Lauţackas, 1999Lauţackas, , 2000; R. W. Tyler, 1949 and others). On the basis of curriculum 6 theory and according to one of its originators, any programme evaluation process including planning should start from the evaluation of teaching/study goals (R.W. Tyler, 1949, p.110). R. Lauţackas (2000), a curriculum theory researcher in Lithuania, emphasises the process of sustainable curriculum renewal, highlighting the circular movement "in which the logical point of reference of this continuous process is qualification research, demanded by activity systems, and the ending is curriculum evaluation and realisation"(2000, p. [24][25]. These ideas are closely related to a constant and systematic need for evaluation of various components of quality of studies and its strategy reflected in the European documents. Literature analysis, international evaluation of quality of studies and assurance policy and curriculum theory lead to the emergence of the essential principle: constant evaluation of the programme (at various stages) is an important part of the quality assurance of study system.

Aspects of non-university study programme evaluation
The analysis of the documents of various study quality evaluation agencies (The Centre for Quality Assurance and Evaluation of Higher Education, 1998) and works by individual researchers (C. Campbell and C. Rozsnyai, 2002; K. Pukelis, I.  showed that there is no clear and concrete distinction between the aspects to be used for study programmes which are being developed and submitted for realisation and for those which are currently carried out. The scientific literature underlines the lack of analyses of the methodologies and methods to evaluate both types of programmes (D. Lepaitė, 2003). The analysis of the content of the evaluation aspects reveals which aspects are important for the study programme to be developed and under realisation and which ones for the developed programme. Aspects summarised in Table 1 are applied for the developed study programme evaluation on the basis of the experiences in Austria, the United Kingdom, Holland, Finland, the USA and the analysis by K. Pukelis and I. . Table 1.

Comparison of the foreign external evaluation aspects of the non-university study programmes which were developed and submitted for evaluation with the aspects applied in Lithuania
Aspects applied in foreign countries for the evaluation of the non-university study programmes which were developed and submitted for realisation

Aspects applied in Lithuania for the evaluation of the non-university study programmes which were developed and submitted for realisation
Relation between the preparation and opportunities to carry out the study programme Justification of the necessity for the study programme Study programme goals and objectives + Programme content + Programme structure and average duration of studies + Didactical conception and philosophy ---, but foreseen Teaching process designing, teaching methods** until 2003, now insufficiently Study content design + Student evaluation, examination (foreseen) + Personnel qualification and activity (including creativity, research work and etc.)

Material, methodological basis and information technologies
level of evaluation is chosen* Internal self-evaluation of quality of studies and assurance system rudiments evaluated 7 Internationalisation of study content ---Compatibility of study content with the latest achievements and the demands of the world of work (employer opinion) +

Teacher opinion level of evaluation is chosen * No data found
Opinion of official professional institutions (according to the need) no data found Correspondence of the title, qualification to the content (in foreign countries a part of evaluation of other 6 There has not been consistent opinion regarding the equivalent of the concept in Lithuania yet. According to the researchers R. Lauţackas (2000) and K. Pukelis (1999), who analysed this issue, in the Lithuanian educational terminology "curriculum" has no direct and full equivalent. In foreign countries there has always existed a rather different and multisided "curriculum" concept: from plan for learning, from courses to teaching goals, content, methods, material and means, teaching organisation and control system, sustainable and gradually changing process from the outset of teaching (B. Holmen, M. McLean 1967; John F. Kerr 1967; K. Pukelis, 1999). According to the study by Ralf W. Tyler, an originator and advocator of the theory, cited by R. Lauţackas (1999Lauţackas ( , 2000 "curriculum" is a goal-oriented teaching programme. 7 Colleges pay attention to study quality by establishing study quality maintenance departments or divisions (M. Misiūnas, 1999; M. Misiūnas, I. Savickienė, 2002; V. Stasiulionienė, 2003).

aspects) no data found
Adequacy to regulations of the Ministry of Education and Science Development and assurance of study quality system More important in institutional evaluation * -it is possible to discern two levels: comprehensive, when evaluation is performed in the institution where the study programme will be implemented; documentary evaluation when an expert evaluates the documents submitted by the institution.
** -study methods in some sources.
Non-university study programme evaluation aspects in Lithuania and other countries are similar; however, several differences have been noticed that are important for well-rounded study quality assurance and international policy in this area (Table 1). Deficits in Lithuania and hence the need for special attention can be found with respect to didactical conception and philosophy; teaching process designing, teaching methods; internationalisation of study content; development and assurance of study quality systems.
Didactical conception and philosophy and teaching methods are important for successful nonuniversity studies in Lithuania. On the basis of the data of the 1999 research, where the research participants were vocational advanced schools that grew up to colleges, the most frequent problems were as follows: the pedagogical personnel has a higher education but lacks didactical qualification, competence to determine teaching goals, skills in teaching content, teaching method planning, undervalues learner learning needs in the organisation of the teaching process (R. Lauţackas, 2002). According to expert evaluation data and the opinion of the first non-university study programme graduates, didactical elements and application of teaching methods calls for development (G. Ţibėnienė, 2003b).
The emergence of the idea of internationalisation of study content was caused by the tendencies in the study quality assurance policy. On the basis of Marijk van der Wende (cited by P. Jucevičienė, 2003), study internationalisation requires systematic and harmonious efforts in higher education to respond more adequately to the challenges of societal, economic and labour market globalisation. The credit transfer ideas declared in the Copenhagen Declaration (2002) contribute to study internationalisation.
Another important aspect, which is more taken into account in the institutional evaluation in Lithuania but still are not sufficiently developed on the programme level, is the development and assurance of study quality systems. The need for study quality to be comparable with international standards (both in institutional and programme levels) is also emphasised by researchers in Lithuania (E. Stumbrys, 2003).
Comparing study programme aspects in Lithuania with the aspects which are regarded as most relevant in other countries makes it possible to provide a good base for the development and realisation of an appropriate quality assurance system in Lithuania. The further steps to be done can be seen (e.g., a social partner, an expert in the relevant area, participates in the process of evaluation in all the cases), which particularly suits the specific needs of non-university studies from the orientation towards the practical activities.

Organisation of the evaluation process of the non-university study programmes which were developed and submitted for realisation in Lithuania.
According to the organisational nature of new non-university study programmes, their evaluation can be divided into documentary evaluation and field visits to the educational institutions.
Documentary evaluation. Documentary evaluation of a non-university study programme is often an introductory but sometimes the only and main evaluation part. However, the qualification the experts and evaluation guidelines are usually insufficient for the documentary evaluation to be objective. The principles of comparability of similar objects (in this case comparison with all similar programmes) or of Benchmarking of programmes outspreading in Europe (Copenhagen Declaration, 2002; The Danish Evaluation Institute, 2003) on the regional level would highlight the objectivity of evaluation. However, these principles are hardly applied or only in single cases of evaluation of study programmes in Lithuania.

Field visits to institutions implementing study programmes.
In the college establishment process, after documentary evaluation, not only the preparation of the institution for a new higher education school status was evaluated but the opportunities for study programme realisation as well. Having conferred a college status, an external visit was performed if new study areas were introduced or on demand. Such practice is applied not only in Lithuania but in other countries as well. But in contrast to the latter, in Lithuania the evaluation of non-university study programmes was organised according to institutions, paying attention to the general level of the respective higher school. The chosen evaluation strategy allowed tosee the range of study programmes at school, however, the opportunities for the comparison of the contents of the study programmes under the same title were not or insufficiently created. The quality level of non-university study programmes would have been more highlighted by the principle of benchmarking of study programmes in the institution or region and organising the evaluation of the same or similar study programmes accordingly. The set approach, the use of standard samples and of pursuable standard, which are elements of comparison popular in contemporary evaluation practice, were not applied. In the process of the creation of colleges two approaches were implemented simultaneously: the evaluation of the preparation of the vocational advanced school to become an institution of higher education and the evaluation of the preparation to perform non-university studies according to various study programmes; this was done for reasons of economical-prudence and rapidity. However, this enhanced indirectly the identification of college status with positive evaluation of the programmes which were developed and submitted for realisation, whereas the world practice indicates that institutions reach a new qualitative level in this case, the status of collegeby gradually realising new study programmes.
Evaluation of non-university study programmes in Lithuania on the basis of qualitative and quantitative indicators. In the pursuit to evaluate the quality of study programmes, typically, both qualitative and quantitative indictors are used (M.C. Alkin and J.D. McNeil, 2002) and provided as statistical and/or descriptive information to help to define the quality of study programmes and create preconditions for the identification of current and possible future drawbacks. When using the quantitative indicators for the evaluation of study programmes, the experts' evaluations coincided. Reliability of evaluation with these indicators was enhanced by a formalised study programme form and clear evaluation methodology with an evaluation scale, which helped to indicate the indicator achievement level. However, lack of formalised fixation of the meanings of terms used in the evaluation led to undesirable interpretations in the quantitative evaluation 8 . Due to the empirical nature of the study programme and to the autonomy of higher education, evaluation based solely on quantitative indicators is inexpedient and even flawy for study programme content evaluation.
Qualitative indicators and content analysis acquire a specific importance in the evaluation of a study programme, the content of a separate subject and the quality of the studies. Generalisation of independent expert evaluations revealed that some qualitative indicators were evaluated rather unanimously: adequacy to professional education standard; adequacy of distant or evening studies to fulltime studies, preparation of non-university study programme according to a set form. However, the remaining qualitative indicators yielded rather contradictory evaluations. One of the reasons lies in the fact that the experts had different education levels and backgrounds: university, college, social partners who were selected with respect to their competence in vocational education and training area. However, there were no programme development experts in the evaluation group who could have evaluated study programme aspects from a methodological point of view. Thus, analysing the evaluation outcomes, it is evident that contradictory judgments using separate indicators were caused by: variety of qualitative indicators; single-sided principle of composing an evaluation group.
Variety of quantitative indicators. A rather contradictory evaluation of separate indicators, e.g. teaching method evaluation, revealed the problem of variety of indicators which caused the problem of lack of clarity, i.e. a given indicator is clear for a teacher but not for a social partner. Taking into account the specificity of quantitative indicators, they could be grouped according to some other features: didactical; teaching content, etc. An expert who has knowledge in didactics cannot evaluate the content of a concrete study programme, but he or she is competent to evaluate the appropriacy of the teaching methods or clarity of the study goals that are hard to understand for a social partner 9 .
Single-sided principle of composing an evaluator group. Taking into account the aim of nonuniversity study programmes to achieve competencies and focusing the evaluation on whether opportunities for competence acquisition are created, experts were selected according to the major criterion: professional and subject knowledge. The analysis of the specificity of quantitative indicators (didactical and study content) revealed the importance of improving the principles of composing an expert group, e.g. programme developer and expert in didactics should be included in the evaluation of any study programme. Up to now, experts recommended by various competent institutions were selected for the evaluation with respect to their achievements in professional subject area, however, with no respect to their know-how in didactical area. Therefore, a number of connoisseurs in their speciality could not evaluate programme quality according to qualitative indicators in a sufficient way.
On the basis of the experience of the performed evaluation, the following seems important for study programme evaluation: scientifically grounded evaluation organisation and performance methodology; valid evaluation indicators; and competence of experts. Quantitative and qualitative evaluation indicators are important and supplement each other; however, considering evaluation a complex type of research, they call for scientifically grounded strategy and methodology of study programme.

Conclusions
1. Sustainable evaluation of study programmes (at various stages) is an important part of a quality assurance system of studies, and the evaluation of the developed study programme is its point of reference. 2. The aspects of the evaluation of the non-university study programmes which were developed and submitted for realisation in Lithuania and in foreign countries are similar, however, in Lithuania, insufficient attention is devoted to the evaluation of some methodologically important aspects: didactical competence and philosophy; teaching process and teaching/learning methods; internationalisation of the study content; development and assurance of the study quality system. 3. Successful evaluations of the non-university study programmes which were developed and submitted for realisation call for the improvement of strategic, organisational and methodological principles. Particular areas need to be discussed, especially in organisation of evaluation: benchmarking of study programmes; need for scientifically grounded evaluation methodology; inclusion of a programme developer-expert, a specialist in didactics, in the evaluation of any study programme.