Unit Introduction
The aim of this unit is to enable learners to understand the principles of evaluating learning programmes and use evaluation methods and data to understand how these can be used to improve learning programmes.
1. Understanding the principles and methods of evaluating learning programmes
1.1 Analyse the principles of evaluating learning:
The principle of evaluating learning starts with determining and clarifying what is to be evaluated. Evaluating learning is a means to an end, not an end itself. As a trainer, it is important that I look back to the learning objectives/outcomes involved when evaluating training and learning activities. One cannot measure what learners have learned without knowing what they are measuring against (i.e. the standards). If I do not think carefully about learning objectives at the planning stage, it will not always be possible to evaluate whether learners have achieved them.
According to a research “Assessing and Evaluating Student Learning” (attached), “Evaluation is the process of analysing, reflecting upon, and summarising assessment information, and making judgements and/or decisions based on the information collected”. On the other hand, “Assessment is the process of gathering information on student learning”. The primary purpose of assessment and evaluation is to inform training and to promote and encourage learning.
A successful learning evaluation should include all involved in the management of learning. It must also be able to: measures success, highlight areas for development, and identifying areas for change.
1.2 Explain how the principles of evaluating learning can be applied to the evaluation of learning programmes:
Programme evaluation refers to the thought process of focusing on questions and topics of concern, collecting appropriate information, and then analysing and interpreting the information for a specific use and purpose (Taylor-Powell et al., 1996). In preparation for the evaluation of a learning programme, I first carry out a SWOT (strengths, weaknesses, opportunities and threats) analysis of the learning programme to collect necessary data. The SWOT would reveal many areas for consideration within the learning programme, including strengths which should be maintained and built upon, opportunities which need to be prioritised and optimised, weaknesses to be remedied or mitigated, and threats for which solutions should be sought. As trainers, we need to ask ourselves “What opportunities arise because of our strengths?” and “What are the threats that the weaknesses expose us to?” (Tyler, 2004).
The principles of evaluating learning can be applied to the evaluation of learning programmes at various stages. It should be applied at the start of a programme for planning to meet student needs, during the programme to gauge success, and at the end of the programme to advise change. It is also applied in checking that resources and assessment are fit for purpose. Data obtained from evaluating learning programmes could also form the basis for team or organisation review and internal quality assurance (IQA) processes.
1.3 Analyse methods used for evaluating the effectiveness of learning programmes:
The effectiveness of learning programmes can be evaluated using various techniques. Some of the methods that could be used include open and closed questions and answers, rubric questions, one-minute question, group discussion – with or without trainer presence, group representative feedback, formal pro forma, tally sheet, questionnaire, etc.
According to the online Collins English dictionary (http://www.collinsdictionary.com/dictionary/english/open-question), an open question is “a question that cannot be answered with a yes or no but requires a developed answer”. Asking learners to state their opinion of something is a very good example of an open question. A closed question, on the other hand, can be answered with either ‘yes’ or ‘no’ (http://changingminds.org/techniques/questioning/open_closed_questions.htm). Closed questions can be answered with either a single word (such as true/false, on/off, yes/no, etc.) or by a short phrase. Multiple choice questions are classified as having the closed format.
The main purpose of rubric question is to assess performances. As a trainer, I sometimes assess learners’ performances by observing them in the process of doing something. An example is when assessing learners’ ability in using an electric drill. For other performances, I observe the product that is the result of the learners’ work, like a written report.
The one minute question is a very commonly used classroom assessment technique. It really does take about a minute and, while usually used at the end of class, it can be used at the end of any topic discussion (Angelo and Cross, 1993). Its major advantage is that it provides rapid feedback on whether the trainer’s main idea and what the students perceived as the main idea is the same. Three examples of one-minute questions that I have used in evaluating the effectiveness of my training are as follows:
- What are the two most significant things you have learned during this session?
- What question(s) remain uppermost in your mind?
- Is there anything you did not understand?
1.4 Analyse methods of data collection and analysis used for evaluating learning programmes:
There are various techniques available for collecting and analysing data in order to evaluate learning programmes. Some of these methods include: summary of responses obtained from open and closed questions and answers, identifying qualitative data (including: written, textual, factual, or opinionated Information) generated through questionnaires, statistical or quantitative data (including: graphs, charts, tables, pictograms, etc.) from tally sheets, frequency analysis based on responses to questioning or questionnaires, use of online data collection format, etc.
The assessment/evaluation process involves the use of multiple sources of information collected in a variety of contexts. Many trainers use observation, work samples, and self-evaluation as tools in the process of assessment and evaluation.
- Observation is the careful consideration and analysis of learners’ behaviour and performance based on a broad range of contexts. In order to use observation effectively, I need to know a lot about my learners and be able to interpret what I observe.
- Self-evaluation has to do with getting learners involved in reflecting on their own learning experience as a way of empowering them.
- Reflecting on their learning leads learners to gain increasing control over their learning and language processes.
- Self-evaluation could be formal or informal. Informal self-evaluation consists of the ongoing reflection about learning that is a natural daily part of the curriculum (for example, What did we learn? How did we solve the problem?).
Trainers can encourage this kind of reflection in a variety of contexts, for example, classroom discussions, and learning logs. Formal self-evaluation could come as classroom portfolios. A portfolio is a collection of work selected by the student for assessment and evaluation purposes to reflect his/her best or representative work.
1.5 Explain the impact of guidelines and legislation on data collection and analysis.
The guidelines and legislation relevant to data collection and analysis that I considered throughout the process of my programme evaluation are the Data Protection Act (1998) and the Computer Misuse Act (1990). The Data Protection Act (1998) came into force in March 2001 when it replaced the Data Protection Act (1984) and shortly by the GDPR 2018. The UK government as part of its “data protection” (https://www.gov.uk/data-protection/the-data-protection-act) strategy, stated that “The Data Protection Act controls how your personal information is used by organisations, businesses or the government”. It explains that everyone responsible for using data has to follow strict rules called ‘data protection principles’. I ensured that my learners understood and adhered to the principles of the Data Protection Act (1998) throughout their use of ICT in the analysis and display of statistical data. When preparing the questionnaire, I ensured that learners are not required to enter their personal details when responding to the questions. Learners were informed that their response to the questionnaire was being collected in anonymity. This made my learners open and eager to respond to the questionnaire and as a result, I got 100% response from all learners present at the session. The Computer Misuse Act (1990) is another legislation that my learners and I had to comply with throughout the programme evaluation process. The computer misuse offences and jurisdiction (http://www.legislation.gov.uk/ukpga/1990/18/contents) were heightened and emphasised at the start of the evaluation process.
I aim to be carrying out an evaluation each time a new technique or process is introduced. The evaluation results would be made available on my organisation’s virtual learning environment (VLE). This is to ensure that my colleagues and other members of staff can access it even when they are off-site. The outcome of this evaluation would let me decide on what is working and what needs to be changed in other to improve the effectiveness of the learning programme. “The primary purpose of assessment and evaluation is to inform training and to promote and encourage learning—to promote optimal individual growth. In order to provide information vital to the trainers, assessment and evaluation must be an ongoing and integral part of the training/learning process”.
2. Be able to plan the evaluation of a learning programme
2.1 Develop a framework for the evaluation of learning programmes
2.2 Devise objectives in order to achieve evaluation aims
2.3 Select methods for evaluating the effectiveness of a learning programme
2.4 Select methods for collecting data to evaluate the effectiveness of a learning programme
Name xxx
Group xxx
Programme to be Evaluated : The effectiveness of involving information and communication technology (ICT) in the use of statistical methods for data display
Hypothesis – Our involvement of ICT in the use of statistical methods to display data was effective
Aims/Objectives
The aim of this evaluation process is to find out from learners how the use of a spreadsheet package compared to the use of graph paper in displaying statistical data. Participants’ feedback on the effectiveness of involving ICT in the use of statistical methods for data display would be collected and analysed.
Methodology for Evaluation
The methods that should be used for the evaluation include open and closed questions, one-minute question, tally sheet and questionnaire. The questionnaire would be designed using smart survey (https://www.smartsurvey.co.uk/). The one minute question is a commonly used classroom assessment technique. It really does take about a minute and, while usually used at the end of class, it can be used at the end of any topic discussion (Angelo and Cross, 1993). Its major advantage is that it provides rapid feedback on whether the trainer’s main idea and what the students perceived as the main idea is the same.
Primary data would be used as learners would be asked to complete a questionnaire to state what their experience of using ICT to handle and display data is. They would also be asked to compare the experience with the use of graph paper for the same activity. The Idea is for me to be able to determine the effectiveness of using ICT as opposed to the manual method of using graph paper. The data would mainly be qualitative as the evaluation is statistical in nature. It will involve the use of graphs, charts, tables and pictograms.
Method of Data Collection
The methods for collecting data for this evaluation are as follows:
- summary of responses obtained from open and closed questions
- identifying statistical or quantitative data (including graphs, charts, tables and pictograms) from tally sheets
- frequency analysis based on responses to questioning or questionnaires
- use of online data collection format
The assessment/evaluation process involves the use of multiple sources of information collected in a variety of contexts
When to use the Evaluation methods/tools I plan to use the evaluation methods/tools that I listed above within the last 30 minutes of my training session. For example, learners would be asked to complete the questionnaire before they leave the classroom. By this, my target is to get as many learners as possible to complete the questionnaire.
Provide the results to others After the evaluation process, I plan to make the results available on my organisation’s virtual learning environment (VLE). This is to ensure that my colleagues and other members of staff can access it even when they are off-site.
Using the results for self- development The outcome of this evaluation would let me decide on what is working and what needs to be changed in other to be delivering inclusive and motivating sessions to my learners. “The primary purpose of assessment and evaluation is to inform training and to promote and encourage learning—to promote optimal individual growth. In order to provide information vital to the trainers, assessment and evaluation must be an ongoing and integral part of the training/learning process”.
3. Be able to evaluate the effectiveness of a learning programme
3.1 Apply selected methods to evaluate the effectiveness of a learning programme:
3.2 Apply selected methods to collect data to evaluate the effectiveness of a learning programme:
3.3 Analyse data collected to evaluate the effectiveness of a learning programme:
3.4 Apply relevant guidelines and legislation relevant to data collection and analysis: (see 1.5)
3.5 Present an analysis of evaluation results:
3.6 Explain how evaluation results can be used to improve the effectiveness of a learning programme:
In my training practice, I have successfully evaluated the effectiveness of involving information and communication technology (ICT) in the use of statistical methods for data display. The method I applied in the evaluation process was to make use of a questionnaire designed on the Smart Survey website (https://www.smartsurvey.co.uk/). Within the questionnaire, learners were asked four closed and one open question. Twenty-one (21) learners responded to all the questions asked. Below are the five questions I asked my learners to respond to:
Q1. You have analysed and displayed statistical data using graph paper, as well as ICT, which method do you consider more effective?
1. ICT is extremely effective
2. ICT is moderately effective
3. Both methods are just okay
4. Graph paper is moderately effective
5. Graph paper is extremely effective
Q2. Which method (ICT or graph paper) is more accurate in the analysis and display of statistical data?
1. ICT is extremely accurate
2. ICT is moderately accurate
3. Both methods are just accurate
4. Graph paper is moderately accurate
5. Graph paper is extremely accurate
Q3. Which method (ICT or graph paper) is more time effective in the analysis and display of statistical data?
1. ICT is extremely time effective
2. ICT is moderately time effective
3. Both methods are just okay
4. Graph paper is moderately time effective
5. Graph paper is extremely time effective
Q4. How satisfied are you with the use of ICT for statistical data analysis and display?
1. Extremely satisfied
2. Moderately satisfied
3. Neither satisfied nor dissatisfied
4. Moderately dissatisfied
5. Extremely dissatisfied
Q5. Which method (ICT or graph paper) do you consider more effective in the analysis and display of statistical data and why?
The methods I utilised in collecting data for this evaluation are the summary of responses obtained from open and closed questions, and frequency analysis based on responses to the questionnaire. According to a research available online, the evaluation process involves the use of multiple sources of information collected in a variety of contexts. The collected data have been analysed and presented using the Smart Survey result analyser. The analysed result has been resented in the form of a bar graph which I have included for examination. I have also presented a summary of the collected data in a tabular form. The tabulated results, as well as the bar graphs clearly show that the use of ICT in the analysis and display of statistical data is much more effective when compared to the use of graph paper. Learners seized the opportunity of the open question (Q5 on the questionnaire) to highlight why they prefer ICT to paper analysis.
2018-4-23-1524494244