Want to make interactive content? It’s easy in Genially!

Over 30 million people build interactive content in Genially.

Check out what others have designed:

NORMANDY 1944

Horizontal infographics

BEYONCÉ

Horizontal infographics

ONE MINUTE ON THE INTERNET

Horizontal infographics

SITTING BULL

Horizontal infographics

RUGBY WORLD CUP 2019

Horizontal infographics

GRETA THUNBERG

Horizontal infographics

Transcript

110 survey responses from FET educators about quality assured assessment processes

72%

Agreed that QA processes strengthen the principles of assessment

Click for overview of survey themes

Open & collegial, leading to authentically enhanced quality

Managerialist, bureaucratic and illogical

Assessors are concerned about assessment reliability and the consistency of awards and standardsAssessors need the opportunity to benchmark

33%

This increases to 41% at QQI Levels 5 & 6

Educators’ Experiences of Quality Assured Assessment in the Further Education Sector:Insights from a practice-based innovation, using ePortfolios for Assessment

Rachael Doherty, MA in Educational Practice

of assessors fully compile their learners' assessment portfolios

Click for overview of focus group themes

A way forward with ePortfolios KWETB ePortfolio Project Focus Group

with assessors having experience being an authenticator having positive effect

Assessors are overwhelmingly positive about the use of technology in asessment

to

Additional QA support is required for new educators

PROMOTION OF ASSESSOR ROLE

PROFESSIONAL LEARNING AND DEVELOPMENT

PROCEDURES TO SUPPORT INTERNAL MODERATION

Implications for Practice

Implications for policy

Survey respondants had vastly different experiences of QA, from:

OVERVIEW OF FINDINGS

Quality assurance

ASSESSMENT

TECHNOLOGY

EducatorsEXPERIENCES

QQI (2018) have themselves determined that: “..designing and implementing summative assessment procedures and tasks that are: a) valid and reliable for the purposes of certification; b) nationally consistent; c) fair and suitable for use in competitive allocation of places in higher education; may be, realistically, we suspect, beyond the unaided capabilities of many providers under the current arrangements.”

Opportunities for assessors to take on the role of authenticator as part of an open and collegial quality process were identified as valuable experiences that enhance quality. In order to explore whether answers to the below statement was different based on whether the respondents had authentication experience, responses were grouped based on this variable. A Fishers exact test found that the frequencies were significantly different (p = .046). Self reporting of assessor confidence in their strategy for quality assuring assessment is significantly different on the basis of authentication experience (p<0.05, Fishers exact test)

The administrative burden associated with compiling and organising every learners assessment portfolio is not recognised by management or in policy/procedure as part of an assessors role. Also, this reduces learners autonomy associated with constructing their showcase portfolio of assessment pieces.

Assessors enrolled in ePortfolio project felt it had benefits for student learner centred assessment, learner autonomy, organisation and QA. It also supported staff to develop greater confidence and clarity about the assessors role in QA. All three of the key survey themes were supported by the opinions of all focus group participants, supporting the triangulation of the study findings (Caillaud & Flick, 2017). Issues of excessive administrative burdens that lack clear purpose were discussed. The use of ePortfolios was described as going some way to reducing this burden. Again, comments by participants illustrated that their complaints are not simply about the amount of work the current system demands, it is that because what is being asked of them feels senseless. Instead, they want an enhanced quality system that is more authentic. The following response highlights this: "Where's the focus on the actual teaching and learning? Hopefully the ePortfolio changes this, because since we've started using it, there's less emphasis on the tick box exercises…It just makes sense. And it's not to be getting out of any work, because like we've all just mentioned, these tick box exercises are just a waste of time." Inauthenticity in the process comes up several times. The following participant names a key issue as a perceived lack of trust in educators in the sector, stating: "We were constantly giving (students) feedback in person, in real time. There's no trust in the tutor that you are providing feedback. It's meaningless writing on a form that you have given a student feedback, because that may or may not be true. It proves nothing." Trust and accountability are inextricably linked, and recent research featuring the higher education sector describe how bureaucratic accountability can burden educators with administrative reporting, which does not contribute to any improvements to quality (Ehren et al., 2019). Only when there is open and genuine conversation about the quality of assessment can there be a meaningful discussion about change. For this to occur, FE tutors must be trusted to fulfil more of their core roles without surface level monitoring and to be active participants in a deep and meaningful conversation about quality. The ePortfolio project promoted open discussion about assessment processes among educators. Some of the participants found this very valuable. One participant also felt more confident about their role in the process when using the new system. "I've gotten one ePortfolio ready for IV and I feel a great sense of confidence. Whereas usually, I'm worried because I could be missing a whole section. I love the organisation of it, and I'm really enjoying it and the community of practice is just brilliant and I hope it keeps going." Opportunities such as this voluntary pilot allow for professional development, peer learning and a genuine improvement in quality, with a supportive framework for educators. While professional development has traditionally been delivered by external agencies, research and experience has demonstrated that professional learning is often best located “in house” where educators can focus on problems and improvements in areas of relevant practice, while offering support and opportunities to learn from each other (Hara, 2009). Other studies have also acknowledged the potential of similar “technology in education projects” for staff, as being an important spark for professional learning (Ambler et al., 2014). The knowledge and expertise held by professionals within an organisation, who have the added understanding of local policies, practices and student needs cannot be bought (Lieberman & Miller, 2011). The focus group conversation also dealt extensively with the issues of assessor confidence regarding reliability and consistency of standards in the sector. The group felt that a move to ePortfolios from the paper-based system provided multiple opportunities that were previously impractical. These included the sharing of exemplar ePortfolios for various disciplines and to represent a range of grades and assessment review meetings, similar to those held at both second and higher-level institutions that enable educators to collaboratively reach consistency in their judgments of student work (National Council for Curriculum and Assessment, 2019; Quality and Qualifications Ireland, 2013). Previously, this idea would have been very challenging, with small education centres and colleges often having only one tutor per academic discipline, making in house cross moderation virtually impossible. Looking further afield, moderation across colleges or providers was highly impractical, with moving and transporting boxes of student assessment folders logistically difficult and problematic from a security point of view. One idea put forward by a participant and strongly endorsed by all others was "if we compiled fictitious ePortfolios with different pieces of work, and then got a small focus group to all grade them using the same marking schemes and see what the results are.” This suggestion was thought to be the most constructive and collegial starting point, in a sector where there is little precedence for second marking and for assessors whom initially may feel vulnerable about cross moderation activities. Finally, from the focus group discussion it seemed that the move to ePortfolios provided benefits for students in managing and having autonomy over their learning. For example, they were able to see their marks and map their overall progress. One respondent noted that “students are able to go back into the ePortfolio and go, ‘oh, yeah, I'm on task for that or I have to up my game for that’”. It was identified by several participants that the portfolio structure was a support to students allowing “…students to see the empty pages and where they need to fill in and what they're missing. They were able to tell me exactly what they were missing, which is unreal”. It also supported the continuity of assessment and gave students a greater opportunity to learn from the process with one participant remarking: “I've been able to correct as I go, and then I've put all their feedback up, they've got it straightaway and then when they come to their next assignment, they're ready.” These comments highlight that the ePortfolio model may elevate the value of the assessment as and for learning, allowing greater opportunity for students to use ongoing assessment and feedback as opportunities to improve and to monitor their own learning, with a greater understanding of assessment criteria (Earl, 2007). These improvements in student experience are important examples of quality enhancement in assessment. Not only do they align with the principles of Universal Design, but they suggest that the ePortfolio model has the potential to strengthen the constructivist, learner-centred process of assessment that it was intended to support (Wolf, 1998). If portfolios are to remain as a primary feature of assessment and authentication in the sector, this is an area that warrants further exploration and analysis in the near future.

Online submission systems were described as being a support to students as well as for educators: “Less stress for students printing out assignments. All assignments will always be submitted online from now on, no hard copies.” This sentiment is mirrored in the findings of a recent survey by QQI, examining how COVID impacted teaching, learning and assessment in the Irish Further Education sector, which found that educators preferred the online tools for provision of feedback and submission of assessments and would like to retain these practices (QQI, 2020).

“Due to Covid, the glass ceiling has been broken" “Technology in the classroom has really come to life due to COVID. The advancements have been extraordinary over the past year and have definitely benefitted both teacher and student.” “The use of electronic portfolios will help efficiency, quality, uniformity throughout the organisation”. Other participants expressed concern about the number of new digital assessment options being presented to staff, such as Moodle, Microsoft Teams and ePortfolios and a lack of consistency in approaches. It is suggested that “one system across the ETB sector should be introduced”.

  • Knowing the standard with ePortfolios
Individual providers or even individual colleges and centres could take immediate action to address concerns about consistency of standards. Providers could look at international good practice in assuring the validity and reliability of assessment and implement such approaches. These approaches could be educator led to ensure a supportive and collegial experience. A move to ePortfolios from the paper-based system provides opportunities to makes this feasible when it was previously impractical. The further education support service website provides a short guide on “Ensuring Consistency of Assessment across Centres” (Further Education Support Service, 2015). This contains suggested models, including the appointment of lead assessors and the hosting of moderation meetings across centres in a multicentre provider. However, other suggestions were provided by participants of the survey and focus group in this study that may be more useful for a first tentative move towards a system of consistent quality standards. These included the design of exemplar ePortfolios in a range of modules, anonymous second marking activities and assessment review meetings, similar to those held at both second and higher-level institutions that enable educators to collaboratively reach consistency in their judgments of student work (National Council for Curriculum and Assessment, 2019; Quality and Qualifications Ireland, 2013). Previously, this idea would have been very challenging, with small education centres and colleges often having only one tutor per academic discipline, making in house cross moderation virtually impossible. In addition, the findings from this research have identified the growth in confidence in quality assurance and more specifically, “knowing the standard” that participation in the authentication process has provided to assessors. In particular, the responses of many educators who had experience of being internal verifiers underlined the value of involvement in this process. While internal verification does not involve the moderation of standards, it does provide the verifier with a rare glimpse of the grading standards set by other educators. As a result, it is recommended that opportunities for educators to act as internal verifiers should be encouraged and supported by management of programmes and providers.
  • Proposed new ePortfolio model for Further Education
Results from the focus group of this study suggest that the use of ePortfolios may support students learning and assessment. Below is an outline of a proposed new model of ePortfolios for Further Education. This model extends beyond being a vehicle for assessment evidence and instead would include all material relating to teaching, learning and assessment. In doing this, the student regains control of the scaffolding and compilation of their assessment material while being able to view it in the context of their entire learning journey. This supports the learner-centred and participative pedagogical approach in the sector (SOLAS, 2014). Elements of the model that support teaching and learning will include a collaboration space, where group work can be completed by multiple students, reflective log spaces for students to document their learning process and a greatly increased bank of accessible content. Built in digital tools have the capability to read text aloud, to translate text into multiple languages and to allow students to dictate their ideas and have them converted to written text by the ePortfolio. In addition, it supports the presentation of content in a variety of formats all on the same page. This approach would fully align with the principles of Universal Design for Learning, which is currently being promoted as a strategy to support learning and diversity in the sector (Quirke & McCarthy, 2020). Finally, the use of the ePortfolio model would have benefits that fall outside of the modules learning outcomes. It would support them in building digital skills, an important aspect of all modern quality education and vital for their futures. It would also foster autonomy and student-centred learning, supporting assessment as a process, rather than it being viewed simply as a product. An overview of the benefits of this model is below. The other function of the existing portfolios in the sector is to support the authentication of the assessment process. This ePortfolio model would contain an assessment section. Students would be involved in the management and presentation of this section, which will represent their product and showcase of assessment material. This would greatly reduce the administrative burden associated with quality assurance measures, as it would become a more authentic process of continuous assessment and quality. It would be updated continuously with assessment submissions as well as with feedback and grading that would always be visible and available to students. This would greatly increase transparency in the process, with all marking schemes and feedback available to students in real time. This would improve current practices were physical portfolios of assessment must be securely retained by assessors and only available to students sporadically. The final step, when the students’ work is completed and their portfolios of assessment are ready for internal verification and external authentication, is now markedly simplified and streamlined. The assessment section of the ePortfolio can simply be copied and isolated for authentication. This would mirror the ePortfolio for assessment as researched in this dissertation and would represent the product of quality assured assessment that requires presentation for authentication. This potential approach is currently being explored in practice at programme level and is something that may be of interest to other educators. Future research will be vital in order to examine the impact of this model and its associated benefits and limitations for both staff and students.

Survey quotations "Doing IV myself gave me more insight into general standards and approaches to measuring learning across multiple modules taught by different staff. The process of having my folders go through IV was less enlightening, as it is all done behind closed doors and feedback is only given if there is an error in the addition of numeric grades or a signature missing." “I think seeing a variety of work as an IV is a positive process, it opens up conversations, allows the sharing of ideas and helps colleagues iron out minor issues” “Internal verification can be a positive learning tool in picking up good practices and seeing how other colleagues interpret assessment"

Survey quotations "The QA processes need to be fully understood by management to ensure they are rolled out correctly. They should not become a labour-intensive process that at times does not make logical sense, it’s purpose needs to be clear and not just additional paperwork for the sake of it." “Lack of appreciation for deep learning. Focus seems to be mainly on yellow tape and paperwork.” “Just a lot of unnecessary duplication of marking sheets which are required...too much paperwork to submit” "Up to now we were told we had to have three written feedbacks for each student per module - this felt very artificial and very much like a paper exercise to satisfy the external examiner or prove that we were giving feedback. We were giving feedback anyway. Documenting it on a specific form was time consuming, especially if the class was large."

A clear subtheme from the survey, in the area of assessor confidence, was that new staff in particular require additional support, as stated very clearly by multiple participants, with one stating “I think colleges need to give new teachers more support in relation to how it works in their particular institution. In my case it's left to each individual teacher there's no real guidance or clarity.” Many survey responses captured some difficulties and challenges that were faced by new and inexperienced staff, who often reached their first assessment period unprepared and lacking in confidence. One participant recalled that in their first assessment experience they were “…left virtually alone without help or support to complete assessment and I made so many mistakes and interpreted things incorrectly. Training and support would be vital.” Another concurred that their first experience “…was very difficult” and they felt “that a lot of emphasis needs to be given to helping new teachers through assessment periods and peer mentoring would be an invaluable help in this area.” In order to determine whether levels of assessor confidence was affected by respondents length of service, respondents were grouped based on whether they had up to or greater than ten years service. A Fisher’s exact tests found results were significantly different (p = .031) in response to the statement below:

Survey quotations “There is a distinct lack of quality control and comparison of learner's work and grades within centres never mind between centres or nationally” “Prior to working as an EA” they had “no opportunity in FE to see work from another teacher of the same subject.” “I believe that more transparency and sharing of materials across ETBs would vastly improve the overall QA. Simply relying on EAs for this is not enough in my opinion”. “Double marking of a percentage of assessments within the college may be beneficial to QA”

Educators’ Experiences of Quality Assured Assessment in the Further Education Sector:Insights from a practice-based innovation, using ePortfolios for Assessment

Rachael Doherty25th August 2021

Introduction

Quality assurance

ASSESSMENT

TECHNOLOGY

EducatorsEXPERIENCES

Background and Rationale

Assessment

Authentication

Quality Assuring Assessment

Portfolios support learner-centred model of education Portfolios support intensive recording and oversight in system ePortfolio Project

Survey: Main Findings

"Behind Closed Doors" Issues with Defending QA with Administrative AccountabilityAdministrative Burden -”Ticking the Boxes” Benefits of an “Enhancing Quality” Culture

Title hereLorem dor

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

survey of assessorsMAIN Themes

Assessor Confidence Support for New TeachersConcerns about Reliability - Knowing the Standard

* p < 0.05

Technology Enhances Assessment Processes for Assessors and Students Increased Accessibility and Efficiency Use of ePortfolios for Assessment

Focus Group: Main Findings

Enhancing Quality Assurance

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

EpORTFOLIOFOCUS GROUPMAIN THEMES

Developing Confidence in Reliable Standards

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Reimagining Student Centred Assessment Processes

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Implications for Practice

Knowing the Standard- Quality Assuring Fair and Consistent Assessment Assessors meeting together Sharing of graded exemplars Double marking of samples Discussion of queries and borderline grades

Implications for Policy

PROMOTION OF ASSESSOR ROLE

PROFESSIONAL LEARNING AND DEVELOPMENT

PROCEDURES TO SUPPORT INTERNAL MODERATION

Future Plans

CONTINUOUS IMPROVEMENT

STUDENT EXPERIENCE

INTERNAL MODERATION

ePortfolioof Assessment

"Behind Closed Doors" Issues with Defending QA with Administrative AccountabilityAdministrative Burden -”Ticking the Boxes” Benefits of an “Enhancing Quality” Culture

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Title hereLorem dor

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

survey of assessorsMAIN Themes

Assessor Confidence Support for New TeachersConcerns about Reliability - Knowing the Standard

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Technology Enhances Assessment Processes for Assessors and Students Increased Accessibility and Efficiency Use of ePortfolios for Assessment

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Enhancing Quality Assurance

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Title hereLorem dor

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

EpORTFOLIOFOCUS GROUPMAIN THEMES

Developing Confidence in Reliable Standards

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Reimagining Student Centred Assessment Processes

LOREM IPSUM DOLOR Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Who Compiles Learners' Portfolios of Assessment?

Confidence that assessment strategy is fully compliant with QA process

Confidence that assessment strategy is fully compliant with QA process

Confidence that assessment strategy is fully compliant with QA process