The four pillars of great assessment - Issuu (2024)

The four pillars of great assessment: a case study

Marco Gemelli

Deputy Head of Senior School

For the past eighteen months Jen Horay and I have been following the Assessment Lead Programme (ALP) online course run by Evidence Based Education (https:// evidence-based-education.thinkific.com/). The school-sponsored, IB-approved course – a BETT Award 2019 finalist in the category Whole-school aids for learning, teaching and assessment – aims to help teachers and school leaders maximize assessment for learning by enhancing formative and summative assessment strategies.

A particular feature of the course is that at least two colleagues need to be enrolled at a given time, and they need to commit to regular meetings throughout the course to discuss their progress and jointly plan their next steps. We all know how hard it is for teachers to carve out time for dedicated work given our busy professional – and personal – lives, and it is fair to say that Jen and I have at times struggled to keep up with the course in the midst of a global pandemic and its additional challenges (lockdown, remote learning, hybrid learning, to name but a few). However, these meetings have been extremely useful. They have enabled me to gain a much fuller grasp of the specific challenges faced by the Science department when setting out their curriculum and when assessing students, and I have relished discussing approaches to assessment with a colleague outside my department. As we are finally approaching the end of the course, I wanted to share with you some of my take-aways.

The four pillars of great assessment

The four pillars of great assessment are purpose, validity, reliability and value.

In this article I will discuss them within the framework of a case study: the Y13 Maths Analysis & Approaches (AA) Standard Level (SL) mocks my students took at the end of January.

The first question we should ask ourselves when assessing students is: Why are we assessing them? What is the purpose of this particular assessment? In other schools assessment may well be synonymous with marking, data input and accountability. Teachers give students a test because reports are coming up; they mark books to comply with directives from HoDs or SLT. St George’s does not adopt such a narrow view. Yes, we review homework, we mark tests, and we enter results in departmental spreadsheets, but we are much more ambitious than that. We aim to use assessment, in its varied forms, to generate high-quality information which is used to make better decisions about student learning, which will then lead to an improvement in student outcomes.

Fig. 2 – Better assessment leads to improved student outcomes (ALP, 2021)

Y13 mocks are an interesting example in that they serve two main purposes: a summative purpose, that is, providing a holistic assessment of each student’s mastery of the subject which will inform termly reports, centre-assessed grades to be submitted to the IB (should they be required, as happened last year) and university predicted grades; and a formative purpose, namely identifying strengths and weaknesses of each student in each area of the syllabus covered to date. Because of this dual nature, compiling mocks is a time-consuming exercise, especially in subjects like maths or the sciences which require lots of different questions – unlike, say, English or History (although colleagues teaching these subjects will probably need to spend more time marking it, and may well have to grapple with issues of inter-rater reliability – see below).

I – together with the rest of the maths department – strongly believe in the formative purpose of our mock exams. A key vehicle to achieve this aim is compiling and then using a detailed question-analysis spreadsheet which shows the marks gained by each student in each question, together with the mean (average) and

standard deviation (a measure of spread) for each paper. I use it to advise students on the areas they haven’t yet mastered and, crucially, to make evidence-based decisions on which topics need revisiting, and with which students. It forms the basis of my planning for subsequent in-class activities, as well as informing the targeted after-school ‘intervention’ sessions I will be offering to selected groups of students. Comparing my class results with their own previous results in different assessments, and with the performance of previous cohorts in the mocks, helps me build a more accurate picture of the strength of each cohort – even if comparisons such as these ones are always fraught with difficulties, as mocks change from year to year, and so do students!

Fig. 3 – Question-level analysis for my Y13 AA SL mocks (students’ names omitted)

As shown by Fig. 3, while the overall grade distribution was rather pleasing for a Maths AA SL class, a few topics clearly needed revisiting, starting from sigma notation with Geometric Progression (GP) – Question 3 in Paper 1 – where seven students out of 11 scored 0 marks, while the other 4 students got full marks – a perfect example of a bimodal distribution! In my first lesson after the mocks I focused on a few questions including this one, we went through them as a group, then students had the opportunity to individually practise similar questions in order to consolidate their learning.

Validity, the second pillar of great assessment, and arguably the most commonlyused word in discussions about the quality of any assessment, is often misunderstood. Rather than referring to a ‘valid test’, for instance, we should be talking about ‘valid inferences’ that we draw out of the information generated by the testing process. Does the test measure what it is intended to measure? Does it provide information which is appropriate for the intended purpose? In order to be able to draw valid summative inferences from the mocks – is this student currently

working at a level 4? 5? 6? – the starting point was to address face validity (by only including questions on Maths AA SL topics) and content validity (by aiming to cover each major topic area). Clearly, maths mocks which leave out an entire topic (integration) or even a smaller sub-domain (conditional probability) are more likely to return a skewed set of results. While this may seem obvious, it is worth noting that shorter tests carried out in lessons, which necessarily include fewer topics, may well fail to address content validity and as such return inaccurate information about student learning.

Agnes, who teaches the parallel Y13 AA SL class, and I met several times to discuss the questions to be included in the mocks, cross-checking the list of topics covered, and the approximate number of marks for each topic, in an effort to create our own version of a valid assessment. As we had been made aware that Y13 students now have access to several websites containing all IB past paper questions (including the official IB sample papers for the new syllabus), we decided to use sample papers compiled by independent providers (Scholastic and In-Thinking). We looked at several recent past papers (older papers tend to be easier) to be fully acquainted with recent ‘trends’ in IB questioning such as more reasoning and application marks, a higher number of unstructured questions carrying several marks, including what I call the ‘sting’ – a particularly challenging question worth 5 or 6 marks (out of 80) right at the end of each paper. The majority of students will have just a few minutes to attempt this question, which can be algebraically challenging, tests higher-order thinking skills, require a particularly skillful application of the graphic-display calculator, or be set in an unfamiliar scenario. As such, it acts as a discriminator at the top end i.e. helps separate students at level 7 from the other students.

Agnes and I went through multiple drafts. I found the first draft too challenging and we decided to substitute a few of the initial questions with some easier ones that a ‘level 4’ student might be expected to successfully tackle. This is when we decided to write some of the questions ourselves (an easier question on binomial expansion) and to amend others (breaking a challenging question on exponentials and logarithmic functions into two parts to provide some scaffolding). As both Agnes and I teach the Higher Level (HL) course in Year 12, we needed to remind ourselves not to make this SL mock too hard!

We paid particular attention to the clarity of the wording for each question, and used IB command terms whenever we edited a question. This was to avoid adding any extraneous load: the difficulty ought to be mathematical, not linguistic (even if, as I have often discussed with Monica and Niki, IB questions often require a sophisticated understanding of language!).

We changed the sequencing of questions from their original versions in an attempt to arrange them in increasing order of difficulty within each section (A and B), as IB papers strive to be. Agnes then solved both papers (thank you Agi!) in order to gauge their relative difficulty, and the time most student were likely to take completing them. A test which is too long would likely compress grades within a narrow band, as the majority of students would not have the time to attempt several of the final questions and would therefore score 0 in all of those questions, and this would affect the validity of the results.

Addressing reliability, the third pillar of great assessment, meant focusing on the internal consistency of the assessment, as well as to the consistency of different markers (inter-rater reliability and intra-rater reliability): clearly student scores should not be affected by whether it is me or Agi marking their paper, or the time of the day when we mark them. I would argue that inter-rater reliability is less prominent in maths tests because mark-schemes are quite prescriptive: 2-mark questions, for instance, often have a method mark (for attempting to solve the question with a valid method) and an answer mark (for a correct answer). However, questions carrying more marks are less straightforward, especially when more than one correct method can be used to solve the question. Agi created a very detailed mark-scheme and we decided to consult each other whenever we found an alternative solution, or if we were unsure on how to mark a particular question.

A key question we should ask ourselves whenever assessing students relates to the fourth, and last, pillar of great assessment: its value. Are the benefits to students’ learning greater than the costs incurred by teachers and students? Agnes and I have spent many hours putting the tests together, marking them and analyzing the results; one would hope that students will equally have spent many hours preparing for them. Was it all worth it? In the case of my Y13 maths mocks I would argue that this was definitely the case. From a summative point of view, performance in the two papers was very consistent on average (the two means are virtually identical) and in line with previous assessments. Yet, the difference in spread (students’ scores in Paper 2, the calculator paper, have much lower values of range and standard deviation) suggests that students are similarly adept at using their graphic calculators, yet vary in their algebraic skills. The latter ties in with my classroom observations, while the former is welcome news. The majority of students achieved their Key Stage aim and this is very reassuring as it would support the view that they have not been penalized by the lockdown, or by remote or hybrid learning – quite possibly the opposite. Formatively, I will continue to use the detailed questionanalysis spreadsheet in two ways: at whole-class level I will ensure we revisit certain topics, including sketching graphs, sigma notation, compound probability, and practise not yet well-developed calculator skills such as finding the maximum rate of change. And at individual level, I will work with each student to help them improve exam skills – accuracy and speed of execution – and their mastery of specific content.