Comparative Judgement and Quality

The school at which I work has around fifteen regular “feeder schools” – local primary schools whose children attend our secondary school. As part of the entry process new students (along with their parents) have an interview with the Head of School to discuss their experiences of schooling, what they are hoping to achieve in their new school and to answer any questions or alleviate any anxieties that students and families may have. Usually the Head of School will make reference to the latest student report from primary school in order to discuss how things are going and what the student’s goals are for the coming year.  

What can confound the discussion are the differences in the notions of quality of achievement between different feeder schools. As part of the funding arrangements in Australia schools are required to rate student achievement relative to the standard expected of students at that year level. The students are assessed to be “At standard” (in a coloured band for a particular year level) or above/below standards linked to particular year levels. These reports tend to look like this, with student achievement represented by a dot and the expected level indicated by the vertical coloured band:

 

For most of the schools these reports are reasonably accurate – the judgement agrees with the results from the NAPLAN national assessment and also our teachers’ assessments when those students enter our school. However, there are some feeder schools that are systematically biased in their judgement of student progress. This bias (most often overestimating the achievement of students) has to be factored in when interpreting the reports from those schools.

Although there are some inducements for schools to report that students are more proficient compared to their peers than they actually are, I believe that in these cases it is more of a difference in their assessment of the relative quality of student work. I use quality to describe what students can do and its relationship to what regular students from around the state can do at the same stage of schooling. A work of good local quality would be commensurate with that produced by regular students from a particular school, while good global quality is commensurate with average students from around the state.

In essence the question that the reports of biased schools are answering is “How is this student achieving relative to other Year 6 students that have attended this school in the past?” rather than “How is this student achieving relative to other Year 6 students in the state?”

All teachers use a local sense of quality to judge student work – what is important is how close that local sense of quality matches what quality looks like across the national sample (a national sense of quality). When your idea of relative quality is too strongly linked to the pool of students that are (and have) attended your school it is easy to lose sense of what quality looks like in the entire population, particularly if your local students are stronger or weaker than the average. Having a good sense of quality is important – as a teacher it is difficult to know what to teach and what feedback to give students if you don’t have an accurate sense of what a quality piece of work would look like. To combat this you need a means of recalibrating what quality of achievement looks like compared to a larger sample of students.

Similar difficulties in judging quality arise in the final examinations for our secondary students, particularly in subjects where the exams require more complex performances (like essays and text responses) and are exacerbated when there is only one teacher for that subject. In these cases, when there are fewer teachers and a smaller sample size to calibrate the sense of quality it is easy to lose track of what global quality looks like. One way to combat this is to have teachers work as examiners for the exam board. Being part of the conversations with other examiners about what quality looks like and then seeing hundreds of scripts from students all over the state can help refine that sense of quality. However, this is available only for particular courses and have a limited number of places.

A possible alternative to this process would be to use Comparative Judgement. I believe that comparative judgement has a strong role to play in helping schools refine their concepts of quality. Chris Wheadon and his team at NoMoreMarking have recently embarked on a trial involving marking pieces from 220 schools across the UK, with teachers from each of the schools being involved in making judgements about student scripts from other schools.  

One of the important outcomes of such a process is the opportunity to see and judge hundreds of samples of work from other students at other schools. Making judgements about quality and seeing the range of quality in work from across such a large sample would help a teacher get a better sense of what quality would look like in a national sense and so would help calibrate the local sense of quality. The advantage of a process like comparative judgement is that large samples can be stored electronically and the training can be quite short as it isn’t dependent on knowing the ins and outs of a rubric. I am looking forward to the reports from the trial and how it larger cohort judging be expanded in the future.

The Negotiated Assessment

The negotiated assessmentI have always been fascinated by the methods and principles of assessment and how it applies to classroom practice. I want to know whether students in my class are improving, whether my instruction is helping (or not helping) students to improve and what I can learn from my colleagues in the next classroom. Being part of various teaching teams across a number of schools I have been struck by an interesting phenomenon that seems to occur reasonably often and, despite seeming innocuous, can actually be

 

This phenomenon is the negotiation of the items or focus of an assessment after a sequence of teaching and typically goes a little something like this:

 

We, as a teaching team, might start with a curriculum that we agree to, but often is light on the details of what students are supposed to know and be able to do as a result of our teaching. It might comprise something like “Teaching operations with fractions” with some chapters and exercises from the text as a sample. This is our intended curriculum – the domain of knowledge and skills that we are trying to teach students. Then we head to our classrooms and teach what we understand the curriculum to be – the enacted curriculum. The overlap of the classroom domains in the classroom might look like this: 

 

     
 

It then comes to test time. Someone writes a test, based on their understanding of the curriculum and it gets passed around the group but, because of the lack of overlap in the enacted curriculum in each classroom, the test needs work. We feed in our comments:

 

“I didn’t teach addition of fractions involving different denominators. Can we get rid of that question?”

 

“My kids wouldn’t be able to work out the fractional amount of a whole from a worded question.”

 

“I spent a fair bit of time on ordering mixed numbers – let’s add a question about that.”

 

With that sort of a conversation what we are doing is negotiating the domain of knowledge and skills the students need to demonstrate on the assessment based on the overlap of the enacted curricula in our classrooms. We are effectively renegotiating the assessment domain so that it looks like this:

 

​          

 

This may be a good thing for assessing the ability of our students to demonstrate the skills and knowledge that were commonly taught across the classrooms, but it is a terrible device for assessing the ability of the students in the intended, full domain of the curriculum. It looks like we are consistent in our classrooms (because the scores on the test say so) but they are just an artifice of the negotiation. We have only tested a fraction of what was actually taught in each classroom and it is not a good representation of the domain that we initially agreed to.

 

In my mind a curriculum should work along these lines:

 

• As a group of teachers (teaching, say, maths) make some decisions about what the aims and outcomes of our work will be over a specified amount of time. These decisions are informed by national curricula, school curricula and the expertise in the team. In the best cases we are quite specific about what we would expect students to know and be able to do at the end of the sequence of learning.

 

• Once we have agreed on the goals and outcomes then teach to the goals and outcomes of the curriculum. Although this may look a little different in each classroom the general focus and intent of the learning is defined according to the curriculum we have agreed to before we started teaching. We are clearer about what we are trying to achieve.

 

• An assessment is constructed which assesses the domain of the intended curriculum.  

 

If we were to represent the domains of the intended curriculum and the enacted curriculum in each classroom in the case above it might be like this:

 


 

I believe there are two ways in which we can approach the negotiated domain problem. The first is to describe our intended curriculum in more detail. The looser the curriculum the more difficult it is for teachers to be consistent with what they are teaching and so the enacted curriculum will be more disparate.

 

Another way is to change the assessment process. The assessment could be developed before the unit is taught using the intended curriculum as the basis for the assessed domain.
You could also have someone who is not teaching in that team write the assessment. It is quite an interesting test of your intended to curriculum to have someone else, knowledgeable in the subject matter, to write the assessment without input from the teachers in the team. If the intended curriculum is loose it will mean that the assessment that is produced by the external person is less likely to represent the taught curriculum.  However, with a decent curriculum it can be a revealing process.

The First Year – What I have learned about implementing a phonics program (Part 2)

Part 1 of this blog described some the mechanisms of the phonics program, both in classrooms and in meetings, that I think have contributed to the learning of our students. The purpose of this section is to describe some of the things I have learnt about what happens when the team gets together to discuss what has been observed in classrooms over the week.

Harness what you have. The phonics team runs on the principle that the way that we will improve as a teaching team is to identify and adopt the most effective strategies from the members of the team. As a team we are always looking to improve our instruction and are on the lookout for better ways of teaching the phonics program. However, the first place we need to start, and the place where we will get the best traction, is to learn from the other members of the group. The tricky bit is not sharing practice, because that happens everywhere. It is sharing effective practice. To do this you need some of the conditions mentioned in the previous post. Some common way of identifying where students are making more progress in class compared to another (positive deviance), some sort of standard to compare that to and then a willingness to discuss and share. 

This approach is also useful because everyone has something to share. It is rare that one individual in the team will have all the answers – more often different people in the team will be able to contribute some effective practice. This helps build the concept of the team and the self-efficacy of its members, rather than an expert lecturing from the stage.
 
Grain size is important. When discussing practice, the closer the discussion gets to the specifics of what actually happens in the classroom the better. It is particularly easy for a team, when discussing the differences in the learning of students, and the related instruction, to use broad terms to describe what they are doing. “We did /k/” or “we discussed the GPC a couple of times” is really common. It is like when someone starts talking about a generalised term such as “guided reading”. Every person in the group has their own idea of what “guided reading” looks like and the assumption is that all of the other members of the group have the same idea. The broader the discussion the more room there is for interpretation and miscommunication. 

However, when you start to really break down what actually happened in the classroom differences quickly become apparent. Asking people in the group to show you, rather than describe, what they did (even if they have to play act the events) is really useful. It is when members of the group see this level of detail that the differences emerge – “I didn’t do that” or “ok, I just got the kids to start without that intro”. A good guide is that if you think the grain size is right, go a little smaller!
 
Independent eyes are useful. The school at which I work heavily resources the Prep literacy program. There is an assistant in each classroom for the literacy component of the day who rotates from classroom to classroom. One of the unintended consequences of having an extra person in the room is that they have the ability to see the instruction across a variety of classrooms. These sets of independent eyes tend to help the group identify the differences in the instruction between classroom when the teachers themselves are stuck. Many times, when two teachers were furiously agreeing that their instruction was the same, one of the assistants would point out that there was actually a big difference in what was happening in the classroom: “Jim, you spend less time working through the GPCs from the previous lesson. You are doing similar things but the amount of time you spend on them is quite different. Judy, your class wrote those letters on paper with lines rather than the mini-whiteboards”. These differences that are identified may or may not be causes of the variation in the learning of the students; however, being able to discuss these differences is an important part of the process.

It’s not you, it is your instruction on this concept. Sometimes it is difficult to separate the practice from the person. I am not sure if this comes with the profession but this conflation of instruction and self does seem to occur more with teachers than in other professions. For the discussions to go well it is important the team know and understand that it is the teaching strategy that is being discussed. This is easily changed. Nothing reinforces the concept that it is the instruction that is important than someone trying something new and finding out it works. And they didn’t have to sacrifice a skerrick of themselves.

Interlude – Asterix, Puns and General Knowledge

Over the most recent school holidays I had the fortune to spend time with my eldest nephew. He is a bright boy with a voracious appetite for reading. He has always read well – he figured out the alphabetic principle early and has adopted generalisations and exceptions well enough for him to access a great variety of teenage texts despite being nine years of age. 

 
As all relatives with only a very partial responsibility for a young child feels compelled to do, I arranged to provide him with a life-enhancing gift. I introduced him to one of my teenage loves: Asterix comics, with all the zany and pun-riddled text it contained. I have vivid memories of giggling away at the various gags at the Romans’ expense, the punny names (my sides still hurt from the first day I learnt the meaning of cacophony) and the random acts of gluttony and violence. I handed my nephew a copy of one of the Asterix books, sat on the couch next to him and waited for him to bring the laugh.
 
“I tell you, this book is fantastic. Funny, awesome characters. You’ll love it!”
 
My nephew was dazzled by the illustrations and tucked in to the character bios at the inside the front cover. He was devouring the book. I was still surprised when, after a minute of two, he lifted his head and asked about what one of the characters was saying, a pun of some kind. A minute later he asked the about another joke. His chuckles were reserved for some fight scenes and a boat that kept sinking. Soon he stopped asking. Something was very wrong with the world
 
He finished the book, carefully closed the last page and sat in contemplation for a second.
“I thought you said that would be funny.”

“It is funny – look, that guy’s name is Cacofonix, and he has a terrible singing voice! And hey, there are the British people and they even stopped the battle to have tea and biscuits. Hilarious!”
 
He was not convinced.
 
So, I picked up the book and started to leaf my way through it. When I got to the first couple of puns I tried to explain them to him. Nothing brings home the realities of how much contextual knowledge and understanding of the world is required for Asterisk than trying to explain a pun to a nine year-old. In fact, if you remove the puns and in-jokes from an Asterisk book you are pretty much left with a basic plot with three elements providing the laughs:

  • Characters slurring and acting stupidly while drunk.
  • Obelisk eating massive amounts of boar.
  • Gauls inflicting random act of violence on Roman soldiers.

What it brought home to me is that a good vocabulary and an understanding of the world has a strong influence on your understanding of a text. My nephew can read most words and knows at least one meaning for them. However, being just nine, he is still learning about history and geography, multiple meaning of words and how language can be used to entertain. All things that will come in time but without which a fantastic read is rendered to quite a patchy bit of comic relief. We need to teach students new vocabulary and homophonic meanings, important aspects of history and world culture. 
 
If for nothing else then the puns. Given I am a father and my entire catalogue of jokes are entirely pun-based, I think, for puns’ sake, a knowledge curriculum is vital.

The First Year – Some things I have learned about implementing a phonics program (Part 1)

A little over a year ago I had the fortune of starting to work with a group of Prep teachers teaching 5 year olds. When I began this blog it was my intention to share some of the things I was learning along the way and perhaps gain some advice. Somewhere along the way I got a little lost but I thought I would take the time to reflect on what I have learnt so far. 
As a secondary teacher with some familiarity with the alphabetic code and a little expertise in helping teams use data to inform instruction I thought I could help the group with the phonics program they were running. I didn’t exactly know what I was getting into – I worked hard to get some understanding of what a systematic phonics program would look like and asked as many questions of experts as I could.

The team was excellent – although the teachers were very young in their careers, they had the desire to improve their instruction and a lack of defensiveness when we discussed what was working and what was not. We had the benefit of some experienced staff who worked alongside us so the mix seemed perfect.

I have split the post into two parts: Part 1 deals with the mechanisms of the program, and the work of the teachers within it, while Part 2 deals the aspects of what makes our team discussions about the progress of kids more effective.

This is rocket science. I know I have shared this before but it really is not a simple thing to help students decode and encode the English language. Even the start of the process was difficult! We had to consciously learn what movements our mouths make with particular sounds, watch for particular movements in students and help them see those movements in mirrors. To then apply letter formation, pencil grip and posture to the learning of the alphabetic code – it required a significant level of thinking, expertise, planning and patience. Our team needed to spend a long time reading, learning and practising before we go to the classroom. Teaching Prep students to decode is not an easy task.

Sticking to the plan is harder than it looks. The Prep decoding program at my school is incredibly well resourced. Staff are keen and hard-working, meeting time is devoted to discussing and developing instruction, teaching assistants are present in each of the classrooms and we have a plan for learning. Despite all of this, fidelity to our plan was a difficult thing “to achieve on a consistent basis. On reflection there seems to be a number of factors that contribute to going a little “off reservation”. In order for the team to stick to the plan we need to:

  •  Develop a shared understanding, particularly early, of what some of the terms used so frequently actually mean and look like. It is so easy to paint our own mental image of what an activity looks like over the description or name someone else is using to describe their practice.
  • Understand that routine is important and careful thought about when and why we do something different is required. The routines serve the kids well and so we need to maintain them even when we might feel like we are going a bit batty!
  • Develop our understanding around what each activity we use with the kids is meant to achieve and what in the activity helps us achieve that. I wrote about activity substitution last year – it is so easy to start substituting one activity for another because the surface features are the same yet the deep structure may actually be very different. This has a remarkably large impact on student learning.

Teach a variety of children but in more homogenous groups. In our model teachers in the Prep group teach a particular phase of the alphabetic code. For ease of use we adopted the Letters and Sounds Phase model. The students are grouped together at the Phase most appropriate for them. The benefits of this arrangement are threefold: teachers gain a better understanding of the teaching of particular concepts as they repeat the teaching several times with different students; each Prep teacher will, at some point, teach each of the students in the year level, which allows the team to take responsibility for each student’s progress; and students have instruction which is quite narrowly focused on what they need to learn at a particular time.
 

Tracking progress over time is important.
A key plank in our work is the use and discussion of data that relates to our instruction. It is important to meet each week to examine the progress students are making and the difficulties teachers are facing to see if the team can produce a solution. Each term the team tests the students to get a measure of what they know and are able to do in terms of their decoding, including using the UK Phonics Screening Check. This comparative information allows examination of our instruction. 

  • Why is it that, in one class, students pick up particular GPCs faster than in others? 
  • What is different in terms of the instruction in that class compared to the others and how can we share that instruction? 

Also, it gives us a sense of whether our kids are making the progress we want them too – compared to last year’s cohorts, the UK sample, the assessment last term. Particularly with inexperienced staff, the use of a formal assessment to help begin to link instruction to student learning and then discuss differences in instruction to help improve is a key component of a team that learns together.

Part 2 will expand a little more on what we learnt from how to use the information we had to improve our instruction and how the members of the team worked together.

Why ghoti gives me the tiotce

I am usually a reasonably measured person but I have just about reached my limit. Four times in a little over a week I have encountered somebody using ghoti as an argument against using an aspect of systematic phonics instruction. For those unfamiliar with the term often misattributed to George Bernard Shaw, the expression ghoti (pronounced like ‘fish’) is supposed to be a clever wordplay using a number of the grapheme-phoneme correspondences found in the English language. In this case ‘gh’ as /f/ from enough, ‘o’ as /i/ from women and ‘ti’ as /sh/ from notion.

I even read in a recent paper the use of phtheighchound (not entirely tongue-in-cheek) to be pronounced as taken. Take it as a challenge to work out the what words the GPCs in that nonword are taken from.

What gives me the tiotce is that each of these examples involves clear ‘illegal’ applications of the GPCs: ‘gh’ is never /f/ at the beginning of the word and ‘ti’ is never /sh/ at the end. Yet this doesn’t stop the seemingly incessant delivery of this ‘word’ as a key component of an argument against use of a systematic phonics program or using decoding as the primary reading strategy. It is infuriating because the ignorance of such ‘illegalities’ would never be accepted in other areas like mathematics. I have never seen someone argue that, although 1 + 1 = 2, you could rearrange the components of the expression to give 1 1 + = 2 and so that rearrangement somehow invalidates the principles that underpin the equation.

We know that English has a deep orthography and it is not a transparent alphabet but that is not reason enough to ignore that regularities exist and that systematic instruction in phonics is a key component in teaching kids to read.

So, I will not be teaching my kids to ghoti but I will be teaching them to read.

Balkanism – decoding separate from reading comprehension?

A few months ago I wrote a post describing my first encounter with phonics instruction and some of the early lessons I learned as I tried to understand more about its principles and approaches.  In this post I am trying to work my way through a couple of observations that are troubling me a bit; I don’t quite know what they mean.   

In Australia, most schools would indicate that they include phonics in their early years reading programs. This inclusion can range from homeopathic doses through to a program using the understanding of the alphabetic code as its guiding principle. Despite the range of incorporation, phonics as an add-on or back-up seems to be more of the rule than the exception – the AITSL website includes a demonstration video showing a school which uses phonics as a secondary strategy.

 

I understand that decoding doesn’t fully equate to reading and that there also needs to be focus on comprehension. Given the work of work of McGuinness and others indicating that most of the variance in reading comprehension can be explained by decoding and oral comprehension skills, decoding is an integral part of the reading process.  Therefore the development of an understanding of the alphabetical principle should underpin the work we do with early readers.

 

On Saturday I was lucky enough to attend a symposium presented by Learning Difficulties Australia, in which Pr Julian Elliott presented some of his research and thoughts on dyslexia, its diagnosis and the implications of diagnosis in a range of settings. There was disagreement and debate around a number of the opinions presented by the panellists which helped to engage the audience. Even those who were put off by the frank views of the panellists on Reading Recovery and Arrowsmith (and there were a few!) would still have walked away learning more about reading and the factors affecting decoding. As an added bonus I had the opportunity to speak to a number of teachers from various schools across the state about the ways in which they approached early years reading in their school. 

 

In two separate conversations an odd approach was described to me: the schools in which the teachers worked had separate times in the day for the “Phonics Program” and the “Reading Program”. What was gobsmacking for me was that the principles of the two never overlapped. The approach described by the teachers goes a little like this:

 

In the morning kids come in and work on the grapheme-phoneme correspondence of the day, practise encoding using the graphemes they have been working on and read decodable texts. Then, later in the day, they head off to the “Reading Program” where they use “authentic” texts and multi-cueing strategies. In the “reading” session students aren’t expected to use any of the decoding strategies they learnt in the morning session to makes sense of the text they are reading. They wouldn’t point to a grapheme in a book they are listening to and enquire about the associated phoneme. They wouldn’t try to decode a sentence that contained words that the students were able to decode.

 

What has me a little confused is that the teachers talking to me were enthusiastic advocates of phonics approaches. They not only demonstrated that they have more than a passing familiarity with various phonics teaching strategies but they celebrate the progress kids have been making in decoding non-words. They were not averse to phonics nor terrible at their application within the prescribed session. Yet they seemed comfortable with the idea that the elements of the alphabetic principle they taught in one session were not applied at all in a session referred to as “Reading”.

 

Meeting representatives from schools running this way at the symposium may just have been fortuitous but it matched some similar observations I have made when visiting other schools. It makes me think that there is a cluster of schools that have programs that run like this.

 

So now I am left musing. Why is it, when you have an obvious support for the development of the alphabetic principle, that “decoding” and “reading comprehension” gets Balkanised like this? Pushing aside the methods used to teach reading comprehension for a moment, surely it makes sense to embed the use of the alphabetic principle whenever using text. What I am wondering about is: What makes this balkanisation occur? 

I am interested in your thoughts.