Can’t see the language for the errors

Welcome to my latest exam time post

IMG_2467The Idea

It’s exam time again in Verona. Well, to be honest, it is often exam time, but this is one of the big sessions. We had 2,200 enrolments for English on Monday last week for our written tests, which makes for a considerable amount of marking. Our approach to marking written work in sessions like this is threefold: 1) we use a criterion referenced approach to each level which describes in general terms what each area means so that a band 3 mark (and we allow for half marks as well) on C1 writing as far as task achievement is concerned would be awarded to a candidate who is able to produce: “an adequate answer to the question, the meaning is always clear  although some parts may require further development and where the style is generally appropriate for chosen genre.” Our learners do not have the exact grids that we use but they have their own descriptions provided in a more learner friendly way to help them develop the skills they need. Extending ideas, for instance, at C1, which is one of the requirements of the Common European Framework (CEFR) generally needs quite a lot of work as many learners are not used to developing ideas and arguments about particular ideas, and tend to list ideas or summarise their reading. That, however, is a topic for another day. Let’s get back to the examiner’s viewpoint. 2) The second step is to hold examiner meetings at each level to standardize our approach to correction where questions such as what does “adequate” actually mean, and how different is it from “good” etc. This stage is essential if our marks are to be valid. 3) Since we have so many candidates we cannot guarantee double marking of all of them so we choose random samples to double or triple mark. So far so good, you might think. This is a logical, systematic approach to the whole process which is doing its best to be a fair, valid system for the candidates taking the exams, but what is interesting, I think is what happens next. 20110924-194940.jpg Can’t see the language for the errors Once you, as an examiner, have standardized with the others it is just you and the script… or is it? No, actually, it’s you, the script, your everyday concerns, your prejudices and beliefs etc. etc. do you see what I mean? Well, let’s take the classic example or “errors”. Even in the 21st Century there are still those, who despite all the standardization in the world still count errors to come to their conclusions. In a way, it’s natural. What hits you in the face when you read a text is what is wrong with it, but actually, as most of us, in fact, know, there is a lot more going on in that text.

Errors or Mistakes?

The first point to be made here is whether examiners are looking at errors or mistakes; a distinction which is often swept under the carpet as being too technical, but which I actually think is essential. Mistakes are slips or the wrong production of something you really know and they are thought to be related to problems with retrieving information rather than not knowing something. This happens with a slip of the tongue for instance, and is often easy to correct. In fact, people often correct themselves on the spot. Mistakes may not even, then be related to your linguistic competence. I might say “I got to the airport at 8pm” instead of “I got to the station at 8pm” and they happen for all kinds of reasons such as tiredness or stress etc.

An error, on the other hand, is generally linked in our field to your knowledge or lack of it when it comes to language. This may come about because of something that you have not yet studied, so a B1 level student would probably not have studied complex rhetorical structures like “No sooner had we arrived...” If a B1 learner tries to use this and gets the word order wrong it is something that, in my opinion, should be given credit as the learner is trying to use complex language to express their thoughts. It should definitely not be penalized simply because “No sooner I arrived“, for instance is “wrong”. This would be too simplistic. Another occurrence of error is the result of not having internalised grammatical rules yet. This is related to something a learner has studied perhaps or heard, read, but cannot actually produce perfectly so a learner may have seen the expression “It’s worth fighting for freedom of speech“. (It was on the C1 question paper with reference to Charlie Hebdo, so you would hope most people read it, but that learner, then, in the composition writes womething like “Freedom of speech worths to fight.” This is an error but once again the question is should we be giving learners credit for trying to use more complex language or should we be penalizing them for “getting things wrong”? My own personal take on this is that it is a matter of common sense. If a learner is doing a C1 exam and there are so many errors, many of which occur in simple rather than complex language, then that individual is unable to express their thoughts clearly and, to go back to the descriptor “the meaning is not always clear” (although there may even be some discussion of “always” here too and “clear”. This is the problem with words 🙂 ) If, on the other hand the candidate uses generally clear language and is attempting to use more complex forms, but produces the occasional error like the example above with “worth” then that candidate should be given credit. This means that the item, in this case, worth, is now in their interlanguage and presumably, with enough exposure and use, which is crucial, will probably correct itself. The issue of exposure and use is key here too, but that again is another topic. So, when an examiner sits down with a script it is really important, I feel, to differentiate between mistakes, errors and whether they  impede meaning or not. OK, you may well be saying, but what about the language and the trees… oh, sorry I meant “errors”? Well this is my next point. Not only do we need to differentiate between mistakes and errors but we also need to look at what the learners actually can do and not just be swayed by what they can’t. This is the guiding philosophy behind the CEFR and yet how often do we really apply it?

Looking for the Language

Consider this example from a learner script: “In the long run these problems will continue to be common in the society .” What hits you on the nose about this? Yes, there is an article error “in the society”. In fact the use of articles is extremely problematic for learners and often features on error classification systems. I wonder, though, how many people notice the perfectly correct use of the article in the expression “In the long run”. This is what I mean by not seeing the language for the errors. This learner may well have acquired the correct use of the article here as part of the “phrase” in the long run, but very often we are so busy looking at the errors that we simply don’t see the correct usage. It is apparently easier, of course, to assess error and to determine which errors may correspond to, although there may well be disagreement of this too. The third person ‘s’ which is taught at an early stage but often not acquired until much later, is one common example. What, howeer, is much harder is to assess successful language use and to assign a corresponding level to it. Tools such as The Vocabulary Profile might help us, and I have personally foound it to be very useful. It is a corpus informed description of the words and sometimes phrases learners produce at different levels, but even so, this can only be considered to be a guide.

Look at the text not the sentence

Lose yourself in a good book
Lose yourself in a good book
My own final conclusions on this at the moment are that it is as always important to look at the whole text as well as simple examples of language and to assess as fairly as possible the learner’s capacity of expression over the whole stretch of a text both holistically and in more detail. Our descriptors are guidelines but the process should always include discussion, standardization and the awareness that we are not perfect. Finally there is the candidate whose personality shines through these scripts, and we need to remember that each of these answers was written by someone who is doing their best to use the language they have learned to express themselves, but that they are doing so under exam conditions and we can sometimes perhaps allow for this too.

Heaven or Hell

It all makes me think of the parable of Heaven and Hell with long spoons, which I first came across in “Dictation” by Paul Davis and Mario Rinvolucri (CUP 1995 edition p.72) It is actually a parable though and it goes like this: do you know the difference between Heaven and Hell? Well in Hell everyone has long spoons and tries to eat out of the same bowl of soup but cannot get the spoons into their mouths so they starve. In heaven everyone has long spoons and tries to eat out of the same bowl of soup but cannot get the spoons into their mouths so they feed each other. This, I think is a principle we can apply both to teaching and examining. By discussion and helping each other, and giving clear guidelines to teachers and learners we can help to take some of the stress out of the process for everyone. Tests, and all the stress they involve, both for learners and teachers, are part of our world. What we. As examoners can do is to help learners by giving credit where it is due.

Here is video of the parable produced by Caritas