The art of gathering… even for exams

Exam time is here… again

Exams are on my mind at the moment. Exams are often on my mind, in fact but this period in the summer is one which is particularly busy for us here at the University of Verona. We tend to focus on content, tasks, levels, intake, grades and so on when we talk about assessment but today I wanted to think about the exam as a “gathering”.

Exams are part of our psychology

The idea of exams is deep-rooted in our society. As children we listen to fairy tales where princes are given tasks so that their worth, integrity and valour can be tested. Passing a test unlocks power and enables experts to practise their expertise. The idea of being wise, skillful and knowledgeable is one that we all buy in to. After all, the argument goes, we want to know that the doctors who are treating us or the pilots who are flying the planes we travel in, know what they are doing.

The question, however, remains of how suitable the test they are taking is. If a prince can slay a dragon, does that necessarily mean that he is a suitable match for the princess? Probably, yes, because if he can slay dragons then he can protect a kingdom too, and if that is the purpose of the test, then all well and good. It does not, however, mean that he will automatically be a good husband, because that is not what was being tested. The idea that the test should be difficult is also deeply ingrained, it should sort the sheep from the goats so that we can recognise the very best… or should it? In our present educational system we are moulded by humanistic thinking that puts the learner at the centre of the process. If, when teaching, we do this, fostering the best in each learner, then surely our exams should do the same, enabling the candidates to do their own, personal very best.

Competition

https://hartlelearning.files.wordpress.com/2011/08/inspiration-and-poetry.jpg”> Making the most of your own inner strengths: each test a world to discover[/caption

This, however, brings me to my second point which is that competition is also part of our world. This is reflected, for instance, in the hundreds of game shows that abound on television, on the Internet, on the sports field and just about everywhere else. When I was at school there was talk of healthy competition and how it brought out the best in individuals. Well, it does in some individuals but not everyone. In a learner centred system the problem with competition is that  there can only be few “winners” and the “losers” are doomed to think of themselves as such, which can condition them throughout their lives. If our system is truly learner centred, and that is a big “if”, then competition should be personal the competition you take part in with yourself to be the best you can be.

How not to organise an exam

The psychological importance of the way exams are organised is a key factor in how successful candidates are or are not in a particular exam. and it is an aspect that I think we may tend to underestimate. Part of this is the idea of the gathering as being a coming together of people, and how this “gathering” should be organised. We tend to take it for granted that those who take the exam can cope with it, but, speaking as one who has taken quite a few exams myself, the organisation of how candidates are gathered can actually be nerve-wracking for the candidates, even though I am quite sure that this is often completely unintentional.

I remember one exam I took, for instance, where we had taken the written part in the morning and those who, hearts beating frantically, rushed to the noticeboard to see that they were among the few who had passed, were then told to come back at the beginning of the afternoon for the oral test. I met a friend, had lunch and was, actually, quite relaxed by the time two o’clock came round, but little did I know that this was when the true ordeal was about to start.
We were all waiting together in a clinically grey corridor on plastic chairs that showed how much our comfort meant to those who had invested in them. Some people, who had taken the exam several times before , lost no time telling everyone how gruelling the experience was and then the examiners, three of them, arrived looking harassed, probably rushing there from a previous job or engagement that was still very much on their minds.

The afternoon wore on…
Each candidate was called in, took the oral exam, and was then sent back to the “waiting area” until the examiners reached a decision. Time after time, someone came out of that room, and in full view of everyone else said something like. “I’m very sorry you haven’t passed this time, we hope to see you again soon.” Can you imagine the effect this was having on all the other candidates? Some passed but a lot of people did not. I was one of the last to be called. I had spent several hours in psychological anguish waiting for my turn and was now so depressed by the whole thing that I said to myself:
“You are an adult human being”
“You can speak this language well”
“The examiners are human beings too”
“The only thing to do is to put the interview on an equal footing by smiling at them and talking to them, making jokes when appropriate and asking them questions too.”
“What have you got to lose?”
My strategy worked because I was one of the lucky few who passed, but I had a lot of experience and was able to talk myself into the appropriate frame of mind to be able to do the exam. How true was this of many of the others who were younger, less experienced and perhaps less confident than me? What, in fact, was the true test here: a language competence test or a psychological fitness test?

My point, however, is that none of this was intentional. The examiners were obviously unaware of the effect the way the people had been gathered was having and yet it would have taken very little to change things.

The power of the gathering

I have recently been reading an interesting book on this subject called “The Art of Gathering” by Priya Parker: https://www.penguinrandomhouse.com/books/319055/the-art-of-gathering-by-priya-parker/9781594634925/
She discusses many aspects of what is key in a gathering ranging from having a clear purpose to focusing on the participants in a gathering rather than the content, flowers, food etc. The purpose ostensibly in this case was to conduct an exam, presumably to assess language competence enabling the candidates to do their best. Small changes would have been helpful such as:

1) time slots for people to come for the exam instead of the whole afternoon;
2) a comfortable waiting area which is separate from the exam taking area;
3) an other waiting area for those who have already taken the exam, so that they can leave without being seen by the others;
4) something for test takers to do to lesson the anxiety of waiting ( a drinks machine, reading materials, information about the exam centre etc.)

These are just a few ideas off the top of my head, but there could be many more.

My exam last week

So, how would you do it? I can hear you all asking and I don’t pretend to have all the answers but I do try to focus on my students and their needs. Last week my third year university students did their final written test. There were about fifty of them in a university lecture theatre, which can be off putting. Some of them arrived about half an hour before the exam clutching their dictionaries with worried expressions on their faces. I asked one or two of them to do practical things like switching on lights or closing windows so that the air conditioning would work… just to calm them down a bit. When quite a few people had arrived I told them I’d be back in ten minutes and that in the meantime they could chat to each other in English to psyche themselves for the exam. (I organised them into small groups of four or five people and left them to it). When I came back the atmosphere was lighter, there was quite a bit of chatter and more people had arrived.

At this point, I could have given them a lot of procedural instructions but I decided not to. I handed out the papers,
Then I said something like:
“I know some of you are feeling anxious. This is an exam and it goes with the territory but please remember to breathe… Breathe, use the exam to show me how much you know and enjoy. Off you go!”
The atmosphere changed and they started work. I then went round to check the things I had to check with each student and wrote the other information they needed to know on the board. Half way through the exam I interrupted them for a minute, told them that we were half way through and could they please just read this important information.
They then went back to work and at the end handed in their papers and left.

Yes, I know that I could probably have done more, and despite my best efforts I heard some of them in the corridor terrifying each other with a post mortem of the exam…. Habits die hard.

Gathering starts well in advance of the event

018/07/helena-lopes-592971-unsplash.jpg”>Helena Lopes[/caption]I do think, though, that I did my best to

I do think, though, that I did my best to make the gathering conducive to helping them produce their best work. This, in fact, started well before the event of the exam itself. All those students knew exactly how the exam was organised, what sort of questions and tasks there would be and how it would be marked. They had been working on practice tests to prepare (at least most of them had) and they could contact me if they had doubts, which some did.

My students have a closed Facebook group and as the moment of the exams approaches I post lighthearted videos that are relevant as well as other questions or discussions that they can take part in, in an informal way. This pre-exam interaction is all part of it, I think, and helps to build a community spirit which then helps them feel that they belong on the day of the exam itself. The purpose of my exam is for students to show me that they can not only remember the content of the course but think critically about it and express that. I am not interested in perfection or perfectly memorised answers to questions. I see my course as whetting my students’ interest in certain areas of linguistics, translation studies and corpora. I do not expect them to be experts after one course. They know this too, and what it means is that, although it was clear that some people had not really prepared at all, they were in the minority and the majority , in fact, did well.

And finally…

I’m not saying that it is easy to gauge the gathering techniques needed in an event like an exam, but I do find it worrying, as I said at the beginning, that many exam organisers do not take these factors into consideration at all, or are unaware of how important they are. I’d be very interested to hear your experiences too. 🙂

What’s the point?

What’s the point of teaching, testing, teacher training and conferences?

At the risk of sounding as though I’m copying Barry O’Sullivan, I may say that, at the beginning of this post, you are reading but you don’t know where this is taking you, and I’m not sure I do too, but I hope that it will all come together along the way.

The image I chose for this post is one of cycling along on a fixed course whilst everything blurs around you, and I sometimes wonder if this is what I’m doing. During every big conference, like Iatefl, there are moments of confusion, where the input from so many presentations becomes just a bit too much and these are then followed by moments where questions arise. At some stage, later on, usually after the event there come moments of clarity as well, where everything seems to crystalize (no pun intended!) into various themes or topics or ideas, and that is the answer, I think to my first question: what is the point of conferences? Conferences, of course, as we all know, are much more than presentations, they are, as a friend of mine put it, a time when you ‘hang out with your tribe’, something we all need to do. Then, however, we go back home and everything is swirling around in the background as you try to catch up on sleep and the hundreds of emails waiting for you, that you couldn’t answer due to the slow Internet in the hotel etc. Then, slowly, ideas and thoughts emerge from the surrounding blurriness and occasionally there are moments of insight. My insight is connected to the cyclist in the image above.

Cycling along without being aware of the bigger picture around you

I suppose most of us do this, to some extent, we go into our classrooms or our offices, and we do the work that is infront of us, to the best of our ability but how often do we ask ‘Where is all this leading?’ I am not referring to the paycheck or making ends meet. Most educators, I still believe, are not ‘in it for the money’ but do have some type of, if not vocation, belief in what we are doing. Barry O’Sullivan, in his incredibly thought-provoking plenary, managed to convince me that at best we are muddling along and that our assessment system is actually pretty much a case of trial and error, no matter, how scientifically we dress it up. Reliability is a term that is used a lot in testing, and actually, as he said, it actually only means ‘consistent’ not whether the test is consistently good or consistently ‘crap’ as he put it. We buy into the scientific principle that bigger is better so that if test results are reliable over large numbers than that must be a good thing, mustn’t it? Well, actually, possibly not. We don’t actually know. We only know that the numbers or the results are reliable, or consistent, not that the test is good or even valid, in that it tests what our particular learners need, and here I come to the second point from Barry’s presentation that I’d like to stop and think about. The ideas that testing, teaching, teacher training all need to be ‘local’ and not that they need to be developed locally, necessarily, but that they need to reflect local needs. Local, in this case meaning relevant to individuals, to you and to me and to each of our learners. This takes learner centredness to a whole new level, as Barry called for everyone to work together: trainers, teachers, testers etc. informed by standards with a capital ‘S’. If you’d like to see the plenary, where he says so much more and which, as my friend Sian said, was ‘a cracking plenary’, follow this link to the British Council online coverage.

Barry, in fact, encourages us to ‘create the future’. This really means stopping your bicycle, getting off, looking around to see the bigger picture and to start smelling the roses.

What’s the point of my teaching?

So, I went for a walk in the countryside near Verona where I live. OK, so I wasn’t really smelling the roses, but I was admiring the arrival of Spring, and a doubt that had been at the back of my mind for a while, surfaced once again. To put it bluntly, I was wondering what the point of some of my courses is. For instance, in my specific university course for third year undergraduates, I am spending  36 academic hours teaching language students to go someway towards being able to analyse discourse, something that has taken me years, in fact, to even approach being able to do, on a good day, and then, just for good measure, we are also doing an introduction to corpus linguistics and to translation theory and practice. This is all fascinating stuff, that I am personally quite passionate about but I am really feeling the time constraints, and I also feel that it is unfair to expect the students to be able to gain much expertise in these fields, so that is why I was questioning the point of it all, but, and there is a ‘but’, as Barry O’Sullivan said, we can ‘create the future’ so the point is actually, in my mind, to whet my learners’ appetite for these things, to teach them the value of critical thinking and not taking what they read, hear, watch or translate at face value. Hopefully, I can communicate my own enthusiasm to my learners and some of them will then choose to go more deeply into these things. So, my moment of lucidity is this: creating the future means whetting our learners’ appetites, catering for their and our own needs and putting what we do into a bigger context. As I walked with the dogs around the fields, admiring the blossoms and the wild flowers in the grass I actually felt a moment of peace. So, that you Iatefl and thank you Barry, for helping me once again to come back from the conference and clarify a whole range of ideas.

Getting back to normal

So, I hope that this little post has actually taken us somewhere, and tomorrow, when I go into clas, I think I will share the idea of ‘whetting appetites with my learners as I allay their fears about my expectations of them in their final exams. After all I want to be able to assess what they can do, and what will help them in their English, their lives and for their futures once they leave the university.

 

Welcome to Brighton 2018

Welcome to Brighton 2018, come and join the fun!

The annual IATEFL conference starts officially tomorrow, Tuesday, but quite a few people were already here this morning, braving the blustery early morning rain to sign up for the Pre Conference (PCE) day. This is a day when each of the special interest groups (SIGs) organises a complete day dedicated to an interesting aspect of its ‘area of expertise’. I belong to the Tea SIG,, which does not mean that we sit around drinking cups of tea all day (although there was some of that as well) but Tea stands for ‘Testing, evaluation and assessment’. This year was dedicated to testing listening and the day consisted of very thought-provoking talks in the morning followed by lunch and then a practical focus in the afternoon as we mapped a listening activity and then developed items on it. These were then given a mini trial by another group who provided feedback. This way of structuring the day worked well, as everyone was fresh in the morning and receptive to the input provided in the traditional presentation format, but by the afternoon, I think, most people were happy to be doing something more practical. The idea was that everyone should go away, having learned something that both made them think and was practical too.

Authenticity or Reliability… Hey, what about both?

John Field began the day by calling for listening tests to be really designed in such a way as might realistically test what listeners at different levels can be expected to do. He underlined the need for ‘cognitive validity’ or rather asked: does the behaviour elicited from test takers correspond to the requirements of listening in real world contexts. His model of the way listening structures follows various stages, although they are not necessarily linear: decoding the sounds comes first, followed by word searches, recognising the boundaries between one word and another. Then comes parsing where different elements are recognised and labelled to some extent followed by the construction of meaning and finally the construction of discourse. In a nutshell, which I probably shouldn’t say, but anyway, what this means is that at lower levels the focus should be testing at word level and higher that this discourse meaning can be tested. He stressed that ‘knowledge is not recognition’ and if we are testing higher levels, we should be careful not to be testing complex cognitive processes which go beyond listening.  John said a lot more, and I have barely done his fascinating talk any justice at all, but it was a great start to the day and he left us with this thought: the perceptual prominence of any word or clause is central to a correct response to the item. If something is not stressed in some way in a listening text, then, it is not realistic to expect it to be identified and unfair to create an item around it.

Sheila Thorn, then, took over and talked about authentic listening. She has long battled for this taking on examination boards and doubters of all kinds. Her basic intuition is that so many people study a language and then go to the country where it is spoken and flounder around in the dark, understanding very little. Something is definitely wrong here. She suggests that rather than simplifying texts at lower levels we should be providing them with longer texts but testing them on the content which is comprehensible, so that they will be exposed to authentic listening bu tested on content that they can understand. She also stressed how unrealistic the idea of doing multiple choice tests whilst listening would be ‘in real life’. You don’t listen to a podcast and answer multiple choice questions, after all. She suggested that the tasks should be more natural and connected to summarising skills, which is similar to what we might do normally when listening.

Yes, but what about the Stats?

Rita Green talked about the need to collect statistical evidence when developing tests, which means trialing items, ‘playing the detective’ as you evaluate the data you collect and interpret it and only then can you actually bank those items if they correspond to your requirements, so that if, for instance, an item on a test proves to have distractors that are far too easy, or if very few people answer one question, these need to be looked at and either revised or dropped. She described classic test theory which measures mostly test taking populations and the tests themselves but she added that modern test theory takes things a step further actually examining individual test takers, for instance, and the degree of error associated with every item and every test taker. Modern test theory also looks at Fit statistics, which does not mean how fast the test taker can run away from the examiner but whether items or individuals perform in predictable or unpredictable ways. Care of course must be taken with how the data is collected and interpreted but Rita concluded by saying that ‘without field trials and data analysis we are working blind: the more valid the test, the more reliable the test scores.’

The Afternoon Session

After lunch attention tends to flag somewhat so this was the perfect moment to do something practical. Under the expert guidance of the afternoon moderator team: Thom Kiddle, Felicity O’Dell, Russell Whitehead and Russell Whitehead, we embarked on a voyage of discovery through the process of item writing. This involved firstly listening to a text and mapping it for gist, key points etc. and then comparing our results in small groups. We then wrote items for that text (our groups were assigned multiple choice). We began by deciding on the context, the learners, age, interests, needs etc. and whether we would allow them to watch the video or not. We decided not to as a text appeared in the middle, which provided the gist of the news story, s we would not have been testing listening. We then swapped items and trialed the ones produced by another group and added our constructive feedback. The items were then returned to the original writers. This was a perfect way to work in the afternoon, and whilst time was short, it gave a glimpse of what it means to be an item writer which, I think, was extremely valuable for all involved.

A few thoughts

All this gave us a lot to think about and the discussion with the panel later was interesting and quite lively at times. The question of test purpose was broached as were other issued such as Global English or test context, and test taker aims and needs. The thorny issue of whether or not to opt for multiple choice also came up and the answer was that although they may not be natural they are practical. Practicality was another key issue which jars somewhat with the notion of authenticity. To mark authentic tasks such as summarising requires a lot more ‘rater power’ than multiple choice questions. I, personally, do not feel that multiple choice items are ‘evil’, but they should be one of more options and the needs of the test takers must be central. It is useless for PhD students who have to write long articles or theses to take tests that only require 250 word essays. I know this is not listening but it is just as true of listening. the Ielts listening exam is not ‘academic’, even for those doing the academic version. For students who intend to do MAs etc. surely it makes more sense to test them on their abilities to listen to lectures. Apparently, Ielts, however, will soon be revised, so one step at a time, we are perhaps moving in new directions.

All the fun of the fair

The fun?

If this all sounds quite serious to you and you’re wondering about the fun element, I have to say that serious things can be fun but a large element of this conference is the social side of things. Friends meet up at the conference and exchange their news and experiences, and new friends are always made here. This evening was the first event which was a welcome reception culminating in dancing, so things definitely got off to a good start.

Hope to see you around the conference. 🙂

What are you ‘sinking’ about? The ELF Debate continues…

The Elf Debate

The elf debate is still at the back of my mind, as it often is and the other day, here at Glasgow Iatefl, Peter Medgyes, in a very well presented speech which supported the importance of learning English as a language in its own right, rather than learning a not very well defined ‘elf version’, quoted this video as an example of one reason why in real life situations we need to be able to speak ‘proper’ English.

There is so much wrong with the stereotyping of this ad, which is actually a Berlitz advert, that I’m not sure where to start, both as far as language learning is concerned and as far as stereotying the Germans… However, my point here is not the stereotyping in itself but how relevant this is to the question of ELF. The point being made here seems to be very much in favour of traditional English models although who, in their right mind, in a context such as this, would react in this way??

I, personally, keep thinking that there is a distinction to be made between ELF as the traditional researchers such as Jennifer Jenkins or Barbara Seidelhofer, and others, see it and Global English as described by david Crystal as the usage of English as a lingua franca on a global scale. There is no denying that English is a global language, and this means that it is in rather a different position from other languages perhaps that are studied with the express purpose of contributing to or integrating into L1 communities. This means, in my view, and as I have said before, that when it comes to assessment we need to take into consideration the notion that our learners need to aim for clear expression rather than to adhere to unreachable native speaker norms, which has to be taken into account in assessment. When it comes to teaching, however, there still needs to be a clear model to present in the classroom, and this is the closest native speaker variety to those learners, so that in Europe this will probably still be British English to a great extent. After all, I may, in a test situation, decide that using ‘informations’ as a countable form rather than the traditional, uncountable ‘information’ does not impede the message particularly (although it will affect the grammar and text references that go with it when writing, which may well hamper reader comprehension). So, when testing this may be acceptable but when teaching we are sulely doing our learners a disservice if we do not point out that even though many now use this word in a countable way it is, actually, uncountable. The model that is presented, in fact, is often just that: a model, and then each individual will, as they do in their own language, develop their own voice and means of expression. As Peter Medgyes also said in his presentation, this is actually not ELF but simply the way we use language.

What do you think?

Iatefl Teasig Webinar branches out to do Facebook Discussions

What are the Iatefl Teasig Webinars?

If you look at the image on the left you may be forgiven for thinking that the Iatefl Teasig (Testing, Evaluation and Assessment Special Interest Group) is a bit of an informal affair, and actually you would be partly right. Testing, evaluation and assessment is often thought of as being rather dry or difficult to deal with, so why not bring a warm association of a “nice cup of tea” into the picture.? In our webinars Neil Bullock and Judith Mader, the coordinators of Teasig, with a little help from me, have tried to keep an informal but informative style, reaching out to members of the sig but also others who are interested in testing and who might, in the future, become members of our sig too. The aim of the webinars is to invite interesting speakers who have something to say about testing, evaluation or assessment, to discuss their topic in a one-hour webinar. These are held regularly every few months (For more information follow this link to the Teasig site) on Adobe Connect, and are generally well received. We have been fortunate so far in having had some excellent speakers who really reach out, embracing the medium of the webinar and include the audience in their discussions. The discussions, however, tend inevitably to be “top-down” in the traditional sense. The speaker presents his or her ideas and the audience listens, comments in the chat box and asks questions. Speakers then answer some of the questions at the end of the session, or if there is not much time they answer them in a feature in the Teasig newsletter.

This has been successful so far, but we have now decided to take the process one step further to allow for greater exploration, discussion and sharing of resources by the participants. How are we doing this?

Why use Facebook for webinar discussions?

In the C21 we actually have the chance to question things like discussions and use social media to do this in interesting new ways. In the past conferences and seminars have often been about listening to experts and asking questions, learning something new and then going home. Now we have the chance to take the discussion further to reflect and share our insights with each other drawing on the largely untapped resource of audience experience and insight.  Instead of just “going home” or rather switching off the computer and heading towards a bar for a Prosecco (in my case) this week we extended the discussion of  ‘Assessing and Marking Writing” by Clare Fielder to take things further on Facebook. Why use Facebook?

Well, Facebook is a space that many of us know and use all the time, which means that like a familiar room, we can meet there to discuss the ideas that have come up, just as we might do in a café, for instance. Being “somewhere” that we already know makes people feel comfortable and willing to post their own ideas and comments in a freer way than they might do in the actual webinar chat feed. An added adva

A Space for Reflection

ntage of extending our event in this way, is that although the actually discussion itself was synchronous with me moderating it, the posts actually stay online so that all those interested in the event can go back to see them. In fact, some comments were added after the event itself, which means that a whole new asynchronous exchange starts to develop. One person, for example, Aimee Johansen, watched the recording (avaiable after the event itself)  and then commented on the Facebook Events page, that whe had found it interested and it had reminded her of some things and introduced her to other feedback methods that she would not have thought of but would like to try out. I then asked her what she would like to try in particular, so the discussion continues even a few days after the actual event.  Kent’s research about Facebook use in class discussions shows clearly that students, for instance, are happier to post on facebook than on official course discussion boards, and even though our discussions are professional and not part of a course I believe the same principle applies. As in real life there are those who like to post and others who like to follow the discussion “silently”. Whichever way you choose to use the discussion is up to you, and catering for different needs is all part of the show. For all these reasons, and particularly because Facebook is so well known, then, and many are happy using it, this was what we opted for. This was our first experience, it went well and I hope it will get even better in the future.

What happened in the Facebook Discussion?

We used the Teasig Facebook Page, which has been set up and managed by Ceyda Mutlu. Ceyda had already set up an event to advertise the webinar, as she always does and Participants were directed to this page at the end of the webinar. Some people, in fact had already accepted the invitation to attend the webinar and had posted questions and comments in advance. This meant that the discussion was already underway, in fact, before the webinar had even started!

On the evening of the webinar particpants were directed to the “Facebook Event” at the end of the webinar, and I posted the questions that had come up during the event here. Clare had been speaking about using Correction Codes to provide feedback to learners on their writing and there was a whole range of questions. I myself had quite a few including a question about how to include this kind of feedback in courses where time constraints are already an issue. Clare had outlined some of the disadvantages such as learner participation, which often comes about because learners receive a piece of written work corrected with a code that they do not understand. Time, then, must be devoted to familiarising learners both with the process and the code. I’m a big believer in learner centred teaching and developing online dialgogues with my learners, possibly becuase I tend to have very large classes, so here is a post I wrote after the 2015 Iatefl Conference which touches on developing asynchronous dialgues with learners to provide feedback and growth, so I wanted to know what Clare thought about integrating all these things into a teaching system.

We all discussed these and other ideas and shared resources and screenshots to explain what we meant, etc. This was our first “live” discussion, but which I mean that there was a moderator and participants knew that we were all “there” at that particular time, and I’m sure that things will only get better with practice, but as a first attempt it went well, so if you’r einterested go along to the discussion and have a look :-).

 

Can’t see the language for the errors

Welcome to my latest exam time post

The Idea

It’s exam time again in Verona. Well, to be honest, it is often exam time, but this is one of the big sessions. We had 2,200 enrolments for English on Monday last week for our written tests, which makes for a considerable amount of marking. Our approach to marking written work in sessions like this is threefold: 1) we use a criterion referenced approach to each level which describes in general terms what each area means so that a band 3 mark (and we allow for half marks as well) on C1 writing as far as task achievement is concerned would be awarded to a candidate who is able to produce: “an adequate answer to the question, the meaning is always clear  although some parts may require further development and where the style is generally appropriate for chosen genre.” Our learners do not have the exact grids that we use but they have their own descriptions provided in a more learner friendly way to help them develop the skills they need. Extending ideas, for instance, at C1, which is one of the requirements of the Common European Framework (CEFR) generally needs quite a lot of work as many learners are not used to developing ideas and arguments about particular ideas, and tend to list ideas or summarise their reading. That, however, is a topic for another day. Let’s get back to the examiner’s viewpoint. 2) The second step is to hold examiner meetings at each level to standardize our approach to correction where questions such as what does “adequate” actually mean, and how different is it from “good” etc. This stage is essential if our marks are to be valid. 3) Since we have so many candidates we cannot guarantee double marking of all of them so we choose random samples to double or triple mark. So far so good, you might think. This is a logical, systematic approach to the whole process which is doing its best to be a fair, valid system for the candidates taking the exams, but what is interesting, I think is what happens next. Can’t see the language for the errors Once you, as an examiner, have standardized with the others it is just you and the script… or is it? No, actually, it’s you, the script, your everyday concerns, your prejudices and beliefs etc. etc. do you see what I mean? Well, let’s take the classic example or “errors”. Even in the 21st Century there are still those, who despite all the standardization in the world still count errors to come to their conclusions. In a way, it’s natural. What hits you in the face when you read a text is what is wrong with it, but actually, as most of us, in fact, know, there is a lot more going on in that text.

Errors or Mistakes?

The first point to be made here is whether examiners are looking at errors or mistakes; a distinction which is often swept under the carpet as being too technical, but which I actually think is essential. Mistakes are slips or the wrong production of something you really know and they are thought to be related to problems with retrieving information rather than not knowing something. This happens with a slip of the tongue for instance, and is often easy to correct. In fact, people often correct themselves on the spot. Mistakes may not even, then be related to your linguistic competence. I might say “I got to the airport at 8pm” instead of “I got to the station at 8pm” and they happen for all kinds of reasons such as tiredness or stress etc.

An error, on the other hand, is generally linked in our field to your knowledge or lack of it when it comes to language. This may come about because of something that you have not yet studied, so a B1 level student would probably not have studied complex rhetorical structures like “No sooner had we arrived...” If a B1 learner tries to use this and gets the word order wrong it is something that, in my opinion, should be given credit as the learner is trying to use complex language to express their thoughts. It should definitely not be penalized simply because “No sooner I arrived“, for instance is “wrong”. This would be too simplistic. Another occurrence of error is the result of not having internalised grammatical rules yet. This is related to something a learner has studied perhaps or heard, read, but cannot actually produce perfectly so a learner may have seen the expression “It’s worth fighting for freedom of speech“. (It was on the C1 question paper with reference to Charlie Hebdo, so you would hope most people read it, but that learner, then, in the composition writes womething like “Freedom of speech worths to fight.” This is an error but once again the question is should we be giving learners credit for trying to use more complex language or should we be penalizing them for “getting things wrong”? My own personal take on this is that it is a matter of common sense. If a learner is doing a C1 exam and there are so many errors, many of which occur in simple rather than complex language, then that individual is unable to express their thoughts clearly and, to go back to the descriptor “the meaning is not always clear” (although there may even be some discussion of “always” here too and “clear”. This is the problem with words 🙂 ) If, on the other hand the candidate uses generally clear language and is attempting to use more complex forms, but produces the occasional error like the example above with “worth” then that candidate should be given credit. This means that the item, in this case, worth, is now in their interlanguage and presumably, with enough exposure and use, which is crucial, will probably correct itself. The issue of exposure and use is key here too, but that again is another topic. So, when an examiner sits down with a script it is really important, I feel, to differentiate between mistakes, errors and whether they  impede meaning or not. OK, you may well be saying, but what about the language and the trees… oh, sorry I meant “errors”? Well this is my next point. Not only do we need to differentiate between mistakes and errors but we also need to look at what the learners actually can do and not just be swayed by what they can’t. This is the guiding philosophy behind the CEFR and yet how often do we really apply it?

Looking for the Language

Consider this example from a learner script: “In the long run these problems will continue to be common in the society .” What hits you on the nose about this? Yes, there is an article error “in the society”. In fact the use of articles is extremely problematic for learners and often features on error classification systems. I wonder, though, how many people notice the perfectly correct use of the article in the expression “In the long run”. This is what I mean by not seeing the language for the errors. This learner may well have acquired the correct use of the article here as part of the “phrase” in the long run, but very often we are so busy looking at the errors that we simply don’t see the correct usage. It is apparently easier, of course, to assess error and to determine which errors may correspond to, although there may well be disagreement of this too. The third person ‘s’ which is taught at an early stage but often not acquired until much later, is one common example. What, howeer, is much harder is to assess successful language use and to assign a corresponding level to it. Tools such as The Vocabulary Profile might help us, and I have personally foound it to be very useful. It is a corpus informed description of the words and sometimes phrases learners produce at different levels, but even so, this can only be considered to be a guide.

Look at the text not the sentence

Lose yourself in a good book

My own final conclusions on this at the moment are that it is as always important to look at the whole text as well as simple examples of language and to assess as fairly as possible the learner’s capacity of expression over the whole stretch of a text both holistically and in more detail. Our descriptors are guidelines but the process should always include discussion, standardization and the awareness that we are not perfect. Finally there is the candidate whose personality shines through these scripts, and we need to remember that each of these answers was written by someone who is doing their best to use the language they have learned to express themselves, but that they are doing so under exam conditions and we can sometimes perhaps allow for this too.

Heaven or Hell

It all makes me think of the parable of Heaven and Hell with long spoons, which I first came across in “Dictation” by Paul Davis and Mario Rinvolucri (CUP 1995 edition p.72) It is actually a parable though and it goes like this: do you know the difference between Heaven and Hell? Well in Hell everyone has long spoons and tries to eat out of the same bowl of soup but cannot get the spoons into their mouths so they starve. In heaven everyone has long spoons and tries to eat out of the same bowl of soup but cannot get the spoons into their mouths so they feed each other. This, I think is a principle we can apply both to teaching and examining. By discussion and helping each other, and giving clear guidelines to teachers and learners we can help to take some of the stress out of the process for everyone. Tests, and all the stress they involve, both for learners and teachers, are part of our world. What we. As examoners can do is to help learners by giving credit where it is due.

Here is video of the parable produced by Caritas

An Energy Dip: approaching high demand teaching (Part One)

After all the excitement of Iatefl last week, I hit reality back in the classroom this week, and what with all the to do about Sugata Mitra’s ideas and the need to make up a whole series of lessons I suddenly found myself in a very dark professional place. I am teaching a group of learners, or rather, trying to help them, prepare for the Cambridge Advanced Exam in June this year. However, do we have an academic year to prepare for this? No, we have 7 lessons, and now, due to Easter and various national holidays we are about to have a three week break. This, as is understandable, has caused a certain amount of panic both for the students and for me. Far from following high demand teaching, something I truly believe in, we have been focusing on exam strategies and they have been doing practice tests at home, independently.

Learning or Devouring exercises?

Focusing on exam strategies! Good idea, I hear someone say. Well, yes, that is what I thought too, until I realised that what they are doing is trying to do as many tests as possible, swallowing entire volumes of tests, emptying the library and then coming back for more. They are definitely motivated, but my energy dip and depression came from the realisation that this is all very shallow, and that I really have no idea if they are actually learning anything at all. I had just finished gong painstakingly through two exercises designed to make them aware of the need not just to focus on single words but to focus on words and the other words they collocate with. It was an exercise where they had to look at sentences, which were examples of candidate errors in the Use of English exam, and decide where the preposition errors were. I went over the activity and they got the answers right with very little trouble.

I knew I was in trouble, though, when I saw someone checking “something” could it have been a train timetable? on her phone. This activity was so far away from engaging these learners that they may as well have been on the Moon.

Time to stop and Reflect

So, what did I do? Well, I said, “OK”… and they were already focusing on the next exercise, when I said “No, let’s think about this.” The smartphone was put aside for a minute. I said. “Yes, you can identify the prepositions very well but can you use these expressions? Will you remember any of them five minutes after you have walked out of this door?” This was said humorously and so greeted by  nervous laughter but I then took pity on them, smiled, and said “So how can you remember new language?” This was familiar territory and someone, with a rsigned expression said “write sentences” and I replied that yes, they could do that, but today we were going to do something different.

Mechanical study or “Meaningful” study?

Stage 1: Cognitive Study

I then asked them in two groups to take one exercise each and paraphrase the phrasal verbs and expressions( these were things like I congratulated him on getting a new job. Absolutely not memorable for these students or anyone else for that matter) I asked them to think about the context the expressions were being used in, and monitored. It soon became clear that there were several problem areas. What could you set up, for instance? Could you only set up a business or could you set up a committee as well? This led us naturally into dictionary work, and they found out how to use these expressions in a lot more depth. They then mixed the two groups and shared their findings, and by this time the questions were coming thick and fast. The smartphone was being used for the dictionary app and the train timetable was a thing of the past.

Stage Two: personalisation and experimentation: making the language your own

Then I asked them to choose five of the expressions and to make them into questions to interview someone in the other group. This was where the meaningful action came as they asked “real questions” and created content that neither I nor the traditional exercises could have predicted. “to congratulate someone on something” was transformed into “Has anyone ever congratulated you on passing your exams? Much more meaningful for these students as was the reply: “Yes, my parents have… when I deserved it.”

Before long we were involved in a discussion about Starbucks and why the concept probably wouldn’t work in Italy. This was communication and, I think, it was definitely demand high teaching.

Reflection again

At the end I asked them why we had spent 30 minutes of our precious lesson on two exercises. They answered that it had helped them to really understand this language. At that point I felt that the energy dip had passed and I was back on track. I agreed with them and warned against mechanical exam practice without going into the language in depth and sent them of with a series of things to do over the three week holiday including the idea that they should “Make the exercises they do really work for them”.

I’m quite excited to see what they bring back with them 🙂

 

 

 

Teaching mindfully and Facing exams courageously

I’m Back

I suddenly realised that I haven’t blogged on my poor, neglected space since September!! I can only hang my head in shame and mutter excuses about how tied up I’ve been with conference sin Prague, the student blog and its sister Facebook page,. etc. Please feel free to check them out and I’d love you to participate in some of our discussions if you feel like it. 🙂

Mindful Teaching

Well, I actually wanted to talk about mindful teaching today, which I am trying to put into practice. What this means is basically that I slow down, stop trying to do three million things at the same time in the classroom, worrying about whether we are doing too much, too little, whether it is appropriate or not for my students, what we should go onto next, oh yes, and where did I put that DVD?… You get the general idea? Instead of doing all that what I want to do is to slow down, breathe and notice what is going on in the classroom, notice how I am feeling, what people are saying and how they are sitting, moving and, (tricky this one) feeling. This may seem obvious but I felt the need to remind myself this week. What I noticed as I walked into my C1 university lesson in general English was this:

1) some people looked tired and had various types of colds and snuffles;

2) there was a general sense of agitation in the room and people were whispeirng together;

3) some people had very large wads of paper with reams of photocopies of Powerpoint slides on them.

You didn’t need to be a detective to realise that exams were approaching and everyone was worried about this. Obviously, if you have an important exam to do you concentrate on that rather than on your other classes, so I did the rest of the class in an understanding way, and if people had not done their preparation I made it clear that I realised why. This in itself had a positive effect because everyone relaxed and appreciated their teacher trying to relate to them. (I think)

Accepting Fear and Vulnerability

After this class, though, I went on to think about how students (or anyone else for that matter) can deal with these feelings of fear and nervousness, which can be so strong that they actually stop you concentrating; the exact opposite of what you want.

At some level, of course, this is tied up with the need to be perfect. We live in a world that frowns on mistakes and where we apportion blame rather than trying to learn from our mistakes helping each other. Of course we are frightened of exams, where someone sits in judgement on us, and our performance in twenty minutes or so becomes our very being.

To move beyond this is quite a challenge, but I strongly believe that the secret to success lies in “being who you are” together with all your imperfections, giving the answers you know and understand, in exams, and using the exam to show your examiners all this, rather than showing them your fear.

Courage and Speaking from the Heart

We all need to be courageous, and not only in exams, but the origin of the world lies in “heart” or “cor” in Latin, something we perhaps forget. So to be courageous in an exam means really communicating what you feel about the subject. If you are at unviersity studying languages you chose this degree course because languages are something you love.. so find the meaning and personal relevance of the exam you are preparing (Yes, even if it is a “law exam”, after all “law” is related to human beings and life, so find ways to make it important and meaningful to you.)

I include this amazinf TED talk on the subject of “vulnerability” here. It lasts twenty minutes, but if you take the time to listen to Brené Brown’s words, it will be well worth the effort.

So, make yourself a cup of tea, sit down and enjoy:

Lessons from My Office Hours

What is the point of Failure and what can it teach us?

From Microsoft Clipart

 

I know this title sounds pretty bleak but it is partly the result of an episode that happened this week during my office hours, and that I thought I’d like to share with you. Before I do so, though I need to outline our C1 exam so that you will understand how both the student and I felt:

The University of Verona Language Centre C1 Test

Our C1 exam consists of three parts, and students need to pass all three to be given the credits they need. The first part is a computer based test which focuses on Use of Language in Context, Reading and Listening. This is followed by a Written test, which at this level involves producing a text of approx. 200 – 250 words in either a narrative, descriptive or discursive style on various subjects. The final test is an oral test of about 10-15 minutes which is usually in a paired format and students have to show that they can communicate in effective English about familiar and more abstract topics. In order to pass all this they need to get a minimum of 60% over the three parts of the test, and they are allowed to score from 55% to 60% on one of the parts if this is then compensated for on one of the other parts.

What this means, then, is that a student can pass the whole test if he or she scores only 55% on the computer test but this is then compensated for if he or she goes on to score 65% on the written and then 60% on the oral test, giving a total of 60% overall.

Failing your exams … or, even worse, Almost Passing

So, to come back to the episode last week, this is what happened: a student came to my university office hours last week, clutching an enormous exercise book and communicating in an English that was perfectly comprehensible. She told me that she had tried to do our C1 written paper various times and was having trouble. She had passed the computer test but couldn’t pass the composition paper. This girl had been in my course and had religiously printed out all the work she and others had done on writing during the course. (This is available for all students on my wiki, with the questions, the students’ answers and my comments and correction code. See this example page.) She had also gone on to develop this by focusing on specific areas such as phrasal verbs etc. all lovingly colour coded in her exercise book. She had come with a few intelligent questions such as what did I mean by “effective language”. (I explained that I meant language that “did its job” of communicating the message you wanted it to, clearly and well.) All this told me that she was a student who had studied hard and thought about what she was reading. This is what we want in our students, isn’t it?

Well, then she showed me a practice essay she had written in a narrative style, using an old exam question as a model. She was convinced that this was “perfect” and yet as soon as I read the first sentence I knew that we were up against quite a few problems and I could feel a sinking feeling come over me.

What was the Problem?

The first sentence was already problematic:

“I can’t descend.” Emma said.

I asked her whether the story was informal or formal and she said that that was another thing she had not really understood. What did informal language or formal language mean. I explained and gave her examples in Italian, so that she realised that the use of “descend” here wasn’t natural. You wouldn’t say that to a friend. She said that she had been avoiding simple words like “get down” because she thought they were too easy, aren’t they more of a B1 level??? was the question. It took quite some explaining to show here that words as such can have many different levels and what is important is to know how to use them.

Coffe pot and mug the perfect collocation

Collocation, colligation and Word Choice

We all know how important collocations are in natural language use, and we spend quite a lot of time talking about this in class. Despite this, however, my student had quite a few unnatural collocations such as “It was snowing abundant” (as well as the fact that she was using aan adjective instead of an adverb). She knew what collcations were but didn’t know how to find them, check them or notice them. She said, the problem was that she did not know when something was not possible. The same was true of colligation or “grammatical collocation” as you might think of this. She had looked up the word “discesa” which in Italian means a slope but can also be used as an adjective or adverb meaning downhill. She had found the word “downhill” and was using it to mean “slope” so she produced a sentence like this: “They skied along the downhill with ease.” We have a problem here of incorrect word choice followed by inappropriate colligation as a result of this.

You can see where all this is going and I’m not going to go through the whole composition but, as I explained to her, this language was not “effective” because it did not communicate what she wanted to, because of the words she was choosing, how she was using them and also because of the register. These are all things that, at a C1 level, you need to be able to do.

Two Major Problems

By this time the poor student was nearly in tears and I was feeling very depressed too, because there were two major problems here:

1) she did not have a good monolingual dictionary, despite the fact that I had recommended several. As soon as we started looking at her “problem language” in the Longman Contemporary Dictionary of English (the online version, in fact) all these problems were clear and she began to see what I was talking about. She needed to learn how to use a dictionary well, and yet, this is something we do all the time in class, so for me, as a teacher, this means there is no guarantee that students sitting in my lessons will actually benefit from what I’m teaching them to do;

2) She chose words, which were often false friends etc. and she was convinced that they were correct. One example of this was the word “structure” which can be used in italian to mean the company or firm etc. In her case it referred to a ski resort, and she wrote “The director of the structure” which does not really work in English. When I showed her that this was unnatural and we looked at structure in the dictionary she said: “But I had no idea! How do I know when something has the right meaning or not?”

This is the crux of the matter and it is very difficult to answer. My answer is that enough exposure to a language teaches you what is appropriate and what isn’t, but I’m not sure that this is always true. This student told me that she read widely in English but she read for the ideas and didn’t really notice the language, so I suggested reading first and then taking a page or so every now and then and analysing the language. Part of me, however, wonders whether there are simply some people who are interested in the way language words and others who are not. I was watching an Austrian detective series, for example, the same day and something in my mind noticed that they used the verb “recherchieren” and without even realising it, I was thinking, “Would that be used in Germany or South Tyrol?” It comes naturally to me to question these things, but does it come naturally to everyone? If it doesn’t come naturally to you, then you simply have to train yourself to do it, but I think that many of my students who have been through a traditional style of school system do not believe in the value of these things. They think they need to do a lot of grammar exercises and practice tests and then they’ll be alright, which is rather sad in university language students.

An hour later my student was thoroughly depressed at the thought that this composition would not have passsed the test either and when I asked her what she had learned she said that she had learned not to discount simple words, and to get a better dictionary.

I felt extremely drained after this and felt the need to write about it because I feel as though I have failed her in some way. This was someone who was willing to study, who told me that she uses English every day in emails at work, but I had not been able to help her prepare for her exam. One of the most difficult things for a teacher to do is to chip away at student beliefs such as the sacrosanct nature of the grammar exercise, and all you can do is keep on going in the hope that some will understand the message.

So, what can failure and suffering teach us?

We all know the value of passing a test, but failure can teach us something too. In this case the student has failed her exam but she has learned some important lessons about how she approaches another language. In my case I failed to communicate my message to her in class, but I helped here in my office hours, and the whole episode has led me on to think about the process, so that I can use it in an exemplary way to others.

Failure inevitably leads to suffering but suffering in itself has lessons to teach us as well. ON a personal level, if I never suffer then I don’t appreciate so many of those little pieces that go together to form the amazing mosaic that is every simple moment of the day. If I never have to live without “hot water” for instance, I don’t appreciate how wonderful it is to have hot water readily available when I have a shower.

On another level suffering helps us to understand what other people are going through, and if we have never suffered ourselves then we cannot develop any real empathy. I know what it means to fail a test, so I can understand what my student is going through. Empathy is essential in a teacher, and not only, because it is by understanding what somene is going through that we can start to help them to come out of it.

Rome wasn’t built in a day, as they say, but everything begins somewhere. I don’t know if my student has really understood what she needs to do for her exam, or whether I have been able to help her, but both her and my failure have taught us something as has the suffering.

Sorry to go on at length about this but I felt the need to write it down and share it to work it through in my own mind. I’d be really interested to hear other people’s take on it too.

 

“How to Pass Exams” a distance learning course on Udemy

I haven’t actually blogged for a while, unfortunately, which is largely because I’ve been busy developing a new distance learning course entitled “How to Pass Exams” on Udemy.com.

The Reasons for this Course

I decided to develop this course for my students because we spend a lot of time concentrating on language, but not so much actually on developing study strategies or revision strategies. One of the key problems for those about to do an exam is motivation during study time. We start off feeling motivated but keeping that motivation going over a period of time is another matter, so this course also looks at ways of maintaining your level of motivation. The other killer is fear, which is our companion all the way through the course “Am I good enough?” “What will the examiner be like?” etc. etc. This reaches its peak generally in the days just before an exam so there are some tips and anecdotes here as to how to deal with these feelings.

How is the Course Organised?

The course is a type of experiment. If you know me and my work you will know that I have taught in many blended learning contexts such as WizIQ or in my university courses but this is my first experience of creating a complete distance learning course. I wanted to have something that my students could access freely and work through as and when they wanted to. The course is organised in short screen cast videos made to highlight the main points of a topic and these are followed by worksheets in the form of PDFs that learners can download to their computers, tablets, smartphones etc. and work through at their own pace. The PDFs include a series of activities and links to useful sites and references so that those who are interested in a particular topic can go into it more thoroughly. If you are interested in working on memory techniques, for instance, in the section on memory you will find useful links to further reading and study as well as some simple exercises to do straight away.

Advantages and Disadvantages of Udemy

Udemy is an excellent platform and those developing courses can upload both videos and other files easily as well as linking to material you develop on other sites such as YouTube or Vimeo. The PDFs are displayed in a reader which is highquality and allows students to scroll down the document easily as well as dowloading a copy for their own use.

The site also has a “notepad” feature where students can pause the video they are watching, for instance, and take notes. They can then download these dotes directly onto their computers too, which is a very useful feature. There is also a space for asking questions. Instructors can access the course once a week, for instance to answer questions and students can ask whatever they want to.

You can also schedule and run live classes on this site, which I have yet to do… that will be the next adventure.

There are, however, a few disadvantages as well. The main one is that it is sometimes difficult to navigate the site. If you look for a course in the “discover courses” section using tags such as “exams” or “passiing exams” or “education”, all of which I had added as tags for my course, you do not find it, and you find a whole selection of courses that are not necessarily connected to what you want to do. This is a prblem that I hope will soon be worked out as it has been mentioned to Udemy several times and they promise that they are working on it.

If you want to find my courses, the best way is simply to follow the link above (at the beginning of this article) or to search for me as a person in the Udemy search box. If you search for “Sharon Hartle” you will find my page and you can access the course from there.

Anyway, the course is up and running so now I hope that it will be useful to whole generations of students so, if you are a student, or is you are about to start preparing for an exam, or if you have students who would be interested, please join the course. It is absolutely free of charge, but I would appreciate hearing people’s comments.