Welcome to Brighton 2018

Welcome to Brighton 2018, come and join the fun!

The annual IATEFL conference starts officially tomorrow, Tuesday, but quite a few people were already here this morning, braving the blustery early morning rain to sign up for the Pre Conference (PCE) day. This is a day when each of the special interest groups (SIGs) organises a complete day dedicated to an interesting aspect of its ‘area of expertise’. I belong to the Tea SIG,, which does not mean that we sit around drinking cups of tea all day (although there was some of that as well) but Tea stands for ‘Testing, evaluation and assessment’. This year was dedicated to testing listening and the day consisted of very thought-provoking talks in the morning followed by lunch and then a practical focus in the afternoon as we mapped a listening activity and then developed items on it. These were then given a mini trial by another group who provided feedback. This way of structuring the day worked well, as everyone was fresh in the morning and receptive to the input provided in the traditional presentation format, but by the afternoon, I think, most people were happy to be doing something more practical. The idea was that everyone should go away, having learned something that both made them think and was practical too.

Authenticity or Reliability… Hey, what about both?

John Field began the day by calling for listening tests to be really designed in such a way as might realistically test what listeners at different levels can be expected to do. He underlined the need for ‘cognitive validity’ or rather asked: does the behaviour elicited from test takers correspond to the requirements of listening in real world contexts. His model of the way listening structures follows various stages, although they are not necessarily linear: decoding the sounds comes first, followed by word searches, recognising the boundaries between one word and another. Then comes parsing where different elements are recognised and labelled to some extent followed by the construction of meaning and finally the construction of discourse. In a nutshell, which I probably shouldn’t say, but anyway, what this means is that at lower levels the focus should be testing at word level and higher that this discourse meaning can be tested. He stressed that ‘knowledge is not recognition’ and if we are testing higher levels, we should be careful not to be testing complex cognitive processes which go beyond listening.  John said a lot more, and I have barely done his fascinating talk any justice at all, but it was a great start to the day and he left us with this thought: the perceptual prominence of any word or clause is central to a correct response to the item. If something is not stressed in some way in a listening text, then, it is not realistic to expect it to be identified and unfair to create an item around it.

Sheila Thorn, then, took over and talked about authentic listening. She has long battled for this taking on examination boards and doubters of all kinds. Her basic intuition is that so many people study a language and then go to the country where it is spoken and flounder around in the dark, understanding very little. Something is definitely wrong here. She suggests that rather than simplifying texts at lower levels we should be providing them with longer texts but testing them on the content which is comprehensible, so that they will be exposed to authentic listening bu tested on content that they can understand. She also stressed how unrealistic the idea of doing multiple choice tests whilst listening would be ‘in real life’. You don’t listen to a podcast and answer multiple choice questions, after all. She suggested that the tasks should be more natural and connected to summarising skills, which is similar to what we might do normally when listening.

Yes, but what about the Stats?

Rita Green talked about the need to collect statistical evidence when developing tests, which means trialing items, ‘playing the detective’ as you evaluate the data you collect and interpret it and only then can you actually bank those items if they correspond to your requirements, so that if, for instance, an item on a test proves to have distractors that are far too easy, or if very few people answer one question, these need to be looked at and either revised or dropped. She described classic test theory which measures mostly test taking populations and the tests themselves but she added that modern test theory takes things a step further actually examining individual test takers, for instance, and the degree of error associated with every item and every test taker. Modern test theory also looks at Fit statistics, which does not mean how fast the test taker can run away from the examiner but whether items or individuals perform in predictable or unpredictable ways. Care of course must be taken with how the data is collected and interpreted but Rita concluded by saying that ‘without field trials and data analysis we are working blind: the more valid the test, the more reliable the test scores.’

The Afternoon Session

After lunch attention tends to flag somewhat so this was the perfect moment to do something practical. Under the expert guidance of the afternoon moderator team: Thom Kiddle, Felicity O’Dell, Russell Whitehead and Russell Whitehead, we embarked on a voyage of discovery through the process of item writing. This involved firstly listening to a text and mapping it for gist, key points etc. and then comparing our results in small groups. We then wrote items for that text (our groups were assigned multiple choice). We began by deciding on the context, the learners, age, interests, needs etc. and whether we would allow them to watch the video or not. We decided not to as a text appeared in the middle, which provided the gist of the news story, s we would not have been testing listening. We then swapped items and trialed the ones produced by another group and added our constructive feedback. The items were then returned to the original writers. This was a perfect way to work in the afternoon, and whilst time was short, it gave a glimpse of what it means to be an item writer which, I think, was extremely valuable for all involved.

A few thoughts

All this gave us a lot to think about and the discussion with the panel later was interesting and quite lively at times. The question of test purpose was broached as were other issued such as Global English or test context, and test taker aims and needs. The thorny issue of whether or not to opt for multiple choice also came up and the answer was that although they may not be natural they are practical. Practicality was another key issue which jars somewhat with the notion of authenticity. To mark authentic tasks such as summarising requires a lot more ‘rater power’ than multiple choice questions. I, personally, do not feel that multiple choice items are ‘evil’, but they should be one of more options and the needs of the test takers must be central. It is useless for PhD students who have to write long articles or theses to take tests that only require 250 word essays. I know this is not listening but it is just as true of listening. the Ielts listening exam is not ‘academic’, even for those doing the academic version. For students who intend to do MAs etc. surely it makes more sense to test them on their abilities to listen to lectures. Apparently, Ielts, however, will soon be revised, so one step at a time, we are perhaps moving in new directions.

merry go round
All the fun of the fair

The fun?

If this all sounds quite serious to you and you’re wondering about the fun element, I have to say that serious things can be fun but a large element of this conference is the social side of things. Friends meet up at the conference and exchange their news and experiences, and new friends are always made here. This evening was the first event which was a welcome reception culminating in dancing, so things definitely got off to a good start.

Hope to see you around the conference. 🙂

Advertisements

Iatefl Teasig Webinar branches out to do Facebook Discussions

IMG_0598What are the Iatefl Teasig Webinars?

If you look at the image on the left you may be forgiven for thinking that the Iatefl Teasig (Testing, Evaluation and Assessment Special Interest Group) is a bit of an informal affair, and actually you would be partly right. Testing, evaluation and assessment is often thought of as being rather dry or difficult to deal with, so why not bring a warm association of a “nice cup of tea” into the picture.? In our webinars Neil Bullock and Judith Mader, the coordinators of Teasig, with a little help from me, have tried to keep an informal but informative style, reaching out to members of the sig but also others who are interested in testing and who might, in the future, become members of our sig too. The aim of the webinars is to invite interesting speakers who have something to say about testing, evaluation or assessment, to discuss their topic in a one-hour webinar. These are held regularly every few months (For more information follow this link to the Teasig site) on Adobe Connect, and are generally well received. We have been fortunate so far in having had some excellent speakers who really reach out, embracing the medium of the webinar and include the audience in their discussions. The discussions, however, tend inevitably to be “top-down” in the traditional sense. The speaker presents his or her ideas and the audience listens, comments in the chat box and asks questions. Speakers then answer some of the questions at the end of the session, or if there is not much time they answer them in a feature in the Teasig newsletter.

This has been successful so far, but we have now decided to take the process one step further to allow for greater exploration, discussion and sharing of resources by the participants. How are we doing this?

Why use Facebook for webinar discussions?

In the C21 we actually have the chance to question things like discussions and use social media to do this in interesting new ways. In the past conferences and seminars have often been about listening to experts and asking questions, learning something new and then going home. Now we have the chance to take the discussion further to reflect and share our insights with each other drawing on the largely untapped resource of audience experience and insight.  Instead of just “going home” or rather switching off the computer and heading towards a bar for a Prosecco (in my case) this week we extended the discussion of  ‘Assessing and Marking Writing” by Clare Fielder to take things further on Facebook. Why use Facebook?

Well, Facebook is a space that many of us know and use all the time, which means that like a familiar room, we can meet there to discuss the ideas that have come up, just as we might do in a cafĂ©, for instance. Being “somewhere” that we already know makes people feel comfortable and willing to post their own ideas and comments in a freer way than they might do in the actual webinar chat feed. An added adva

Looking at old things in new ways
A Space for Reflection

ntage of extending our event in this way, is that although the actually discussion itself was synchronous with me moderating it, the posts actually stay online so that all those interested in the event can go back to see them. In fact, some comments were added after the event itself, which means that a whole new asynchronous exchange starts to develop. One person, for example, Aimee Johansen, watched the recording (avaiable after the event itself)  and then commented on the Facebook Events page, that whe had found it interested and it had reminded her of some things and introduced her to other feedback methods that she would not have thought of but would like to try out. I then asked her what she would like to try in particular, so the discussion continues even a few days after the actual event.  Kent’s research about Facebook use in class discussions shows clearly that students, for instance, are happier to post on facebook than on official course discussion boards, and even though our discussions are professional and not part of a course I believe the same principle applies. As in real life there are those who like to post and others who like to follow the discussion “silently”. Whichever way you choose to use the discussion is up to you, and catering for different needs is all part of the show. For all these reasons, and particularly because Facebook is so well known, then, and many are happy using it, this was what we opted for. This was our first experience, it went well and I hope it will get even better in the future.

What happened in the Facebook Discussion?

We used the Teasig Facebook Page, which has been set up and managed by Ceyda Mutlu. Ceyda had already set up an event to advertise the webinar, as she always does and Participants were directed to this page at the end of the webinar. Some people, in fact had already accepted the invitation to attend the webinar and had posted questions and comments in advance. This meant that the discussion was already underway, in fact, before the webinar had even started!

On the evening of the webinar particpants were directed to the “Facebook Event” at the end of the webinar, and I posted the questions that had come up during the event here. Clare had been speaking about using Correction Codes to provide feedback to learners on their writing and there was a whole range of questions. I myself had quite a few including a question about how to include this kind of feedback in courses where time constraints are already an issue. Clare had outlined some of the disadvantages such as learner participation, which often comes about because learners receive a piece of written work corrected with a code that they do not understand. Time, then, must be devoted to familiarising learners both with the process and the code. I’m a big believer in learner centred teaching and developing online dialgogues with my learners, possibly becuase I tend to have very large classes, so here is a post I wrote after the 2015 Iatefl Conference which touches on developing asynchronous dialgues with learners to provide feedback and growth, so I wanted to know what Clare thought about integrating all these things into a teaching system.

We all discussed these and other ideas and shared resources and screenshots to explain what we meant, etc. This was our first “live” discussion, but which I mean that there was a moderator and participants knew that we were all “there” at that particular time, and I’m sure that things will only get better with practice, but as a first attempt it went well, so if you’r einterested go along to the discussion and have a look :-).