Assessment in EAP: a continuously improved means to carelessly examined ends

I am really pleased that Jane Pearson (a PhD student at Nottingham and EAP Lecturer at Kings College) has provided a stimulating post on assessment. Comments very welcome!

I was struck recently by a quotation used in an article by George Madaus (1993) which seemed to me to sum up the current state of affairs in EAP assessment. Assessment is described in the article as “a continuously improved means to carelessly examined ends” (Merton 1964 p vi in Madaus, 1993), and although this is being used to refer to the state of mainstream education in the USA, it seems especially pertinent to our politically constrained and time restricted context. In my experience, EAP course test writers tend to be divided into three groups: those with test writing experience for large scale exam boards; those with an interest in testing but little experience other than in large scale testing administration and prep courses; and those who are interested in assessment innovations, keen to emphasise authentic testing over statistical reliability, but have no framework within which to work given the dearth of examples or research evidence.  The assessment culture of EAP departments seems to lean towards one of the three points on the triangle.
Is this problematic? According to Bachman and Palmer (2010), two common testing misconceptions are that ‘experts’ should be the ones to write tests and that there is one standard method or framework, a ‘one size fits all’ approach to assessment. This means that the first and second group are likely to appropriate the most familiar method of assessment onto their EAP courses, regardless of the pedagogical aims and outcomes of the course. The third group may have the best intentions at heart, but, without a rigorous design specification, test construction cycle and validation procedure in place, can actually end up doing more harm than good, with confusing tests that change on a regular basis and do not meet criteria for either appropriate summative or effective formative assessment.
However, all of these people make invaluable contributions of EAP assessment. Therefore, it may be important to take a step back from the design of tests to allow open dialogue regarding how exactly we are defining the constructs we are testing. Some questions which could be put on the table are:
1.    Are we testing achievement or proficiency? Paran (2010)’s book Testing The Untestable points to a narrowing of the assessment agenda to include only that which is measureable, but where do critical thinking, autonomy and intercultural competence fit into this? If we are teaching and emphasising these skills, is it appropriate or logical to not test them? If we are not teaching these, but only language proficiency, on which it has been indicated that pre sessional EAP has little effect (Green, 2005), then what are the benefits for students of our courses over taking (and retaking and retaking) the IELTS or TOEFL exams for direct university entry?

2.    Are we testing four skills or integrated academic literacies and discourses? As we know, a significant body of research ( Lea and Street, 1998; Zamel,1998; Lillis,2003) suggests that deficiencies in the latter are the main barrier to success for international students. Yet the shift to a means of testing which acknowledges the complexity of skills and literacies in academia, while still providing a score in the ‘four skills’,  can lead to unthoughtout  assessments which may have face validity but little else. How many of us test speaking using a presentation, mainly because it represents an authentic means of assessment in HE, without considering if it is a fair assessment of the linguistic construct of ‘speaking’? Students may spend weeks researching, planning and practising a critical presentation, only to receive a low score due to poor grammar, lexis or pronunciation which is weighted more heavily because, while paying lip service to authentic assessment, we are obliged to assess mainly language proficiency. On the flip side, students with a high level of proficiency may receive a lower grade than expected due to a poor lack of planning or evaluative skills. Again, if we are testing achievement of academic skills learned and applied, this is fair; if testing language proficiency, it may not be. Perhaps the answer to this is to remind ourselves of Spolsky’s (1997) warning that the search for a ‘fair test’ may lead us down a dead end, and rather, that we need to make it transparent to all stakeholders what our assessments are trying to do. This may include an explicit definition of our constructs and how these link to pedagogy, along with the acknowledgement that they represent a theory of language and academic discourse particular to our context and imposed by us, as those in control of the process, rather than objective truth.

3.    Are we bound by the need to test in ways which are most familiar to us? And if we try to test in an alternative way, do we leave ourselves open to criticisms of lack of robustness? Alternative assessments do not often lend themselves to statistical validation procedures and thus are they considered unreliable or invalid? What kinds of evidence would we need in order to claim our alternative, integrated, process oriented tests are meeting all stakeholders’ needs? Do we need to redefine our paradigms of assessment validation to include a more interpretivist approach (Moss, 1992, McNamara, 2001)? What is preventing research from being conducted on alternative assessments in EAP contexts in the same way as in mainstream education? Obviously, questions are raised but no answers given and I would be fascinated to hear other practitioners’ views on these matters. As assessment affects us all, it would seem that there are ‘ends’ that need to be examined before we can begin to focus on the ‘means’ with which to assess our students.
References
Lea, M. & Street, B. V. (1998). Student Writing and Staff Feedback in Higher Education: An Academic Literacies Approach. Studies in Higher Education 23(2):157-72.
Lillis, T. (2003). Student writing as ‘academic literacies’: Drawing on Bakhtin to move from critique to design.  Language and education 17 (3 ):192-207
Madaus, G. (1993). A National testing system: manna from above? A historical/ technical perspective   Educational assessment 1 (1): 9-26
Moss, P. A.(1996.) Enlarging the Dialogue in Educational Measurement: Voices From Interpretive Research Traditions  Educational researcher 25(1): 20-28
Macnamara, T. (2001).  Language assessment as social practice: challenges for research. Language testing 18(4): 333 -349
Spolsky, B. (1997). The ethics of gatekeeping tests: what have we learned in a 100 years?.  Language testing 14(3) 242-247
Zamel, V. (1998) Strangers in academia: the experiences of faculty and ESL students across the curriculum p249-264 IN Negotiating academic literacies: teaching and learning across languages and cultures eds Spack, R and Zamel V  Lawrence Erlbaum associates new jersey

Advertisements

12 Comments on “Assessment in EAP: a continuously improved means to carelessly examined ends”

  1. Great post. One observation that resonated with me most was – “we need to make it transparent to all stakeholders what our assessments are trying to do. This may include an explicit definition of our constructs and how these link to pedagogy, along with the acknowledgement that they represent a theory of language and academic discourse particular to our context and imposed by us.”
    In dealing with the the bete noire that is assessment do we actually forget that there are stakeholders involved? I think we do. Do we really have a coherent definition of our constructs and if we do, do we ensure that these are transparently communicated to all stakeholders? Unfortunately not. We should. Assessment should be informed by all stakeholders and should be borne from a dialogue between all parties. Why should it be in the hands of the EAP practitioner alone? Can’t we co-construct assessment? Couldn’t we then ‘test’ language and academic literacies in a meaningful way – craft something truly bespoke to the student body and the institution, something authored by all?
    Ok, I’m clearly in the third group and would probably do more harm than good, but this post has me thinking …

    Like

    • uefap says:

      “Do we really have a coherent definition of our constructs and if we do, do we ensure that these are transparently communicated to all stakeholders?” Yes, we do. “We should.” We do. ” Assessment should be informed by all stakeholders and should be borne from a dialogue between all parties. ” It often is. “Why should it be in the hands of the EAP practitioner alone?” It isn’t. ” Can’t we co-construct assessment?” We do. “Couldn’t we then ‘test’ language and academic literacies in a meaningful way – craft something truly bespoke to the student body and the institution, something authored by all?” We do.

      Like

  2. sys says:

    I strive to measure both academic achievement and language proficiency in the speaking component I teach in an EAP program. I do use presentations (but only as one of several assessment tools) and consider both the academic skills and the language use exhibited by the student. A highly proficient student is asked to recognize the need to plan carefully and scrutinize sources critically, while an academically-skilled student is asked to be aware of the importance of speaking accurately, fluently, and with confidence. One challenge is in the weighting of the assessment. Are achievement and proficiency equally important, or should one be given primacy over the other, and, if so, to what extent? What is the point of a display of proficiency when sources are not understood, let alone evaluated for credibility? Conversely, how effective is a well-researched and planned speech when the audience’s comprehension is continually impeded by failures in expression? Another challenge is that with so many academic skills needing to be taught in such a limited time, proficiency becomes almost an afterthought. Perhaps this partly explains Green’s (2005) conclusion that EAP programs have little effect on language proficiency.

    Like

  3. uefap says:

    I am surprised that you should feel this way. As an external examiner for the last 15 years, I have seen much that goes on in assessment in EAP and have found a lot of good quality, well thought-out, informed assessment. The three groups you mention bear little resemblance to what I have seen.

    Like

    • Jayne Pearson says:

      I wholeheartedly agree with you that there is a lot of good quality assessment out there and I don’t think falling into any of the three groups ( or straddling them) is actually a bad thing. I think the point I was trying to make is that is common within the EAP community, and this is largely due to external constraints, for us to sideline those difficult questions about what and why we are testing for a focus on the mechanics. I definitely don’t think we can afford to rest on our laurels regarding assessment when we are continually questioning and exploring other aspects of our practice that are directly affected by assessment.

      I agree with Susie that the responsibility for assessment is rarely, in my experience, a collaborative institutional effort, and almost never takes the student into account. How often is the rationale( whatever it may be) behind our tests articulated to them?

      Like

  4. uefap says:

    I am very surprised that you should feel this way. I have been involved in externally examining EAP courses for the last 20 years and have seen some excellent examples of innovative, well informed EAP testing. I do not recognise the three groups you mention at all. Look around you.

    Like

    • Diane Schmitt says:

      Great post Jayne. I wholeheartedly agree with you and unfortunately feel I have to disagree with Andy. I’ve also externally examined a lot of EAP and I find the state of EAP assessment quite depressing indeed. This is, in my view, not because of a lack of hard work and commitment on the part of teachers, but more because language testing requires skills and knowledge beyond those required for teaching. The DELTA doesn’t teach much about assessment and the last time I looked neither do most UK masters programmes. Even though few teachers have this skill set, they are often required to develop high stakes assessment without having a clear idea of the construct of what’s being assessed. EAP providers both private and public need to invest more in developing assessment literacy among staff who are required to develop assessments.

      Like

  5. Ebefl says:

    Very interesting and it reminds me of a central dilemma I saw at the BALEAP conference, namely ‘What are we’? Every speaker seemed to have a different view on this. Some saw us as part of the academia while others, language teachers, the university sees us (oul for example) as part of ‘corporate services’.

    If we don’t know what we are, then we can’t know what we should be teaching, and thus how we ought to test students. Is it our responsibility to teach critical thinking or should we focus on noun phrases? I don’t know the answer.
    I read with interest that pre sessionals hardly effect our student’s language ability. I wonder though if that is argument to teach something else (skills?) or look more closely at the type of language we are trying to teach?

    Like

  6. Steve Bolton says:

    Interesting…
    Green 2005 – Do you have the full reference for this?
    Many thanks!

    Like

    • Jayne Pearson says:

      I do apologise for the omission! The reference is Green, A ( 2005) EAP study recommendations and score gains on the IELTS academic writing test. Assessing Writing 10 pp 44-60.
      I have taken the reference a little out of context in my post, as the point Tony Green is making is that proficiency test such as IELTS are of little effect in measuring gains on EAP courses, rather that that there are NO gains. It indicates to me the issues that we have in mapping our criteria to international tests such as IELTS and the tensions between assessing achievement and proficiency.

      Like

  7. […] tutors included in assessment process And vice versa! See this post on Teaching EAP blog. Carol Bailey from Wolverhampton University mentioned exploring co-marking […]

    Like

  8. I was also really interested to read these posts, as the same kind of questions have been intriguing me. In a similar spirit to how people have responded above, my concerns about EAP assessment don’t relate to colleagues’ lack of commitment to producing high-quality assessments but rather stem for a concern that there is often limited time for EAP professionals to refer to language testing research, some of which can be somewhat complex to interpret.

    The issues which have grasped my own attention also relate to the social implications of poor interpretations drawn from testing, namely limited construct validity and limitations in measurement good practice.Experts on this topic include peple like Shohamy(2001) and Benesch (2001).

    This is what led me to complete my EdD on the topic of EAP Assessment Literacy. The research which I undertook investigated the views of EAP teachers through interviews and a questionnaire protocol. I have to say, the process was fascinating. The responses I collected revealed a whole host of areas which can usefully be developed in order to assist many busy EAP teachers in acquiring and implementing skills for assessment, with closer reference to advice from testing research.

    My research has it’s limitations, as you can imagine, but I’m currently working on a number of ideas for trying to share the recommendations which emerged, so that hopefully they can form a resource for colleagues with limited time, who wish to enhance or maintain their EAP assessment skills.

    Anyone interested in Assessment Literacy might like to look at the work of Stiggins (1991,1995) Popham (2001,2012). With more reference to EAP, see Taylor (2009).

    I’m happy to go into more detail if anyone wishes.

    Benesch, S (2001) Critical English for Academic Purposes: Theory, Politics, and Practice
    Popham, W. J. (2001). The Truth about Testing: An Educator’s Call to Action
    Popham, W. J. (2012 Mastering Assessment: a self service system for educators
    Shohamy, E. (2001). The Power of Tests
    Stiggins, R. (1991). Assessment Literacy. Phi Delta Kappan
    Stiggins, R. (1995). Assessment Literacy for the 21st Century
    Taylor, L. (2009) Developing Assessment Literacy. Annual Review of Applied Linguistics 29.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s