Can AI reliably assess Speaking and Writing? A practical guide for institutions

 

If you’re reviewing how your institution currently assesses language proficiency, we’d be happy to explore whether Langoo could fit your process. Talk to us about your assessment process

AI can score language performance. But can institutions trust the results?

Universities, language schools, and training organisations increasingly face the same challenge: assessing language proficiency accurately without turning the process into an administrative bottleneck.

Speaking and Writing are particularly demanding. Unlike multiple-choice questions, they require evaluating complex language performance against defined criteria.

Artificial intelligence offers an attractive possibility: faster assessment, more consistent processes, and the ability to handle larger numbers of candidates.

But there is an important distinction.

An AI system’s ability to generate a score does not automatically mean that the score is reliable, valid, or appropriate for a particular decision.

Recent research helps explain why.

What does the latest research tell us?

A study published in October 2026 in English for Specific Purposes, titled Automatic conversational assessment using a GPT-based model for EAP speaking, investigated whether a GPT-based conversational assessment system could produce scores comparable to those assigned by experienced human raters.

The researchers examined speaking performance among first-year university students and compared the system’s evaluations with those of four experienced lecturers.

The results were promising, but nuanced.

Overall, the AI-generated scores were slightly lower than human ratings, without a statistically significant difference in mean scores.

However, three of the eight assessment categories showed statistically significant differences between AI and human evaluations.

The researchers also found evidence of consistent scoring behaviour within the system studied.

These findings suggest that AI-based speaking assessment can demonstrate useful psychometric properties under specific conditions.

They do not establish that every AI assessment system is reliable, or that automated scoring can replace human judgement in every context. doi.org

Reliability and validity are not the same thing

When evaluating an automated language assessment solution, institutions should distinguish two concepts.

Reliability: does the assessment produce consistent results?

Reliability concerns the consistency of scores.

For example, an institution might want to know whether two students demonstrating similar language proficiency receive comparable evaluations.

It should also examine whether results remain sufficiently stable across equivalent tasks and assessment conditions.

Automated scoring may help reduce some forms of variation associated with human assessment.

However, consistency must be demonstrated through testing rather than assumed because a system uses AI.

Validity: does the assessment measure what the institution needs?

A system could produce highly consistent scores while measuring the wrong thing.

Imagine a university evaluating whether international students can participate effectively in an English-taught master’s programme.

A useful assessment might need to examine whether candidates can explain ideas, organise arguments, respond to questions, and communicate appropriately in an academic context.

A system that primarily rewards grammatical accuracy might produce consistent scores without adequately representing those abilities.

Reliability asks whether scores are consistent. Validity asks whether their interpretation and intended use are justified.

Both matter.

Why Speaking is particularly challenging

Speaking assessment involves more than recognising words and identifying grammatical errors.

Depending on the purpose of the test, institutions may need to evaluate:

• Fluency and coherence.
• Pronunciation and intelligibility.
• Grammatical range and accuracy.
• Vocabulary and appropriateness.
• Interaction and responsiveness.
• Task fulfilment.

The importance of each dimension depends on the assessment’s purpose.

For example, evaluating a student’s ability to participate in academic discussions is not identical to evaluating their performance in a prepared presentation.

This is one reason why the design of assessment tasks matters as much as the scoring technology.

The 2026 study is particularly relevant because it examined an interactive conversational system rather than treating speaking assessment solely as the evaluation of isolated recordings. ScienceDirect

What about Writing assessment?

Writing presents a related but distinct challenge.

Automated systems can analyse written responses and generate feedback on language use, structure, vocabulary, and task completion.

However, institutions should examine whether the system evaluates the intended writing abilities rather than simply rewarding surface-level linguistic features.

An academic writing assessment, for example, may need to consider argument development, organisation, use of evidence, and appropriateness for the intended audience.

A recent review of research on AI in language assessment identifies validity, fairness, transparency, and ethical considerations as important areas requiring continued attention.

The review also highlights the importance of explaining clearly which AI systems are used and how they contribute to assessment decisions. doi.org

Importantly, evidence about automated Speaking assessment should not automatically be treated as evidence about Writing assessment.

Each skill, assessment design, and intended use requires appropriate validation.

Five questions institutions should ask before adopting AI language assessment

The practical question is not simply whether AI can evaluate language proficiency.

It is whether a particular assessment system is appropriate for a particular institutional process.

1. What decision will the assessment support?

There is a substantial difference between using a language test to assign students to an appropriate class and using it to determine eligibility for admission.

The consequences of an inaccurate result are different.

Institutions should define the intended decision before selecting the technology.

2. Which language abilities need to be assessed?

Does the institution need an overall proficiency estimate?

Or does it need evidence about specific skills such as Speaking and Writing?

The assessment should reflect the competencies relevant to the programme, role, or learning environment.

3. How are scores linked to proficiency levels?

If results are reported using the Common European Framework of Reference for Languages (CEFR), institutions should understand how those levels are assigned.

Simply instructing an AI model to classify a response as B1, B2, or C1 is not sufficient evidence that its classifications are accurate.

4. What evidence supports the scoring system?

Institutions should ask providers about:

• Comparisons with qualified human evaluators.
• Consistency across assessment tasks.
• Performance across proficiency levels.
• Potential differences between groups of test-takers.
• Procedures for reviewing unexpected or disputed results.

The answers should relate to the actual system being considered.

5. What human oversight is available?

The appropriate degree of human involvement depends on the assessment’s purpose and consequences.

The International Language Testing Association’s 2026 Code of Ethics emphasises meaningful human oversight, validation, accountability, fairness, and transparency in the use of AI for language assessment.

These principles are especially important when test results influence educational or professional opportunities. iltaonline.com

AI assessment is not the same as official language certification

Another distinction matters for institutions.

A language assessment can provide information about a candidate’s proficiency without constituting an officially recognised language certification.

For example, a university may use an internal language assessment to understand a student’s level or determine placement.

However, when a programme requires a specific recognised certificate, an internal assessment cannot automatically replace that requirement.

Institutions should therefore distinguish between assessing proficiency and meeting formal certification requirements.

The appropriate solution depends on the process.

The future of language assessment is evidence-based

The research suggests that AI has genuine potential to support language assessment, particularly where institutions need scalable processes and structured evaluation.

But technological capability alone is not enough.

Reliable language assessment requires clearly defined competencies, appropriate tasks, transparent scoring criteria, validation evidence, and safeguards proportionate to the decisions being made.

For universities, language schools, and training organisations, the most useful question is not:

Can AI replace human examiners?

It is:

How can we use AI to make language assessment more efficient while preserving the quality and trustworthiness of the results?

That is the question institutions should be asking before adopting any automated assessment solution.

 

 

---

References

1. Aryadoust, V., et al. (2026). Automatic conversational assessment using a GPT-based model for EAP speaking. English for Specific Purposes, 84, 181–200. Read the study
2. Artificial Intelligence in Language Assessment Research: A Scoping Review (2026). Read the review
3. International Language Testing Association (2026). Code of Ethics. Read the guidelines

Looking for a more efficient way to assess language proficiency?

Langoo helps universities, language schools and organisations assess candidates’ language proficiency across Speaking, Writing, Listening and Reading, with results aligned to the CEFR.

If you’re reviewing how your institution currently assesses language proficiency, we’d be happy to explore whether Langoo could fit your process.

Talk to us about your assessment process