Laudably, PISA have made their science tests publicly available online in many languages, including four Arabic versions labelled Israel, Qatar, Tunisia, and the UAE - don't ask me what happened to Algeria, Jordan, and Lebanon. Browsing through these, one immediately notices that the Tunisian translation (unlike the Gulf ones) has a remarkable number of grammatical errors, typos, and phrasings so awkward as to be barely comprehensible. For instance:
- Bird Migration 1: "يستعملون العدّ الذي يقوم به المتطوّعين" - wrong case: should be المتطوّعون
- Bird Migration 1: extremely awkward phrasing: "هجرة الطيور هي حركة موسمية كبيرة، يتنقل أثناءها الطيور نحو أماكن تكاثرها أو هي تعود منها." ("Bird migration is a great seasonal movement, during which birds move to the places of their reproduction and they come back from them.") Contrast the clearer phrasing in the Qatar version: "هجرة الطيور الموسمية هي انتقال واسع النطاق للطيور من وإلى مناطق تكاثرها. وفي كل عام يتولى متطوعون إحصاء عدد الطيور المهاجرة في مواقع محددة."
- Bird Migration 3: the bird's name is "الزقزوق الذهبي" in the text, but in the question it turns into "الزقزاق الذهبي".
- Running in Hot Weather 1: Garden path title: anyone looking at "العدو في الطقس الحار" is going to read it as "the enemy in hot weather", at least until the context is established. Contrast the Qatari translation "الجري في الجو الحار", using a better known, graphically unambiguous term for "running".
- Running in Hot Weather 1: Grammatical error in "يدل على ذلك {كمية العرق | ضياع الماء | درجة حرارة الجسم} العداء بعد ساعة من السباق": for the sentence to make sense (even in dialectal Arabic!), none of the alternatives should contain the definite article, since they form part of an idafa genitive. Contrast the Qatari version, which avoids the problem by putting "للعداء".
- Running in Hot Weather 2: Garden path sentence: "شرب الماء خلال السباق يمكن أن يكون له تأثير على حصول تجفّف وضربة حرارة بالنسبة إلى العداء. أيّهما؟ " Anyone reading this will start by reading the first word as šariba "he drank", giving "he drank water during the race, it can have an effect..." and only after the fifth word will they be in a position to read it, as intended, as "Drinking water during the race can have an effect on the occurrence of dehydration and heatstroke for the runner. Which of the two?" Having gotten that far, they'll still be given pause by the need to decide the intended referents of "Which of the two?" Contrast, yet again, the much easier to read Qatari version: " ماهو تأثير شرب المياه خلال الجري على تعرض العداء للجفاف وضربة الشمس ؟ " (What is the effect of drinking water during the race on the runner's exposure to dehydration and heatstroke?")
I could keep going, and no doubt more fluent Arabic speakers can find problems I haven't even noticed, but the pattern is clear: Compared to Qatari students, to say nothing of Western ones, Tunisian students were systematically disadvantaged in the PISA 2015 science tests by bad translation.
Whose fault is this? Clearly there was a failure at the level of PISA's international verification, which should have eliminated such problems. But the translations themselves are carried out at the national level (PISA2012 Technical Report Ch. 5). In other words, this mess was produced by Tunisian translators under the direction of the Tunisian government.
How is that possible? Simple: in Tunisia, appallingly enough, science is taught in French from the start of secondary school onwards. Science teachers have little need to keep up their Standard Arabic proficiency. Which raises the question of why this test, targeted at 15-year-olds, was administered in Arabic there to begin with.
12 comments:
Doubtless because Arabic is the only official language.
I'm very surprised that translating the PISA tests is done at the country level: I would have expected there to exist one version per language, not one version per country. Doesn't such needless variation from country to country defeat the whole point of standardized testing?
Also, is the test always administered in the official language of the country, as opposed to the language of instruction (wherever the two differ)? Or is that also decided at the country level?
Wow. What a clusterfuck!
The problem is even more complicated than that. Scientific Arabic terminology is quite complicated to understand with all its weird neologisms. Furthermore, I am sure that the Tunisian translators avoided using MSA words which resemble dialectal ones (for example عدو and جري or معظم and أغلبية). However, using the French language will make the results even worse. Most students don't have a good level in French. In Maths and biology classes we simply use Darja when speaking, with many weird loanwords from French and then we write in the French language. Even teachers have a medium level in French with many grammar mistakes.
I wonder if the situation is different in Algeria and Morocco?
I wonder why Arabic-speaking societies think it would be too difficult to teach science in Arabic. Some reasons that I've heard from Arab educators, the lack of terminology and the expense of publishing textbooks, can't really be valid, because other non-Western societies do teach science in their own languages. This is true with Persian in Iran and Hebrew in Israel. (I don't know about Turkey or about Arabic-language schools in Israel.) I suspect it's at least partly a matter of fear that if they create terminology it will be criticized as not grammatically "correct", and reluctance, on the other hand, to use loanwords. I also wonder whether other linguistic communities with diglossia have similar anxieties.
Etienne: Country-level localization is not entirely unnecessary - there may be differences in spelling (UK vs US), or even in accepted terminology. You would think the main body of the text at least could safely be shared, though. I suspect Tunisia simply didn't want to cooperate for some reason. It's not always the country's official language - I notice that in Spain they had a Valencian version, for instance...
Anonymous: Thanks for your observations! I assume you're Tunisian? Weird neologisms characterize scientific terminology in any language - that's par for the course. If you haven't been taught them at school they are expected to be more or less incomprehensible. In Algeria science is taught in Arabic at secondary school, then in French at university - but, as in Tunisia, science teachers tend to use a lot of Darja, and tend to be not that great at Standard Arabic.
Bob: That's probably part of the explanation. I think another part of it is lack of coordination, exacerbated by political division: different countries, and sometimes even different professors, keep coming up with mutually incompatible scientific terminologies. Persian and Hebrew are spoken in basically one country each (Afghanistan is too much of a mess to count for much in this regard), so standardizing terminology should be easier. But really, it's a puzzling belief.
I bet Turkey teaches everything in Turkish; but then Turkey is very centralized, has a certain tradition of coining neologisms and making them official, and isn't afraid of Western loanwords either (e.g. there's a scientific journal called Türkiye Jeoloji Bülteni).
Interesting observations, Lameen.
The level of sophistication of the approach to translation and adaptation in PISA does at times make it difficult to disentangle the odds and ends, and to make valid linguistic comparisons.
First, regarding the mysterious absence of the Algerian, Jordanian and Lebanese versions of “Bird Migration” and “Running in Hot Weather”: these are computer-delivered science literacy questions. Algeria, Jordan and Lebanon took a pencil-and-paper test consisting of science literacy questions taken over from previous rounds of PISA, so they didn’t use the computer-delivered test questions.
Second, in PISA there are two international source versions of the assessment materials: one in English, and one in French. Differences between the English and the French source version are indicative of the level of translation freedom deemed acceptable by domain experts (bilingual science teachers or curriculum designers). Now it turns out that Tunisia translated from French, and Qatar translated from English. You are right that it is a pity not to have a common starting point and, indeed, in PISA 2018, the Arab countries participating in the computer-based version have pooled resources to produce a common Arabic version as their starting point. All countries but one will adapt this common version as needed to conform to local usage and context.
To produce a national version, countries double translate and reconcile (two independent translators each produce one translation, and a senior translator then merges the two translations into a pre-final version that keeps the best features of each translation). The reconciled version is submitted to an international verifier, who screens it for equivalence issues and residual errors.
You have identified at least one grammatical error that the verifier did not pick up (a genitive case instead of a nominative case). However, I would think that it is a bold statement to make that such an error makes the test almost incomprehensible.
A good research question could be whether the reading workload is too important for a science assessment or not. At least, this question comes to mind in the interesting example you bring up from "Running in Hot Weather 2". You write: "شرب الماء خلال السباق يمكن أن يكون له تأثير على حصول تجفّف وضربة حرارة بالنسبة إلى العداء. أيّهما؟ " Anyone reading this will start by reading the first word as šariba "he drank", giving "he drank water during the race, it can have an effect..."
If someone reads it as a verb despite how the sentence continues, then another hypothesis could be that his/her reading ability is in question. Note that adding diacritics would have helped eliminate this problem.
Wouldn’t you agree that reading šariba as a verb would result in an ungrammatical sentence and the translation in that case would be something along the lines of “he drank water during the race it could have had an effect on... ”?
Anyway, thanks for your careful observations, they bring up good questions.
Steve: Thanks for your informative and detailed comment! I hadn't realized some countries took a different set of science questions. I gather you're involved with the PISA translation in some capacity?
I certainly wouldn't claim the case error has any significant effect on comprehensibility - most of the students probably didn't even notice it. The serious problems come not from minor morphological errors but from badly phrased, badly structured sentences that, while technically grammatical, fail to take into account the principles of information structure in Arabic. Such sentences can be deciphered, but they can hardly fail to place the student at a disadvantage compared to others who don't have to make the extra effort. This does not, by the way, reflect Anglophone vs. Francophone reading habits: Algeria-educated Algerians I've shown these texts to also find the Qatari version easier to read and better written than the Tunisian one.
As for the specific example: my point is not that the test-taker would stick with the ultimately untenable verb reading, but that they would lose time by having to reread the sentence, unlike the Qatari students who get the chance to read it correctly from the start. Diacritics could indeed have helped, but they aren't there.
Also: very glad to hear that "in PISA 2018, the Arab countries participating in the computer-based version have pooled resources to produce a common Arabic version as their starting point".
Is there anywhere we can view the pencil-and-paper questions?
Thank you for your thoughtful and considerate reply.
Yes, I was involved in coordinating translation verification.
Each version was compared to the Master version, but there was no cross-country comparison at linguistic level. Versifier give advice when they believe corrective action is needed, and national centres have the final say. So there can be differences in quality across countries. Fortunately, these should be minimized if most Arab countries use a common Modern Standard Arabic version as a starting point in the next PISA cycle.
As for the paper-based versions: unfortunately, they are under embargo, because they are being used again in PISA 2018, so they cannot be released.
After two years of studying philosophy in a Tunisian high school, it seems that nobody knows how to write a grammatically correct Arabic sentence here, except maybe Arabic teachers.
Even the translations of foreign books to Arabic are terrible.
Post a Comment