Issue Details (XML | Word | Printable)

Key: MDL-13174
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Gordon Bateson
Reporter: Tormod Aagaard
Votes: 0
Watchers: 0
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
Moodle

When importing HotPotatoes quizzes, problem with non-ascii characters in response.

Created: 26/Jan/08 08:26 AM   Updated: 04/Mar/08 12:59 PM
Component/s: Hotpot
Affects Version/s: 1.9
Fix Version/s: 1.6.7, 1.7.5, 1.8.5, 1.9

File Attachments: 1. PNG File editquestion_unicode.png (1.17 MB)
2. File test_unicode.jcl (3 kB)

Image Attachments:

1. test_unicode_ correct.png
(18 kB)

2. test_unicode_ screen.png
(6 kB)

3. unicode.jcl.jpg
(40 kB)

Participants: Gordon Bateson, Martin Dougiamas and Tormod Aagaard
Security Level: None
Resolved date: 31/Jan/08
Affected Branches: MOODLE_19_STABLE
Fixed Branches: MOODLE_16_STABLE, MOODLE_17_STABLE, MOODLE_18_STABLE, MOODLE_19_STABLE


 Description  « Hide
When importing a question in the HotPotatoes format, non-ascii characters in the response part will not get converted, but passed on as html entities. Correct answer "prøve" will be "prøve" in the resulting quiz question.

This only applies to import of questionsto the question bank in the HotPotatoes format (.xml), and not to HotPotatoes exercises added as .htm files.

 All   Comments   Change History   Version Control      Sort Order: Ascending order - Click to sort in descending order
Gordon Bateson added a comment - 29/Jan/08 12:16 PM
Please could you upload the XML file of your HotPot quiz, so that I can investigate further.

If you don't want to make the quiz public, please send it to me via email:
gordon at-sign kanazawa hyphen kanazawa dot ac dot jp

thanks


Tormod Aagaard added a comment - 29/Jan/08 05:09 PM
A test question in cloze format.

Gordon Bateson added a comment - 29/Jan/08 10:10 PM
Please see the attached screen shot (unicode.jcl.jpg) of how your quiz appears after it has been imported into the Quiz module on my Moodle 1.9 site.

Everything looks OK to me, so perhaps I have not understood the problem.

Please could you show me a screen shot of the problem you are seeing.

thanks
Gordon


Tormod Aagaard added a comment - 29/Jan/08 10:47 PM
How did you enter the answer? cut and paste from the web? Maybe you actually entered the html-entity? When I enter the correct answer, "Tromsø" with my norwegian keyboard, it is not accepted as the right answer.

See attached screen shots of how it appears in my system. The first image shows "Tromsø" as wrong answer, the second shows the right answer according to the system( "enter the correct answer" button).

Thank you for helping.


Tormod Aagaard added a comment - 29/Jan/08 10:48 PM
Shows the right answer according to the system.

Gordon Bateson added a comment - 30/Jan/08 06:11 PM
> How did you enter the answer? cut and paste ... ?

Yes. I cut and paste the "Tromsø" prompt on the quiz page.
Unfortunately, I don't know how to enter Norwegian on my keyboard

> When I enter the correct answer, "Tromsø" with my norwegian keyboard,
> it is not accepted as the right answer.

I don't know why that would be.

I further investigated the HotPot import script

I added the following line at line 198:
print htmlspecialchars($wrapped->questiontext);

This produced the following output on the screen:

{1:SHORTANSWER:=Tromsø}

Therefore, I am pretty confident that the HotPot import script is doing what it is supposed to do. i.e. it is passing back the "ø" as a unicode character not an HTML entity.

I will try to find out how to enter Norwegian so I can replicate what you did, but at the moment I do not think this problem is caused by the HotPot import script.

Please could you give details of the OS and browser you are using on your client machine.

thanks
Gordon


Gordon Bateson added a comment - 30/Jan/08 06:29 PM
I found a page telling me how to install the Norwegian input method on Windows XP
http://www.norwegianlanguage.info/resource/keyboard.html

Having installed Norwegian keyboard, the above page says I can input a "ø" by pressing the ";" key on my English keyboard.

I installed the Norwegian keyboard and then retried the imported version of your JQuiz on my Moodle 1.9 site. I entered "ø" usingthe ";" key on my keyboard, and I confirmed that the quiz was judged to be correct by Moodle.

In your initial report you said:

> non-ascii characters in the response part will not get converted, but passed on as html entities.
> Correct answer "prøve" will be "prøve" in the resulting quiz question.

Please can you tell me where you saw the HTML entitiy: prøve

thanks
Gordon


Tormod Aagaard added a comment - 30/Jan/08 09:24 PM
Hi,

You are really working hard to solve this. Thank you!

I've tried both importing and entering "Tromsø" with opera, firefox and IE on Windows XP and firefox on linux.

I'va also tried importing the question to other installations.

It worked correctly on my test installation. 1.9. beta 3 on Windows XP. My site, however, is on Linux.

Could it be a server problem?

Importing to 1.8.2 ( on linux) gives me this message: "Warning: cannot yet handle MBCS in html_entity_decode()! in /home/2/n/nuie/www/question/format/hotpot/format.php(488) : regexp code on line 1"

I'm attaching a screen shot of the "edit question" window of the imported question.

Thank you.
Tormod


Tormod Aagaard added a comment - 30/Jan/08 09:27 PM
This is how the imported question looks when I open it for editing.

Gordon Bateson added a comment - 31/Jan/08 09:18 AM
> Could it be a server problem?

Yes, I think you have hit the nail on the head there.

a Google search for the error your reported, brings up the following page:
http://bugs.php.net/bug.php?id=25670

According to the above page this issue applies to PHP 4. It seems that in certain versions of PHP4, the "html_entity_decode" function cannot decode to UTF-8. This means when Moodle is running on certain PHP4 sites, the HTML entities will not be converted to UTF-8.

I am guessing your server uses PHP4 - would that be right?

Looking elsewhere in the Moodle codebase I see that there is already a function which seems to handle HTML entity to UTF-8 conversion. The function is in the following Moodle script:
lib/tcpdf/html_entity_decode_php4.php

Some time next week, I will change the HotPot import script so that it uses the above library and I will let you know when I have done this so that you can verify whether the fix works for you or not.


Gordon Bateson added a comment - 31/Jan/08 04:35 PM
I have modified the HotPot import script in Moodle 1.6 -> 2.0, so that it does not use the html_entity_decode() function, which appears to be broken on PHP4.

Instead the HTML entities will be converted using hotpot_charcode_to_utf8() which is based on the lib/tcpdf/html_entity_decode_php4.php which is available from Moodle 1.7 onwards.

I have confirmed that the amended HotPot import script will correctly import your JCloze activity. Please could you check that the new script works on your Moodle site too, and let me know what happens.

regards
Gordon


Tormod Aagaard added a comment - 31/Jan/08 05:07 PM
Great work!

I'll tell our systems administrator to update the system, and then I'll let you know if the fix is working.


Tormod Aagaard added a comment - 31/Jan/08 08:54 PM - edited

It works!!!

Thank you so much!

Tormod


Gordon Bateson added a comment - 31/Jan/08 09:58 PM
øøøøøøøøøøøø
Great news!
øøøøøøøøøøøø

Thanks again for reporting the issue and making the test quiz and screen shots available. I will now close this issue.

all the best
Gordon


Gordon Bateson added a comment - 31/Jan/08 10:00 PM
Fix is confirmed for Moodle 1.6 -> 2.0, so I am closing this issue.

Martin Dougiamas added a comment - 04/Mar/08 12:59 PM
Just fixing the versions