Re: HTML::Parser modifies unicode characters

Moshe Kaminsky wrote:

> * Dominic Mitchell <dom@xxxxxxxxxxxxx> [12/09/04 01:53]:
>
> > Moshe Kaminsky wrote:
> >
> > > It appears that HTML::Parser modifies some unicode characters while 
> > > parsing. The following program gives an example:
> > >
> > > #########
> > >
> > > #!/usr/bin/perl
> > > use HTML::Parser;
> > > use utf8;
> > > open TEST, '>:utf8', 'word.txt';
> > > my $p = new HTML::Parser text_h => [sub {print TEST shift}, 'text'];
> > > $p->parse("zespołów\n");
> > > close TEST;
> > >
> > > #########
> > >
> > > After running it, 'word.txt' contains "zespołów" with the funny l and 
> > > the funny o following it transformed to something else. What am I doing 
> > > wrong?
> > > I'm running: perl 5.8.5, HTML::Parser version 3.36 on linux.
> >
> > It looks like HTML::Parser is losing the UTF-8 flag.  XS modules have a 
> > nasty tendency to do this. :(
> >
> > Thankfully the workaround is fairly simple.  Add "use Encode" to the top 
> > of the script, and change the callback slightly:
> >
> >  sub { print TEST decode_utf8(shift) }
> >
> > seems to work ok here.
>
>
> Thanks! That actually works. However, my real situation is that I'm 
> using HTML::FormatText, which uses, eventually, HTML::TreeBuilder and 
> HTML::Parser. So to fix the problem, it appears that the only way is to 
> modify HTML::TreeBuilder? I wonder if the maintainers of HTML::Parser 
> are aware of this problem, and if so, why don't they do this 
> automatically (or at least add an option to do it automatically) before 
> giving the text to the handler?

Hmmm, it's a known problem:

http://search.cpan.org/~gaas/HTML-Parser-3.36/Parser.pm#BUGS

It doesn't look unsolveable, but it's slightly beyond my XS skills.  The 
key is indicating the character encoding of what you're parsing, but 
that's sometimes difficult to determine in advance (think HTML meta tags).

As to how to fix it via HTML::FormatText, I'm not sure.  You'd need to 
read through the code to find out what it's doing and fix at an 
appropriate point.  But perhaps there is another way.  Instead of 
writing out to a file, can you write to an in-memory string?  If so, 
then that string would be in UTF-8-without-the-UTF-8 flag set.  So you 
could fix that by doing "decode_utf8()" over that string before writing 
it to a file.  Or simply write that file out without any encoding which 
would do no transformation of the UTF-8 bytes.

-Dom

--
| Semantico: creators of major online resources          |
|       URL: http://www.semantico.com/                   |
|       Tel: +44 (1273) 722222 / Fax: +44 (1273) 723232  |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |