RE: Character set problem

-----Original Message-----
From: Hannu Niemi
Sent: Wednesday, September 15, 2004 5:12 AM

In short, the problem is our national characters (with umlauts) that 
are converted to question marks (in SELECT queries at least). When 
debugging for example 'a umlaut' is hex 84 (CP850?). The server 
collation is SQL_Scandinavian_CP850_CI_AS.
---End Original Message---

Hej Hannu! 

I assume you're using TDS 0.62 or later.  

Show us your freetds.conf and a short TDSDUMP log, just to make sure things are 
set up right.  It would be good if the log included a query with a failed 
conversion.  

What is the column's datatype?  Unicode datatypes (nchar, nvarchar) have seen 
more testing than single-byte charsets.  Are both broken for you?  

http://www.microsoft.com/globaldev/nlsweb/default.asp?submitted=40b
http://www.microsoft.com/globaldev/reference/oem/850.htm
http://www.microsoft.com/globaldev/reference/sbcs/1252.htm


As a Finnish speaker, is your server configured for Locale ID 0x40B?  In 
src/tds/iconv.c::lcid2charset(), that's interpreted as code page 1252.  That 
may not be right.  You said your 'a umlaut' is hex 84, which implies CP850; for 
CP1252, it's 0xE4.  0x84 is a DOUBLE LOW-9 QUOTATION MARK, which quite likely 
isn't in your client character set, causing FreeTDS to render it as '?'.  

IOW, if your server uses Locale ID 0x40B, I think FreeTDS may be mapping your 
server's char/varchar data according to the wrong charset.  It's using charset 
1252, when it should use charset 850.  

What do these queries return?

1> sp_helpsort
2> select char(0xE4), char(0x84)
3> go
 Server default collation
 ------------------------------
 Latin1-General, binary sort

 - -
 ä ,

I don't see a conversion to any 850 charset, meaning FreeTDS won't tell 
iconv(3) that char/varchar data arriving from the server are encoded as CP850.  
If that's the problem, your TDSDUMP log should provide enough information for 
us to generate a patch.  

By the way, as long as it's broken ;-) you shouldn't be able to insert 'a 
umlaut', either.  The INSERT statement should fail (not convert your data to 
'?').  

Halsingar,

--jkl

-----------------------------------------
The information contained in this transmission may contain privileged and 
confidential information and is intended only for the use of the person(s) 
named above. If you are not the intended recipient, or an employee or agent 
responsible for delivering this message to the intended recipient, any review, 
dissemination, distribution or duplication of this communication is strictly 
prohibited. If you are not the intended recipient, please contact the sender 
immediately by reply e-mail and destroy all copies of the original message. 
Please note that we do not accept account orders and/or instructions by e-mail, 
and therefore will not be responsible for carrying out such orders and/or 
instructions.
If you, as the intended recipient of this message, the purpose of which is to 
inform and update our clients, prospects and consultants of developments 
relating to our services and products, would not like to receive further e-mail 
correspondence from the sender, please "reply" to the sender indicating your 
wishes.  In the U.S.: 1345 Avenue of the Americas, New York, NY 10105.