Re: ActiveSupport::Multibyte for better Unicode support

On 26-sep-2006, at 10:06, Michael Koziarski wrote:

> So,  if we merged in ActiveSupport::Multibyte, and updated helpers
> like truncate to use the chars proxy,  what other changes would be
> required to make this stuff simple?  Normalisation of input
> parameters?   Anything else?

Well, Normalization of input parameters depends on the situation. If  
you want to compare strings you probably want compatability  
normalization (like NFKC), but compatability normalization forms also  
looses data.

For instance, the ligature ffi:

"ffi".chars.normalize(:kc) #=> "ffi"

Or the 'vulgar fraction one quarter':

"¼".chars.normalize(:kc) #=> "1/4"

When you're comparing strings, you might want "¼" to be equal to  
"1/4". When you want your users to use nice glyphs, you can't just  
discard this data.

But _if_ you normalize, you have to make sure you _always_ normalize.  
For instance, when you save a password to the database and normalize  
it, you have to make sure that you always normalize passwords from  
forms otherwise the password might not match when filled out by the  
user. Using NFKC might introduce false positives because  
"¼".chars.normalize == "1/4".chars.normalize, which isn't a very  
large problem if the rest of the password is strong enough.

Currently normalization is implemented in a separate plugin called  
'utf8_plugin' [1], and can be turned on by the class method  
`normalize_unicode_params'.

You can find more information in your Unicode Primer [2].

Manfred

[1] https://fngtps.com/svn/multibyte_for_rails/utf8_plugin
[2] https://fngtps.com/projects/multibyte_for_rails/wiki/UnicodePrimer


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Ruby 
on Rails: Core" group.
To post to this group, send email to rubyonrails-core@xxxxxxxxxxxxxxxx
To unsubscribe from this group, send email to 
rubyonrails-core-unsubscribe@xxxxxxxxxxxxxxxx
For more options, visit this group at 
http://groups.google.com/group/rubyonrails-core
-~----------~----~----~----~------~----~------~--~---