Subject: [jira] Issue Comment Edited: (LUCENE-2019) map unicode process-internal codepoints to replacement character - msg#03605

Previous Message by Date: click to view message preview

[jira] Commented: (LUCENE-2019) map unicode process-internal codepoints to replacement character

[ https://issues.apache.org/jira/browse/LUCENE-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772147#action_12772147 ] Robert Muir commented on LUCENE-2019: ------------------------------------- bq. And if I'm indexing to a RAM directory? The point is, the private-use char is never seen external to the process (which includes both Lucene and it's index). whoah, lets not confuse private-use characters with non-characters. there is a huge difference!!!! > map unicode process-internal codepoints to replacement character > ---------------------------------------------------------------- > > Key: LUCENE-2019 > URL: https://issues.apache.org/jira/browse/LUCENE-2019 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Reporter: Robert Muir > Priority: Minor > Attachments: LUCENE-2019.patch > > > A spinoff from LUCENE-2016. > There are several process-internal codepoints in unicode, we should not store > these in the index. > Instead they should be mapped to replacement character (U+FFFD), so they can > be used process-internally. > An example of this is how Lucene Java currently uses U+FFFF > process-internally, it can't be in the index or will cause problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx

Next Message by Date: click to view message preview

[jira] Commented: (LUCENE-2019) map unicode process-internal codepoints to replacement character

[ https://issues.apache.org/jira/browse/LUCENE-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772151#action_12772151 ] Steven Rowe commented on LUCENE-2019: ------------------------------------- bq. process-internal is somethign that won't be stored or interchanged in any way (internal to the process) Right, this is the crux of the disagreement: you think storage (with the exception of in-memory usage) means interchange. I and Yonik think that storage does not necessarily mean interchange. Section 16.7 (_Noncharacters_) of the Unicode 5.0.0 standand (the latest version for which an electronic version of this chapter is available), says: {quote} Noncharacters are code points that are permanently reserved in the Unicode Standard for internal use. They are forbidden for use in open interchange of Unicode text data. See Section 3.4, Characters and Encoding, for the formal definition of noncharacters and conformance requirements related to their use. The Unicode Standard sets aside 66 noncharacter code points. The last two code points of each plane are noncharacters: U+FFFE and U+FFFF on the BMP, U+1FFFE and U+1FFFF on Plane 1, and so on, up to U+10FFFE and U+10FFFF on Plane 16, for a total of 34 code points. In addition, there is a contiguous range of another 32 noncharacter code points in the BMP: U+FDD0..U+FDEF. For historical reasons, the range U+FDD0..U+FDEF is contained within the Arabic Presentation Forms-A block, but those noncharacters are not "Arabic noncharacters" or "right-to-left noncharacters," and are not distinguished in any other way from the other noncharacters, except in their code point values. Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them. If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as removing it from the text. Note that Unicode conformance freely allows the removal of these characters. (See conformance clause C7 in Section 3.2, Conformance Requirements.) In effect, noncharacters can be thought of as application-internal private-use code points. Unlike the private-use characters discussed in Section 16.5, Private-Use Characters, which are assigned characters and which are intended for use in open interchange, subject to interpretation by private agreement, noncharacters are permanently reserved (unassigned) and have no interpretation whatsoever outside of their possible application-internal private uses. *U+FFFF and U+10FFFF.* These two noncharacter code points have the attribute of being associated with the largest code unit values for particular Unicode encoding forms. In UTF-16, U+FFFF is associated with the largest 16-bit code unit value, FFFF16. U+10FFFF is associated with the largest legal UTF-32 32-bit code unit value, 10FFFF16. This attribute renders these two noncharacter code points useful for internal purposes as sentinels. For example, they might be used to indicate the end of a list, to represent a value in an index guaranteed to be higher than any valid character value, and so on. {quote} (I left out the last part about U+FFFE.) Again, the crux of the matter is the definition of "open interchange". > map unicode process-internal codepoints to replacement character > ---------------------------------------------------------------- > > Key: LUCENE-2019 > URL: https://issues.apache.org/jira/browse/LUCENE-2019 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Reporter: Robert Muir > Priority: Minor > Attachments: LUCENE-2019.patch > > > A spinoff from LUCENE-2016. > There are several process-internal codepoints in unicode, we should not store > these in the index. > Instead they should be mapped to replacement character (U+FFFD), so they can > be used process-internally. > An example of this is how Lucene Java currently uses U+FFFF > process-internally, it can't be in the index or will cause problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx

Previous Message by Thread: click to view message preview

[jira] Issue Comment Edited: (LUCENE-2019) map unicode process-internal codepoints to replacement character

[ https://issues.apache.org/jira/browse/LUCENE-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772142#action_12772142 ] Robert Muir edited comment on LUCENE-2019 at 10/30/09 10:47 PM: ---------------------------------------------------------------- Yonik, i argue in your process-internal usecase, that its in fact process-external. in this case of concatenation you should instead use a control character, or something more suitable for this purpose. for example, you could use "information separator" controls 001C-001F, and this is especially more nice than if you were to use U+FFFE, say for this purpose. some other process could recognize that you were separating information, rather than just using a noncharacter for process-internal use. process-internal is somethign that won't be stored or interchanged in any way (internal to the process) was (Author: rcmuir): Yonik, i argue in your process-internal usecase, that its in fact process-external. in this case of concatenation you should instead use a control character, or something more suitable for this purpose. process-internal is somethign that won't be stored or interchanged in any way (internal to the process) > map unicode process-internal codepoints to replacement character > ---------------------------------------------------------------- > > Key: LUCENE-2019 > URL: https://issues.apache.org/jira/browse/LUCENE-2019 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Reporter: Robert Muir > Priority: Minor > Attachments: LUCENE-2019.patch > > > A spinoff from LUCENE-2016. > There are several process-internal codepoints in unicode, we should not store > these in the index. > Instead they should be mapped to replacement character (U+FFFD), so they can > be used process-internally. > An example of this is how Lucene Java currently uses U+FFFF > process-internally, it can't be in the index or will cause problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx

Next Message by Thread: click to view message preview

[jira] Issue Comment Edited: (LUCENE-2019) map unicode process-internal codepoints to replacement character

[ https://issues.apache.org/jira/browse/LUCENE-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772180#action_12772180 ] Robert Muir edited comment on LUCENE-2019 at 10/30/09 11:41 PM: ---------------------------------------------------------------- bq. So you think that enforcing consistency is worth the cost of disallowing some usages, and I don't. no, i think this a myth. I think this is the original bug that caused index corruption * lucene used a noncharacter (happened to be U+FFFF) process-internally * lucene also treated this noncharacter as an abstract character (it later got truncated by some encoding routine, but basically it didn't correctly handle this case) by disallowing all noncharacters as term text, lucene is *more free* to use them as delimiters, and sentinel values, and such, as specified in chapter 3 of the standard. right now you only have one you treat correctly, and thats U+FFFF. <edit> Steven, by the way, I think something i havent been able to communicate properly, is that I feel very strongly that storing noncharacters in *term text* where they are treated as abstract characters, is very different than using them as sentinel values / delimiters / etc in the index format, I think this is ok and is what they are for. but term text is different, search engines index human language and by putting noncharacters in term text you are treating them as abstract characters. was (Author: rcmuir): bq. So you think that enforcing consistency is worth the cost of disallowing some usages, and I don't. no, i think this a myth. I think this is the original bug that caused index corruption * lucene used a noncharacter (happened to be U+FFFF) process-internally * lucene also treated this noncharacter as an abstract character (it later got truncated by some encoding routine, but basically it didn't correctly handle this case) by disallowing all noncharacters as term text, lucene is *more free* to use them as delimiters, and sentinel values, and such, as specified in chapter 3 of the standard. right now you only have one you treat correctly, and thats U+FFFF. > map unicode process-internal codepoints to replacement character > ---------------------------------------------------------------- > > Key: LUCENE-2019 > URL: https://issues.apache.org/jira/browse/LUCENE-2019 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Reporter: Robert Muir > Priority: Minor > Attachments: LUCENE-2019.patch > > > A spinoff from LUCENE-2016. > There are several process-internal codepoints in unicode, we should not store > these in the index. > Instead they should be mapped to replacement character (U+FFFD), so they can > be used process-internally. > An example of this is how Lucene Java currently uses U+FFFF > process-internally, it can't be in the index or will cause problems. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx