#Iterate over string android codepoints code
#Iterate over string android codepoints series

Currently I have something like this: foreach(var ch in s) My real use case is I need to be able to detect when very large unicode characters are used in a string. What exactly is going on here? I thought that char contains a single unicode character and I never had to worry about how many bytes a character is unless I'm doing conversion to bytes. If I alter my loop to add these characters back to an empty string like so: string tmp='' Īt the end of this, tmp will print just a single character. the string is apparently composed of two characters). Instead of it printing just the single character, it prints two characters (i.e. The problems come when manipulating this unicode character. I know it's for private-use and such, but it does display a single character as I'd expect when displaying it.

#Iterate over string android codepoints code

What does Go do? (i.e.I'm using this code to generate U+10FFFC var s = (new byte ) Furthermore, this would actually necessitate a whole new OwnedUTF8 type (that's like our current String type) to store the backing data. If String and &str differed in internal encodings, there would be no way to cheaply go from String to &str, since it would require a new allocation with a reencoding step. Hence, if you're randomly accessing strings, you're not handling unicode correctly unless you're very, very careful about it.Īnyway, you can get O(1) codepoint access via byte indices, or iterate in sequence using iterators like chars (or bytes or graphemes). char_len, but iterators are more fun), and generate your own unicode abuse here. You can play with it on the playpen (and, yes, I know about. the Vec would have length 58) but only 8 visible characters (it's meant to look something this): the following text is 108 bytes, 58 codepoints (i.e. UTF-32 isn't a fixed width encoding of visible characters, it's just a fixed width encoding of codepoints. Indexing a string by codepoint is usually wrong, just like indexing by byte is usually wrong. UTF-8 is the common case (for the internet, at least), meaning storing things in UTF-8 allows reading and writing directly, no reencoding step. No endianness issues if you send things as UTF-16 or UTF-32, you need to get the byte orderings to work properly on the web, there is a lot of plain ASCII (HTML tags and CSS properties are all ASCII). Memory savings, 4-byte encoded characters aren't particularly common in practice e.g. you can't get those bytes inside the encoding of some other non-ASCII codepoint) There's not really much point in this representation, it just wastes memory for little gain.Ĭompatible with ASCII (and any byte less than 128 is guaranteed to represent that ASCII character, i.e. (Go springs to mind as an example of doing the exact opposite) why the choice to represent String internally as Vec? each character is represented in memory as multiple bytes instead of single bytes)? Is there a downside to this, other than the obvious (ie. I've literally been writing my own String replacement that uses an array of char internally the last few days and found it to be significantly easier to work with, as I can perform simple operations on strings simply using an operator:įn as_chars(&self) -> & mut While I appreciate that storing strings internally for &str is probably convenient, having String do the same thing seems like a strange choice to me. To get the nth character in the string you have to iterate over every previous character that makes representing large blocks of text in String somewhat prohibitive doesn't it? Specifically it means that indexing the string is O(n). typically 1 for ascii and 2-4 for others, depending on the locale), but aren't the downsides pretty severe?

I appreciate this is more efficient memory wise, because each UTF8 character is represented by the correct number of bytes (ie.

#Iterate over string android codepoints series

I'm wondering if someone could explain some of the thoughts behind representing String as Vec instead of as a series of code points (ie.