For a given language, a word breaker tokenizes text based on the lexical rules of the language.

• 对于某种给定的语言,断字符可基于该语言的词汇规则对文本进行标记化。