Move Segmenter to Components

[sffc]

Checklist for Segmenter in Components:

• Apply the boilerplate
  • Ensure everything is documented by adding back #[warn(missing_docs)] (Add missing doc for segmenter, and turn on the missing_docs warning #2366)
  • Add Clippy lints
    • icu_segmenter denies the clippy::indexing_slicing. #2325
    • Add boilerplate to segmenter #2566
    • Enable more clippy lints in segmenter #2804
• Check the public API surface and fix issues:
  • Don't re-export provider stuff at the top level (Provider's data should be in icu_segementer::provider. #2300)
  • Make certain types such as DictionarySegmenter be pub(crate) (Don't make the dictionary segmenter public. #2271)
  • Hide the symbols module from the documentation (Hide icu_segmenter's symbols module from doc #2317)
• Make LSTM no_std Rewrite f32.exp and f32.tanh? #2828
• Make LineBreakIterator borrow the data from inside of GraphemeClusterSegmenter instead of borrowing the whole segmenter #2869
• Fix up features to be additive: Document and potentially fix LSTM feature in icu_segmenter #2829
• Get word break type: Get word break type #2755
• Clean up segmenter datagen:
  • Don't print missing key errors when fallback is performed #2781
  • Support datagen with dictionary for some locales and LSTM for others #2905
  • Remove hardcoded segmenter data from datagen #3003
• Migrate LSTM segmenter to zero-copy: Migrate LSTM segmenter to zero-copy #1426
• Document most important parts of the spec in the segmenter docs and provide links into the spec Improve documentation of Segmenter #3261
• Final sign-off from @aethanyc, @makotokato, and @sffc for feature coverage and an overall easy-to-use, polished class
• Add to the icu meta-crate (after moving to components)

[sffc]

I added one more small bullet point about the symbols module (CC @robertbastian)

[aethanyc]

The symbols module is used in datagen.

icu4x/provider/datagen/src/transform/segmenter/mod.rs

Line 17 in 1d4c6a1

use icu_segmenter::symbols::*;

I wonder whether it is possible to hide it from icu_segmenter's online doc, but let it remain accessible from icu_datagen?

[sffc]

Yeah, okay, let's mark the module as #[doc(hidden)] (at the point it is pub use'd in lib.rs)

[sffc]

Follow-up from #2716: in LineBreakIterator, the grapheme segmenter data should be pre-borrowed so that we don't need to re-borrow it each time. This is a known performance bottleneck that we don't want to propagate in new code.

[robertbastian]

I've noticed that the public APIs eagerly return Vec<usize>s for segmentation points. Ideally we'd return some kind of lazy iterator. I don't think this is something we can implement for 1.2, but I'd like to change returns types to impl Iterator<Item = usize> so that we can do this in the future in a semver compatible way. Thoughts?

[aethanyc]

I've noticed that the public APIs eagerly return Vecs for segmentation points.

Which user-facing public APIs return Vec<usize>? I found internal APIs such as

icu4x/experimental/segmenter/src/complex.rs

Lines 187 to 192 in 3eff54d

	pub(crate) fn complex_language_segment_utf16(
	dictionary: Option<&Dictionary>,
	lstm: Option<&LstmPayloads>,
	grapheme: Option<&RuleBreakDataV1>,
	input: &[u16],
	) -> Vec<usize> {

to be the possible candidates. Returning impl Iterator<Item = usize> seems nice if its caller can benefit from such change.

[robertbastian]

Sorry I got confused by internal APIs that are marked pub.