[Tokenizer] Add Fast Tokenizer

[DrownFish19]

PR types

New features

PR changes

APIs

Description

Add Fast Tokenizer.

• Take the tokenizers as the backend of new fast tokenziers.
• Compatible with the current tokenizers and new fast tokenizers.
• LLaMA3.1 and LLaMA3 can use PretrainedTokenizerFast to achieve better performance. LLaMA 1 and LLaMA 2 also can use LlamaTokenizerFast to improve tokenization performance.

[paddle-bot]

Thanks for your contribution!

[codecov]

Codecov Report

Attention: Patch coverage is 49.03537% with 317 lines in your changes missing coverage. Please review.

Project coverage is 54.81%. Comparing base (e0d2809) to head (e63092e).
Report is 225 commits behind head on develop.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/convert_slow_tokenizer.py	21.73%	126 Missing ⚠️
paddlenlp/transformers/tokenizer_utils_fast.py	60.93%	125 Missing ⚠️
paddlenlp/transformers/llama/tokenizer_fast.py	43.58%	44 Missing ⚠️
paddlenlp/transformers/tokenizer_utils_base.py	58.69%	19 Missing ⚠️
paddlenlp/transformers/tokenizer_utils.py	70.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #8832      +/-   ##
===========================================
+ Coverage    54.79%   54.81%   +0.01%     
===========================================
  Files          636      639       +3     
  Lines        99876   100475     +599     
===========================================
+ Hits         54732    55079     +347     
- Misses       45144    45396     +252     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.