|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
library_name: transformers |
|
--- |
|
|
|
# **ArlowGPT Tokenizer** |
|
|
|
### Overview |
|
The **ArlowGPT Tokenizer** is a byte pair encoding (BPE) tokenizer developed from scratch, optimized for large-scale language modeling and text generation tasks. It features a vocabulary size of **59,575 tokens** and supports a maximum context length of **131,072 tokens**, making it suitable for handling extremely long documents and sequences. |
|
|
|
### Key Features |
|
- **Vocabulary Size**: 59,575 tokens |
|
- **Maximum Context Length**: 131,072 tokens |
|
- **Tokenizer Type**: Byte Pair Encoding (BPE) |
|
- **Special Tokens**: |
|
- `<pad>`: Padding token used for sequence alignment. |
|
- `<mask>`: Special token for masked language modeling tasks. |
|
- `<eos>`: End-of-sequence token. |
|
- `<bos>`: Beginning-of-sequence token. |
|
- **Trained From Scratch**: The tokenizer was trained from scratch using a large corpus of English and multilingual text. |
|
|
|
### Training Data |
|
The tokenizer was trained on **Wikipedia**, ensuring high coverage of general knowledge and domain-specific terms. Although primarily optimized for English, it also includes some multilingual capability due to the nature of the training dataset. |
|
|
|
### Intended Use Cases |
|
This tokenizer is designed for **general-purpose language modeling** and is suitable for tasks such as: |
|
- Autoregressive text generation |
|
- Long-context summarization |
|
- Conversational AI |
|
- Information retrieval over large documents |
|
- General NLP tasks requiring long context processing |
|
|
|
### Supported Languages |
|
- **Primary Language**: English |
|
- **Secondary Support**: Some multilingual content |
|
|
|
### Performance & Benchmarks |
|
No formal benchmarks have been conducted yet, but the tokenizer has been designed for efficiency in both tokenization speed and memory usage, with a focus on handling extremely long contexts up to **131,072 tokens**. |
|
|
|
### Limitations |
|
- **Multilingual Coverage**: While the tokenizer includes some multilingual tokens, it is primarily optimized for English text, and performance on non-English languages may vary. |
|
- **No Benchmarked Metrics**: The tokenizer has not undergone formal benchmarking for speed or performance across various tasks. |
|
|
|
### Citation |
|
If you use the **ArlowGPT Tokenizer** in your work, please cite it as: |
|
``` |
|
@misc{arlowgpt_tokenizer, |
|
title={ArlowGPT Tokenizer}, |
|
author={yuchenxie}, |
|
year={2025}, |
|
howpublished={\url{https://huggingface.co/yuchenxie/ArlowGPT-Tokenizer}} |
|
} |
|
``` |
