tokenization
PackAI & modelsLets your agent break text into tokens using common tokenizer libraries like HuggingFace and SentencePiece.
Unavailable. Delivery for this kind is on the roadmap — not serving yet.
Add ahel to your AI once: Claude, ChatGPT, Cursor, Claude Code or Codex. Then ask it to use this.
About this app
Text tokenization for LLMs including HuggingFace Tokenizers and SentencePiece. Use when training custom tokenizers or handling multilingual text.
Signals
- GitHub stars
- 13k
- Forks
- 931
- Last commit
- Jun 2026
Advanced
- Item type
- plugin
- Key
orchestra-research-ai-research-skills-tokenization- Source
- github.com/orchestra-research/ai-research-skills