#tokenizers — Public Fediverse posts

Great 👌🏽:

“Strategies For Very Fast Lexers”, Matteo / ‘xnacly’ (https://xnacly.me/posts/2025/fast-lexer-strategies/).

Via HN: https://news.ycombinator.com/item?id=44560871

On Lobsters: https://lobste.rs/s/75zw2o/strategies_for_very_fast_lexers

#Compilers #Lexers #Tokenizers #LexicalAnalyzers #Speed #C #Programming #Efficiency #Optimization #PLDI

Introducing the AI Dev Gallery: Your Gateway to Local AI Development with .NET
https://devblogs.microsoft.com/dotnet/introducing-ai-dev-gallery-gateway-to-local-ai-development/

#microsoft #NET #AI #NET_9 #dev_tools #generative_ai #Machine_Learning #tokenizers #vector_search

🔧 #code2prompt: A command-line tool for converting codebases to #LLM prompts

Key features:
• 📁 Generates well-formatted #Markdown prompts with source tree structure
• 🛠️ Customizable #Handlebars templates for versatile prompt generation
• 🔍 Respects .gitignore and supports file filtering with glob patterns
• 🔢 Displays token count using various #tokenizers (cl100k, p50k, r50k_base)
• 📊 #Git diff integration for commit messages and #PullRequest descriptions
• 📋 Automatic clipboard copy and option to save output to file

Additional capabilities:
• 🔢 Line numbering for source code blocks
• 🔀 JSON output option for structured data
• 🚫 Exclusion of files/folders from source tree
• 📝 Support for user-defined variables in templates

#opensource project written in #Rust, available on #crates_io and #AUR

Useful for:
• Quick #LLM prompt generation from codebases
• Code documentation and analysis
• Bug finding and security vulnerability assessment
• Performance optimization suggestions

https://github.com/mufeedvh/code2prompt