#humaneval — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #humaneval, aggregated by home.social.
-
🛠️ Achieves top performance in Fill-in-the-Middle (#FIM) tasks: 85.9% average accuracy across languages, 95.3% pass@1 rate
💻 Excels in multiple languages: 86.6% #Python, 78.9% #Cpp, 82.6% #JavaScript accuracy on #HumanEval benchmarks
-
[Перевод] Сравнение бенчмарков LLM для разработки программного обеспечения
В этой статье мы сравним различные бенчмарки, которые помогают ранжировать крупные языковые модели для задач разработки программного обеспечения.
https://habr.com/ru/articles/857754/
#LLM #бенчмарки #бенчмаркинг #HumanEval #DevQualityEval #CodeXGLUE #Aider #SWEbench #ClassEval #BigCodeBench
-
[Перевод] Сравнение бенчмарков LLM для разработки программного обеспечения
В этой статье мы сравним различные бенчмарки, которые помогают ранжировать крупные языковые модели для задач разработки программного обеспечения.
https://habr.com/ru/articles/857754/
#LLM #бенчмарки #бенчмаркинг #HumanEval #DevQualityEval #CodeXGLUE #Aider #SWEbench #ClassEval #BigCodeBench
-
[Перевод] Сравнение бенчмарков LLM для разработки программного обеспечения
В этой статье мы сравним различные бенчмарки, которые помогают ранжировать крупные языковые модели для задач разработки программного обеспечения.
https://habr.com/ru/articles/857754/
#LLM #бенчмарки #бенчмаркинг #HumanEval #DevQualityEval #CodeXGLUE #Aider #SWEbench #ClassEval #BigCodeBench
-
[Перевод] Сравнение бенчмарков LLM для разработки программного обеспечения
В этой статье мы сравним различные бенчмарки, которые помогают ранжировать крупные языковые модели для задач разработки программного обеспечения.
https://habr.com/ru/articles/857754/
#LLM #бенчмарки #бенчмаркинг #HumanEval #DevQualityEval #CodeXGLUE #Aider #SWEbench #ClassEval #BigCodeBench
-
🚀 #Claude35Sonnet is now rolling out on #GitHubCopilot, bringing advanced coding capabilities directly to #VisualStudioCode and https://GitHub.com
• 🏆 Performance highlights:
- Highest score among public models on #SWEbench Verified
- 93.7% accuracy on #HumanEval for #Python function writing• 💻 Key features:
- Production-ready code generation
- Inline debugging assistance
- Automated test suite creation
- Contextual code explanations• ⚙️ Technical details:
- Runs via #AmazonBedrock
- Cross-region inference for enhanced reliability
- Available to all #GitHub Copilot Chat users and organizations -
🚀 #Claude35Sonnet is now rolling out on #GitHubCopilot, bringing advanced coding capabilities directly to #VisualStudioCode and https://GitHub.com
• 🏆 Performance highlights:
- Highest score among public models on #SWEbench Verified
- 93.7% accuracy on #HumanEval for #Python function writing• 💻 Key features:
- Production-ready code generation
- Inline debugging assistance
- Automated test suite creation
- Contextual code explanations• ⚙️ Technical details:
- Runs via #AmazonBedrock
- Cross-region inference for enhanced reliability
- Available to all #GitHub Copilot Chat users and organizations -
🚀 #Claude35Sonnet is now rolling out on #GitHubCopilot, bringing advanced coding capabilities directly to #VisualStudioCode and https://GitHub.com
• 🏆 Performance highlights:
- Highest score among public models on #SWEbench Verified
- 93.7% accuracy on #HumanEval for #Python function writing• 💻 Key features:
- Production-ready code generation
- Inline debugging assistance
- Automated test suite creation
- Contextual code explanations• ⚙️ Technical details:
- Runs via #AmazonBedrock
- Cross-region inference for enhanced reliability
- Available to all #GitHub Copilot Chat users and organizations -
🚀 #Claude35Sonnet is now rolling out on #GitHubCopilot, bringing advanced coding capabilities directly to #VisualStudioCode and https://GitHub.com
• 🏆 Performance highlights:
- Highest score among public models on #SWEbench Verified
- 93.7% accuracy on #HumanEval for #Python function writing• 💻 Key features:
- Production-ready code generation
- Inline debugging assistance
- Automated test suite creation
- Contextual code explanations• ⚙️ Technical details:
- Runs via #AmazonBedrock
- Cross-region inference for enhanced reliability
- Available to all #GitHub Copilot Chat users and organizations -
🚀 #Claude35Sonnet is now rolling out on #GitHubCopilot, bringing advanced coding capabilities directly to #VisualStudioCode and https://GitHub.com
• 🏆 Performance highlights:
- Highest score among public models on #SWEbench Verified
- 93.7% accuracy on #HumanEval for #Python function writing• 💻 Key features:
- Production-ready code generation
- Inline debugging assistance
- Automated test suite creation
- Contextual code explanations• ⚙️ Technical details:
- Runs via #AmazonBedrock
- Cross-region inference for enhanced reliability
- Available to all #GitHub Copilot Chat users and organizations -
🚀 #Qwen2.5: New #AI model family released by Qwen Team
#LLM variants: 0.5B to 72B parameters, support 29+ languages including English, Chinese, French, Spanish
Specialized models: #Qwen2.5Coder for coding, #Qwen2.5Math for mathematics
128K token context length, can generate up to 8K tokens
#OpenSource under Apache 2.0 license (except 3B and 72B variants)💡 Key improvements:
Enhanced knowledge (85+ on #MMLU)
Better coding skills (85+ on #HumanEval)
Improved math capabilities (80+ on #MATH)
Stronger instruction following and long text generation
Better handling of structured data and outputs (e.g., #JSON)🔬 Performance highlights:
#Qwen2572B competitive with leading models like #Llama3 and #MistralAI
Smaller models (e.g., 3B) show impressive efficiency
#QwenPlus API model competes with #GPT4 and #Claude on some benchmarks🛠️ Available via #HuggingFace, #vLLM, and other deployment options
📊 Comprehensive benchmarks and comparisons provided in the blog post -
Proud to announce our paper on "Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis" has been accepted to Findings of #EMNLP2023 .
This is joint work with Matthieu Zimmer, Gerasimos Lampouras, Derrick Goh Xin Deik, and Ignacio Iacobacci .Code Synthesis, the generation of programming language code from a natural language description, is a challenging problem for #LLMs.
Various Reinforcement Learning methods have been proposed to improve performance of pretrained models.
One #RL approach to this problem is to use functional tests (Unit Tests) as the reward signal; however, this requires data consisting of (i) NL problem prompts, (ii) varied unit tests for each problem to assess functional correctness, which is often unavaible. Some datatasets such as #HumanEval and #MBPP exist; however, these are limited in size and contain (relatively) simple problems.We show how to programmatically derive new training data for functional test-based Code Synthesis RL, generating and converting automatic tests from a strongly typed language (Java) to a weakly typed language (Python). This allows us to generate arbitrary amounts of test-annotated data.
We then introduce a very straight-forward yet effective practical REINFORCE-based Actor-Critic RL approach that makes use of Unit Test annotated data to tune a function-level Code Synthesis LM.
Crucially, we find that keeping the Critic in sync with the Policy yields better results than pretraining and freezing the Critic.
Use of our augmentation data further improves model performance.Preprint available at https://arxiv.org/abs/2310.13669 ; code and model will be made available.
#Machinelearning #AI #ML #ReinforcementLearning #LLM #PLM #CodeSyntheis #Huawei
-
Proud to announce our paper on "Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis" has been accepted to Findings of #EMNLP2023 .
This is joint work with Matthieu Zimmer, Gerasimos Lampouras, Derrick Goh Xin Deik, and Ignacio Iacobacci .Code Synthesis, the generation of programming language code from a natural language description, is a challenging problem for #LLMs.
Various Reinforcement Learning methods have been proposed to improve performance of pretrained models.
One #RL approach to this problem is to use functional tests (Unit Tests) as the reward signal; however, this requires data consisting of (i) NL problem prompts, (ii) varied unit tests for each problem to assess functional correctness, which is often unavaible. Some datatasets such as #HumanEval and #MBPP exist; however, these are limited in size and contain (relatively) simple problems.We show how to programmatically derive new training data for functional test-based Code Synthesis RL, generating and converting automatic tests from a strongly typed language (Java) to a weakly typed language (Python). This allows us to generate arbitrary amounts of test-annotated data.
We then introduce a very straight-forward yet effective practical REINFORCE-based Actor-Critic RL approach that makes use of Unit Test annotated data to tune a function-level Code Synthesis LM.
Crucially, we find that keeping the Critic in sync with the Policy yields better results than pretraining and freezing the Critic.
Use of our augmentation data further improves model performance.Preprint available at https://arxiv.org/abs/2310.13669 ; code and model will be made available.
#Machinelearning #AI #ML #ReinforcementLearning #LLM #PLM #CodeSyntheis #Huawei
