#gpt4v — Public Fediverse posts

🔍 Major breakthrough in multimodal AI research:

#InfinityMM dataset launches with 43.4M entries across 4 categories: 10M image descriptions, 24.4M visual instructions, 6M high-quality instructions & 3M #AI generated data

🧠 Technical highlights:

New #AquilaVL2B model uses #LLaVA architecture with #Qwen25 language model & #SigLIP for image processing
Despite only 2B parameters, achieves state-of-the-art results in multiple benchmarks
Exceptional performance: #MMStar (54.9%), #MathVista (59%), #MMBench (75.2%)

🚀 Training innovation:

4-stage training process with increasing complexity
Combines image recognition, instruction classification & response generation
Uses #opensource models like RAM++ for data generation

💡 Industry impact:

Model trained on both #Nvidia A100 GPUs & Chinese chips
Complete dataset & model available to research community
Shows promising results compared to commercial systems like #GPT4V

https://arxiv.org/abs/2410.18558

🔍 Major breakthrough in multimodal AI research:

#InfinityMM dataset launches with 43.4M entries across 4 categories: 10M image descriptions, 24.4M visual instructions, 6M high-quality instructions & 3M #AI generated data

🧠 Technical highlights:

New #AquilaVL2B model uses #LLaVA architecture with #Qwen25 language model & #SigLIP for image processing
Despite only 2B parameters, achieves state-of-the-art results in multiple benchmarks
Exceptional performance: #MMStar (54.9%), #MathVista (59%), #MMBench (75.2%)

🚀 Training innovation:

4-stage training process with increasing complexity
Combines image recognition, instruction classification & response generation
Uses #opensource models like RAM++ for data generation

💡 Industry impact:

Model trained on both #Nvidia A100 GPUs & Chinese chips
Complete dataset & model available to research community
Shows promising results compared to commercial systems like #GPT4V

https://arxiv.org/abs/2410.18558

🔍 Major breakthrough in multimodal AI research:

#InfinityMM dataset launches with 43.4M entries across 4 categories: 10M image descriptions, 24.4M visual instructions, 6M high-quality instructions & 3M #AI generated data

🧠 Technical highlights:

New #AquilaVL2B model uses #LLaVA architecture with #Qwen25 language model & #SigLIP for image processing
Despite only 2B parameters, achieves state-of-the-art results in multiple benchmarks
Exceptional performance: #MMStar (54.9%), #MathVista (59%), #MMBench (75.2%)

🚀 Training innovation:

4-stage training process with increasing complexity
Combines image recognition, instruction classification & response generation
Uses #opensource models like RAM++ for data generation

💡 Industry impact:

Model trained on both #Nvidia A100 GPUs & Chinese chips
Complete dataset & model available to research community
Shows promising results compared to commercial systems like #GPT4V

https://arxiv.org/abs/2410.18558

🔍 Major breakthrough in multimodal AI research:

#InfinityMM dataset launches with 43.4M entries across 4 categories: 10M image descriptions, 24.4M visual instructions, 6M high-quality instructions & 3M #AI generated data

🧠 Technical highlights:

New #AquilaVL2B model uses #LLaVA architecture with #Qwen25 language model & #SigLIP for image processing
Despite only 2B parameters, achieves state-of-the-art results in multiple benchmarks
Exceptional performance: #MMStar (54.9%), #MathVista (59%), #MMBench (75.2%)

🚀 Training innovation:

4-stage training process with increasing complexity
Combines image recognition, instruction classification & response generation
Uses #opensource models like RAM++ for data generation

💡 Industry impact:

Model trained on both #Nvidia A100 GPUs & Chinese chips
Complete dataset & model available to research community
Shows promising results compared to commercial systems like #GPT4V

https://arxiv.org/abs/2410.18558

🔍 Major breakthrough in multimodal AI research:

#InfinityMM dataset launches with 43.4M entries across 4 categories: 10M image descriptions, 24.4M visual instructions, 6M high-quality instructions & 3M #AI generated data

🧠 Technical highlights:

New #AquilaVL2B model uses #LLaVA architecture with #Qwen25 language model & #SigLIP for image processing
Despite only 2B parameters, achieves state-of-the-art results in multiple benchmarks
Exceptional performance: #MMStar (54.9%), #MathVista (59%), #MMBench (75.2%)

🚀 Training innovation:

4-stage training process with increasing complexity
Combines image recognition, instruction classification & response generation
Uses #opensource models like RAM++ for data generation

💡 Industry impact:

Model trained on both #Nvidia A100 GPUs & Chinese chips
Complete dataset & model available to research community
Shows promising results compared to commercial systems like #GPT4V

https://arxiv.org/abs/2410.18558

🔍 #Microsoft introduces #OmniParser, a new screen parsing module for #GUI interactions:
• Converts UI screenshots into structured elements for improved #AI agent navigation
• Works with #GPT4V to generate precise actions for interface regions
• Achieves top performance on #WindowsAgentArena benchmark

🛠️ Key Components:
• Specialized datasets for icon detection and description
• Fine-tuned detection model for identifying actionable regions
• Captioning model for extracting functional semantics

📊 Performance Highlights:
• Outperforms standard #GPT4V on #ScreenSpot benchmarks
• Compatible with #Phi35V and #Llama32V models
• Functions across PC and mobile platforms without HTML dependencies

🔗 Learn more: https://www.microsoft.com/en-us/research/articles/omniparser-for-pure-vision-based-gui-agent/

Intrigued by multi-modal (text + vision) models like LLaVa, I tried an experiment to create a browser extension that walks the DOM, finds images without good alt, converts the image to Base64, sends it to LLaVa 7B 1.5 (running in LlamaCPP's server) and injects the rich description back into the image tag's alt. Needs much more work and testing, but amazing what a ~5GB (quantised at 5bit) model can do!

Edit: now on Github: https://github.com/daaain/image-alt-text-generator-extension

#buildwithai #ai #llm #llava #gpt4v #web #a11y

Intrigued by multi-modal (text + vision) models like LLaVa, I tried an experiment to create a browser extension that walks the DOM, finds images without good alt, converts the image to Base64, sends it to LLaVa 7B 1.5 (running in LlamaCPP's server) and injects the rich description back into the image tag's alt. Needs much more work and testing, but amazing what a ~5GB (quantised at 5bit) model can do!

Edit: now on Github: https://github.com/daaain/image-alt-text-generator-extension

#buildwithai #ai #llm #llava #gpt4v #web #a11y

Intrigued by multi-modal (text + vision) models like LLaVa, I tried an experiment to create a browser extension that walks the DOM, finds images without good alt, converts the image to Base64, sends it to LLaVa 7B 1.5 (running in LlamaCPP's server) and injects the rich description back into the image tag's alt. Needs much more work and testing, but amazing what a ~5GB (quantised at 5bit) model can do!

Edit: now on Github: https://github.com/daaain/image-alt-text-generator-extension

#buildwithai #ai #llm #llava #gpt4v #web #a11y

Intrigued by multi-modal (text + vision) models like LLaVa, I tried an experiment to create a browser extension that walks the DOM, finds images without good alt, converts the image to Base64, sends it to LLaVa 7B 1.5 (running in LlamaCPP's server) and injects the rich description back into the image tag's alt. Needs much more work and testing, but amazing what a ~5GB (quantised at 5bit) model can do!

Edit: now on Github: https://github.com/daaain/image-alt-text-generator-extension

#buildwithai #ai #llm #llava #gpt4v #web #a11y

GPT-4V(ision) – Features und Möglichkeiten (während Europa jetzt darauf wartet…)

#multimodalAI, #GPT4V, #ChatGPT, #KI, #OpenAI, #Bildverarbeitung, #Sprachmodelle, #NLP, #Zukunft, #Innovation

https://kinews24.de/gpt-4v/