Anthropic AI has launched Claude 3.5 Sonnet, marking the first release in its new Claude 3.5 model family. This latest iteration of Claude brings significant advancements in AI capabilities, setting a new benchmark in the industry for intelligence and performance.

Introduction to Claude 3.5 Sonnet

Anthropic AI introduced Claude 3.5 Sonnet, which is available for free on Claude.ai and the Claude iOS app. The model is accessible via the Anthropic API, Amazon Bedrock, and Google Cloud’s Vertex AI. Enhanced rate limits are provided for Claude Pro and Team plan subscribers. The pricing structure is set at $3 per million input tokens and $15 per million output tokens, with a 200K token context window, making it cost-effective and highly efficient.

Performance and Capabilities

Claude 3.5 Sonnet boasts twice the speed of its predecessor, Claude 3 Opus while maintaining mid-tier model costs. It excels in graduate-level reasoning, undergraduate-level knowledge, and coding proficiency, significantly improving understanding of nuance, humor, and complex instructions. Its ability to write high-quality content in a natural and relatable tone further solidifies its position as a leading AI model.

In internal coding evaluations, Claude 3.5 Sonnet outperformed previous models by solving 64% of problems, compared to 38% solved by Claude 3 Opus. This evaluation tested the model’s ability to fix bugs or add functionalities to an open-source codebase based on natural language descriptions. Claude 3.5 Sonnet demonstrated sophisticated reasoning and troubleshooting capabilities, making it particularly effective for updating legacy applications and migrating codebases.

Visual and Interactive Enhancements

Claude 3.5 Sonnet also improves visual reasoning, surpassing its predecessor in standard vision benchmarks. It can accurately transcribe text from imperfect images, a crucial capability for industries like retail, logistics, and financial services, where visual data interpretation is essential. This enhancement makes Claude 3.5 Sonnet highly effective in tasks requiring visual reasoning, such as interpreting charts and graphs.

Anthropic AI introduced “Artifacts,” a new feature on Claude.ai that allows users to generate and interact with content like code snippets, text documents, or website designs within a dynamic workspace. This feature marks Claude’s evolution from a conversational AI to a collaborative work environment, paving the way for team collaboration and centralized knowledge management.

Safety and Privacy

Safety and privacy remain paramount in Claude 3.5 Sonnet’s development. The model has undergone rigorous testing to minimize misuse, with safety mechanisms evaluated by external experts, including the UK’s Artificial Intelligence Safety Institute (UK AISI). These evaluations ensure the model’s robustness against misuse while maintaining user privacy. Anthropic AI does not train its generative models on user-submitted data without explicit permission, reinforcing its commitment to data privacy.

Future Developments

Anthropic AI aims to continually improve the tradeoff between intelligence, speed, and cost. Later this year, the company plans to release Claude 3.5 Haiku and Claude 3.5 Opus, completing the Claude 3.5 model family. Future developments will also include new modalities and features to support more business use cases, including integrations with enterprise applications. The team is exploring features like Memory, which will enable Claude to remember user preferences and interaction history, enhancing personalization and efficiency.

Conclusion

Claude 3.5 Sonnet represents a significant leap in AI capabilities, offering advanced reasoning, coding proficiency, and visual understanding. With its introduction, Anthropic AI continues to push the boundaries of what AI can achieve, setting new standards for performance and safety. As the Claude 3.5 model family expands, users can look forward to powerful tools to support projects and workflows.

The post Anthropic AI Releases Claude 3.5: A New AI Model that Surpasses GPT-4o on Multiple Benchmarks While Being 2x Faster than Claude 3 Opus appeared first on MarkTechPost.

Large Language Models (LLMs) have gained significant attention in the field of simultaneous speech-to-speech translation (SimulS2ST). This technology has become crucial for low-latency communication in various scenarios, such as international conferences, live broadcasts, and online subtitles. The primary challenge in SimulS2ST lies in producing high-quality translated speech with minimal delay. This requires a sophisticated policy to determine the optimal moments to initiate translation within streaming speech inputs (READ action) and subsequently generate coherent target speech outputs (WRITE action).

Current methodologies face several challenges. Existing simultaneous translation methods primarily focus on text-to-text (Simul-T2TT) and speech-to-text translation (Simul-S2TT). These approaches typically rely on cascading external modules like speech recognition (ASR) and text-to-speech synthesis (TTS) to achieve SimulS2ST. However, this cascaded approach tends to amplify inference errors progressively between modules and impedes the joint optimization of various components, highlighting the need for a more integrated solution.

Researchers have made several attempts to address the challenges in simultaneous speech-to-speech translation, primarily focusing on Simul-T2TT and Simul-S2TT translation methods. In Simul-T2TT, approaches are categorized into fixed and adaptive methods. Fixed methods, such as the wait-k policy, employ a predetermined strategy of waiting for a set number of tokens before alternating between READ and WRITE actions. Adaptive methods utilize techniques like monotonic attention, alignments, non-autoregressive architecture, or language models to dynamically perform Simul-T2TT. For Simul-S2TT, the focus has been on speech segmentation. Fixed pre-decision methods divide speech into equal-length segments, while adaptive methods split speech inputs into words or segments before applying Simul-T2TT policies. Some researchers have also explored applying offline models to Simul-S2TT tasks. Despite these advancements, these methods still rely heavily on cascading external modules, which can lead to error propagation and hinder joint optimization of the translation process.

Researchers from Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS), Key Laboratory of AI Safety, Chinese Academy of Sciences, University of Chinese Academy of Sciences, School of Future Science and Engineering, Soochow University present StreamSpeech, it addresses SimulS2ST challenges by introducing textual information for both source and target speech, providing intermediate supervision and guiding policy through text-based alignments. This direct SimulS2ST model employs a two-pass architecture, first translating source speech to target text hidden states, and then converting these to target speech. Multiple CTC decoders, optimized via ASR and S2TT auxiliary tasks, provide intermediate supervision and learn alignments for policy guidance. By jointly optimizing all modules through multi-task learning, StreamSpeech enables concurrent learning of translation and policy, potentially overcoming the limitations of previous cascaded approaches.

StreamSpeech’s architecture comprises three main components: a streaming speech encoder, a simultaneous text decoder, and a synchronized text-to-unit generation module. The streaming speech encoder utilizes a chunk-based Conformer design, which enables it to process streaming inputs while maintaining bi-directional encoding within local chunks. The simultaneous text decoder generates target text by attending to the source speech hidden states, guided by a policy that determines when to generate each target token. This policy is informed by alignments learned through multiple CTC decoders, which are optimized via auxiliary tasks of ASR and S2TT. The text-to-unit generation module employs a non-autoregressive architecture to synchronously generate units corresponding to the decoded text. Finally, a HiFi-GAN vocoder synthesizes the target speech from these units.

StreamSpeech demonstrates superior performance in both offline and S2ST tasks. In offline S2ST, it outperforms the state-of-the-art UnitY model with an average improvement of 1.5 BLEU. The model’s architecture, combining autoregressive speech-to-text translation with non-autoregressive text-to-unit generation, proves effective in balancing modeling capabilities and alignment capture. In simultaneous S2ST, StreamSpeech significantly outperforms the Wait-k baseline, showing approximately 10 BLEU improvement under low latency conditions across French, Spanish, and German to English translations. The model’s alignment-derived policy enables more appropriate translation timing and coherent target speech generation. Also, StreamSpeech shows advantages over cascaded systems, highlighting the benefits of its direct approach in reducing error accumulation and improving overall performance in Simul-S2ST tasks.

StreamSpeech represents a significant advancement in simultaneous speech-to-speech translation technology. This innovative “All in One” seamless model effectively handles streaming ASR, simultaneous translation, and real-time speech synthesis within a unified framework. Its comprehensive approach allows for improved performance across multiple tasks, including offline speech-to-speech translation, streaming ASR, simultaneous speech-to-text translation, and simultaneous speech-to-speech translation.

Check out the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 45k+ ML SubReddit

The post StreamSpeech: A Direct Simul-S2ST Speech-to-Speech Translation Model that Jointly Learns Translation and Simultaneous Policy in a Unified Framework of Multi-Task Learning appeared first on MarkTechPost.

Fireworks AI releases Firefunction-v2, an open-source function-calling model designed to excel in real-world applications. It integrates with multi-turn conversations, instruction following, and parallel function calling. Firefunction-v2 offers a robust and efficient solution that rivals high-end models like GPT-4o but at a fraction of the cost and with superior speed and functionality.

Introduction to Firefunction-v2

LLMs’ capabilities have improved substantially in recent years, particularly with releases like Llama 3. These advancements have underscored the importance of function calling, allowing models to interact with external APIs and enhancing their utility beyond static data handling. Firefunction-v2 builds on these advancements, offering a model for real-world scenarios involving multi-turn conversations, instruction following, and parallel function calling.

Firefunction-v2 retains Llama 3’s multi-turn instruction capability while significantly outperforming it in function-calling tasks. It scores 0.81 on a medley of public benchmarks compared to GPT-4o’s 0.80, all while being far more cost-effective and faster. Specifically, Firefunction-v2 costs $0.9 per output token, compared to GPT-4o’s $15, and operates at 180 tokens per second versus GPT-4o’s 69 tokens per second.

The Creation Process

The development of Firefunction-v2 was driven by user feedback and the need for a model that excels in both function calling and general tasks. Unlike other open-source function calling models, which often sacrifice general reasoning abilities for specialized performance, Firefunction-v2 maintains a balance. It was fine-tuned from the Llama3-70b-instruct base model using a curated dataset that included function calling and general conversation data. This approach ensured the preservation of the model’s broad capabilities while enhancing its function-calling performance.

Evaluation and Performance

The evaluation of Firefunction-v2 involved a mix of publicly available datasets and benchmarks such as Gorilla and Nexus. The results showed that Firefunction-v2 outperformed its predecessor, Firefunction-v1, and other models like Llama3-70b-instruct and GPT-4o in various function-calling tasks. For example, Firefunction-v2 achieved higher scores in parallel function calling and multi-turn instruction following, demonstrating its adaptability and intelligence in handling complex tasks.

Highlighted Capabilities

Firefunction-v2’s capabilities are best illustrated through practical applications. The model reliably supports up to 30 function specifications, significantly improving over Firefunction-v1, which struggled with more than five functions. This capability is crucial for real-world applications, as it allows the model to handle multiple API calls efficiently, providing a seamless user experience. Firefunction-v2 excels in instruction-following, making intelligent decisions about when to call functions, and executing them accurately.

Getting Started with Firefunction-v2

Firefunction-v2 is accessible through Fireworks AI’s platform, which offers a speed-optimized setup with an OpenAI-compatible API. This compatibility allows users to integrate Firefunction-v2 into their existing systems with minimal changes. The model can also be explored through a demo app and UI playground, where users can experiment with various functions and configurations.

Conclusion

Firefunction-v2 is a testament to Fireworks AI’s commitment to advancing the capabilities of large language models in function calling. Firefunction-v2 sets a new standard for real-world AI applications by balancing speed, cost, and performance. The positive feedback from the developer community and the impressive benchmark results underscore its potential to revolutionize how function calls are integrated into AI systems. Fireworks AI continues to iterate on its models, driven by user feedback and a dedication to providing practical solutions for developers.

Check out the Docs, model playground, demo UI app, and Hugging Face model page. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post Fireworks AI Releases Firefunction-v2: An Open Weights Function Calling Model with Function Calling Capability on Par with GPT4o at 2.5x the Speed and 10% of the Cost appeared first on MarkTechPost.

Researchers from Microsoft, the University of Massachusetts, Amherst, and the University of Maryland, College Park, address the challenge of understanding how Retrieval Augmented Generation (RAG) impacts language models’ reasoning and factual accuracy (LMs). The study focuses on whether LMs rely more on the external context provided by RAG than their parametric memory when generating responses to factual queries.

Current methods for improving the factual accuracy of LMs often involve either enhancing the internal parameters of the models or using external retrieval systems to provide additional context during inference. Techniques like ROME and MEMIT focus on editing the model’s internal parameters to update knowledge. However, there has been limited exploration into how these models balance the use of internal (parametric) knowledge and external (non-parametric) context in RAG.

The researchers propose a mechanistic examination of RAG pipelines to determine how much LMs depend on external context versus their internal memory when answering factual queries. They use two advanced LMs, LLaMa-2 and Phi-2, to conduct their analysis, employing methods like Causal Mediation Analysis, Attention Contributions, and Attention Knockouts.

The researchers utilized three key techniques to manage the inner workings of LMs under RAG:

1. Causal tracing identifies which hidden states in the model are crucial for factual predictions. By comparing a corrupted run (where part of the input is deliberately altered) with a clean run and a restoration run (where clean activations are reintroduced into the corrupted run), the researchers measure the Indirect Effect (IE) to determine the importance of specific hidden states.

2. Attention contributions look into the attention weights between the subject token and the last token in the output. This helps by analyzing how much attention each token receives to see if the model relies more on the external context provided by RAG or its internal knowledge.

3. Attention knockouts involve setting critical attention weights to negative infinity to block information flow between specific tokens. By observing the drop in prediction quality when these attention weights are knocked out, the researchers can identify which connections are essential for accurate predictions.

The results revealed that in the presence of RAG context, both LLaMa-2 and Phi-2 models showed a significant decrease in reliance on their internal parametric memory. The Average Indirect Effect of subject tokens in the query was notably lower when RAG context was present. Additionally, the last token residual stream derived more enriched information from the attribute tokens in the context rather than the subject tokens in the query. Attention Contributions and Knockouts further confirmed that the models prioritized external context over internal memory for factual predictions. However, the exact nature of how this approach works isn’t clearly understood.

In conclusion, the proposed method demonstrates that language models present a “shortcut” behavior, heavily relying on the external context provided by RAG over their internal parametric memory for factual queries. By mechanistically analyzing how LMs process and prioritize information, the researchers provide valuable insights into the interplay between parametric and non-parametric knowledge in retrieval-augmented generation. The study highlights the need for understanding these dynamics to improve model performance and reliability in practical applications.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post Unveiling the Shortcuts: How Retrieval Augmented Generation (RAG) Influences Language Model Behavior and Memory Utilization appeared first on MarkTechPost.

Meta’s Fundamental AI Research (FAIR) team has announced several significant advancements in artificial intelligence research, models, and datasets. These contributions, grounded in openness, collaboration, excellence, and scale principles, aim to foster innovation and responsible AI development.

Meta FAIR has released six major research artifacts, highlighting their commitment to advancing AI through openness and collaboration. These artifacts include state-of-the-art models for image-to-text and text-to-music generation, a multi-token prediction model, and a new technique for detecting AI-generated speech. These releases are intended to inspire further research and development within the AI community and encourage responsible advancements in AI technologies.

One of the prominent releases is the Meta Chameleon model family. These models integrate text and images as inputs and outputs, utilizing a unified architecture for encoding and decoding. Unlike traditional models that rely on diffusion-based learning, Meta Chameleon employs tokenization for text and images, offering a more streamlined and scalable approach. This innovation opens up numerous possibilities, such as generating creative captions for images or combining text prompts and images to create new scenes. The components of Chameleon 7B and 34B models are available under a research-only license, designed for mixed-modal inputs and text-only outputs, with a strong emphasis on safety and responsible use. 

Another noteworthy contribution is introducing a multi-token prediction approach for language models. Traditional LLMs predict the next word in a sequence, a method that can be inefficient. Meta FAIR’s new approach predicts multiple future words simultaneously, enhancing model capabilities and training efficiency while allowing for faster processing speeds. Pre-trained models for code completion using this approach are available under a non-commercial, research-only license.

Meta FAIR has also developed a novel text-to-music generation model named JASCO (Meta Joint Audio and Symbolic Conditioning for Temporally Controlled Text-to-Music Generation). JASCO can accept various conditioning inputs, such as specific chords or beats, to improve control over the generated music. This model employs information bottleneck layers and temporal blurring techniques to extract relevant information, enabling more versatile and controlled music generation. The research paper detailing JASCO’s capabilities is now available, with inference code and pre-trained models to be released later.

In the realm of responsible AI, Meta FAIR has unveiled AudioSeal, an audio watermarking technique for detecting AI-generated speech. Unlike traditional watermarking methods, AudioSeal focuses on the localized detection of AI-generated content, providing faster and more efficient detection. This innovation enhances detection speed up to 485 times compared to previous methods, making it suitable for large-scale and real-time applications. AudioSeal is released under a commercial license and is part of Meta FAIR’s broader efforts to prevent the misuse of generative AI tools.

Meta FAIR has also collaborated with external partners to release the PRISM dataset, which maps the sociodemographics and stated preferences of 1,500 participants from 75 countries. This dataset, derived from over 8,000 live conversations with 21 different LLMs, provides valuable insights into dialogue diversity, preference diversity, and welfare outcomes. The goal is to inspire broader participation in AI development and foster a more inclusive approach to technology design.

Meta FAIR has developed tools like the “DIG In” indicators to evaluate potential biases in their ongoing efforts to address geographical disparities in text-to-image generation systems. A large-scale study involving over 65,000 annotations was conducted to understand regional variations in geographic representation perceptions. This work led to the introduction of the contextualized Vendi Score guidance, which aims to increase the representation diversity of generated images while maintaining or improving quality and consistency.

Key takeaways from the recent research:

• Meta Chameleon Model Family: Integrates text and image generation using a unified architecture, enhancing scalability and creativity.
• Multi-Token Prediction Approach: Improves language model efficiency by predicting multiple future words simultaneously, speeding up processing.
• JASCO Model: Enables versatile text-to-music generation with various conditioning inputs for better output control.
• AudioSeal Technique: Detects AI-generated speech with high efficiency and speed, promoting responsible use of generative AI.
• PRISM Dataset: Provides insights into dialogue and preference diversity, fostering inclusive AI development and broader participation.

These contributions from Meta FAIR underline their commitment to AI research while ensuring responsible and inclusive development. By sharing these advancements with the global AI community, Meta FAIR hopes to drive innovation and foster collaborative efforts to address the challenges and opportunities in AI.

Sources

• https://ai.meta.com/blog/meta-fair-research-new-releases/
• https://x.com/AIatMeta/status/1803107817345393136
• https://x.com/i/trending/1803163175057236209
• https://arxiv.org/abs/2404.19737
• https://arxiv.org/abs/2405.09818
• https://ai.meta.com/blog/audiocraft-musicgen-audiogen-encodec-generative-ai-audio/

The post Meta FAIR’s Groundbreaking AI Releases: Enhancing Creativity, Efficiency, and Responsibility in Open Science AI Research and Development appeared first on MarkTechPost.

Evaluating Large Language Models (LLMs) is a challenging problem in language modeling, as real-world problems are complex and variable. Conventional benchmarks frequently fail to fully represent LLMs’ all-encompassing performance. A recent LinkedIn post has emphasized a number of important measures that are essential to comprehend how well new models function, which are as follows.

MixEval

Achieving a balance between thorough user inquiries and effective grading systems is necessary for evaluating LLMs. Conventional standards based on ground truth and LLM-as-judge benchmarks encounter difficulties such as biases in grading and possible contamination over time. 

MixEval solves these problems by combining real-world user inquiries with commercial benchmarks. This technique builds a solid evaluation framework by comparing web-mined questions with comparable queries from current benchmarks. A variation of this approach, MixEval-Hard, focuses on more difficult queries and provides more chances for model enhancement.

Because of its unbiased question distribution and grading system, MixEval has significant advantages over Chatbot Arena, as seen by its 0.96 model ranking correlation. It also takes 6% less time and money than MMLU, making it quick and economical. Its usefulness is further increased by its dynamic evaluation capabilities, which are backed by a steady and quick data refresh pipeline.

IFEval (Instructional Framework Standardisation and Evaluation)

The ability of LLMs to obey orders in natural language is one of their fundamental skills. However, the absence of standardized criteria has made evaluating this skill difficult. While LLM-based auto-evaluations can be biased or constrained by the evaluator’s skills, human evaluations are frequently costly and time-consuming.

A simple and repeatable benchmark called IFEval assesses this important part of LLMs and emphasizes verifiable instructions. The benchmark consists of about 500 prompts with one or more instructions apiece and 25 different kinds of verifiable instructions. IFEval offers quantifiable and easily understood indicators that facilitate assessing model performance in practical situations.

Arena-Hard

An automatic evaluation tool for instruction-tuned LLMs is Arena-Hard-Auto-v0.1. It consists of 500 hard user questions and compares model answers to a baseline model, usually GPT-4-031, using GPT-4-Turbo as a judge. Although Chatbot Arena Category Hard is comparable, Arena-Hard-Auto uses automatic judgment to provide a quicker and more affordable solution.

Of the widely used open-ended LLM benchmarks, this one has the strongest correlation and separability with Chatbot Arena. It is a great tool for forecasting model performance in Chatbot Arena, which is very helpful for researchers who want to rapidly and effectively assess how well their models perform in real-world scenarios.

MMLU (Massive Multitask Language Understanding)

The goal of MMLU is to assess a model’s multitask accuracy in a variety of fields, such as computer science, law, US history, and rudimentary arithmetic. This is a 57-item test that requires models to have a broad understanding of the world and the ability to solve problems.

On this benchmark, most models still perform at close to random-chance accuracy despite recent improvements, indicating a large amount of space for improvement. With MMLU, these flaws can be found, and a thorough assessment of a model’s professional and academic understanding can be obtained.

GSM8K

Modern language models often find multi-step mathematical reasoning difficult to handle. GSM8K addresses this challenge by offering a collection of 8.5K excellent, multilingual elementary school arithmetic word problems. On this dataset, not even the biggest transformer models are able to obtain good results.

Researchers suggest training verifiers to assess the accuracy of model completions to enhance performance. Verification dramatically improves performance on GSM8K by producing several candidate solutions and choosing the best-ranked one. This strategy supports studies that enhance models’ capacity for mathematical reasoning.

HumanEval

To assess Python code-writing skills, HumanEval has Codex, a GPT language model optimized on publicly accessible code from GitHub. Codex outperforms GPT-3 and GPT-J, solving 28.8% of the issues on the HumanEval benchmark. With 100 samples for each problem, repeated sampling from the model solves 70.2% of the problems, resulting in even better performance. 

This benchmark sheds light on the advantages and disadvantages of code generation models, offering insightful information about their potential and areas for development. HumanEval uses custom programming tasks and unit tests to assess code generation models.

Note: This article is inspired by this LinkedIn post.

The post Key Metrics for Evaluating Large Language Models (LLMs) appeared first on MarkTechPost.

In a significant leap forward for AI, Together AI has introduced an innovative Mixture of Agents (MoA) approach, Together MoA. This new model harnesses the collective strengths of multiple large language models (LLMs) to enhance state-of-the-art quality and performance, setting new benchmarks in AI. 

MoA employs a layered architecture, with each layer comprising several LLM agents. These agents utilize outputs from the previous layer as auxiliary information to generate refined responses. This method allows MoA to integrate diverse capabilities and insights from various models, resulting in a more robust and versatile combined model. The implementation has proven successful, achieving a remarkable score of 65.1% on the AlpacaEval 2.0 benchmark, surpassing the previous leader, GPT-4o, which scored 57.5%.

A critical insight driving the development of MoA is the concept of “collaborativeness” among LLMs. This phenomenon suggests that an LLM tends to generate better responses when presented with outputs from other models, even if those models are less capable. By leveraging this insight, MoA’s architecture categorizes models into “proposers” and “aggregators.” Proposers generate initial reference responses, offering nuanced and diverse perspectives, while aggregators synthesize these responses into high-quality outputs. This iterative process continues through several layers until a comprehensive and refined response is achieved.

The Together MoA framework has been rigorously tested on multiple benchmarks, including AlpacaEval 2.0, MT-Bench, and FLASK. The results are impressive, with Together MoA achieving top positions on the AlpacaEval 2.0 and MT-Bench leaderboards. Notably, on AlpacaEval 2.0, Together MoA achieved a 7.6% absolute improvement margin from 57.5% (GPT-4o) to 65.1% using only open-source models. This demonstrates the model’s superior performance compared to closed-source alternatives.

In addition to its technical success, Together MoA is designed with cost-effectiveness in mind. By analyzing the cost-performance trade-offs, the research indicates that the Together MoA configuration provides the best balance, offering high-quality results at a reasonable cost. This is particularly evident in the Together MoA-Lite configuration, which, despite having fewer layers, matches GPT-4o in cost while achieving superior quality.

MoA’s success is attributed to the collaborative efforts of several organizations in the open-source AI community, including Meta AI, Mistral AI, Microsoft, Alibaba Cloud, and DataBricks. Their contributions to developing models like Meta Llama 3, Mixtral, WizardLM, Qwen, and DBRX have been instrumental in this achievement. Additionally, benchmarks like AlpacaEval, MT-Bench, and FLASK, developed by Tatsu Labs, LMSYS, and KAIST AI, played a crucial role in evaluating MoA’s performance.

Looking ahead, Together AI plans to further optimize the MoA architecture by exploring various model choices, prompts, and configurations. One key area of focus will be reducing the latency of the time to the first token, which is an exciting future direction for this research. They aim to enhance MoA’s capabilities in reasoning-focused tasks, further solidifying its position as a leader in AI innovation.

In conclusion, Together MoA represents a significant advancement in leveraging the collective intelligence of open-source models. Its layered approach and collaborative ethos exemplify the potential for enhancing AI systems, making them more capable, robust, and aligned with human reasoning. The AI community eagerly anticipates this groundbreaking technology’s continued evolution and application.

Check out the Paper, GitHub, and Blog. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post Together AI Introduces Mixture of Agents (MoA): An AI Framework that Leverages the Collective Strengths of Multiple LLMs to Improve State-of-the-Art Quality appeared first on MarkTechPost.

Data curation is essential for developing high-quality training datasets for language models. This process includes techniques such as deduplication, filtering, and data mixing, which enhance the efficiency and accuracy of models. The goal is to create datasets that improve the performance of models across various tasks, from natural language understanding to complex reasoning.

A significant challenge in training language models is the need for standardized benchmarks for data curation strategies. This makes it difficult to discern whether improvements in model performance are due to better data curation or other factors, such as model architecture or hyperparameters. This ambiguity hinders the optimization of training datasets effectively, making it challenging for researchers to develop more accurate and efficient models.

Existing methods for data curation include deduplication, filtering, and using model-based approaches to assemble training sets. These methods are applied to large datasets to reduce redundancy and enhance quality. However, the performance of these strategies varies significantly, and there needs to be a consensus on the most effective approach for curating training data for language models. The need for clearer, standardized benchmarks further complicates this process, making it difficult to compare the effectiveness of different data curation methods.

A team of researchers from various reputed institutes including the University of Washington, Apple, and the Toyota Research Institute have introduced a novel data curation workflow called DataComp for Language Models (DCLM). This method aims to create high-quality training datasets and establish a benchmark for evaluating dataset performance. This interdisciplinary approach combines expertise from various fields to tackle the complex issue of data curation for language models.

The DCLM workflow involves several critical steps. Initially, text is extracted from raw HTML using Resiliparse, a highly efficient text extraction tool. Deduplication is performed using a Bloom filter to remove redundant data, which helps improve data diversity and reduces memorization in models. This is followed by model-based filtering, which employs a fastText classifier trained on high-quality data from sources like OpenWebText2 and ELI5. These steps are crucial for creating a high-quality training dataset known as DCLM-BASELINE. The meticulous process ensures that only the most relevant and high-quality data is included in the training set.

The DCLM-BASELINE dataset demonstrated significant improvements in model performance. When used to train a 7B parameter language model with 2.6 trillion training tokens, the resulting model achieved a 64% 5-shot accuracy on MMLU. This represents a substantial enhancement over previous models and highlights the effectiveness of the DCLM method in producing high-quality training datasets. The research team compared their results with state-of-the-art models, such as GPT-4 and Llama 3, demonstrating that the DCLM-BASELINE model performs competitively, even with reduced computational resources.

The proposed DCLM workflow sets a new benchmark for data curation in language models. It provides a comprehensive framework for evaluating and improving training datasets, which is essential for advancing the field of language modeling. The research team encourages further exploration of data curation strategies to build more effective and efficient language models. They highlight the potential for future research to expand on their findings, exploring different data sources, filtering methods, and model architectures to continue improving the quality of training datasets.

In conclusion, the DCLM workflow, a product of a collaborative effort by institutions like the University of Washington, Apple, and the Toyota Research Institute, offers a robust solution to improve dataset quality and model performance. This approach sets a new benchmark for future research in data curation and language model development. The collaborative nature of this research underscores the importance of interdisciplinary approaches in addressing complex research problems. This innovative workflow not only advances the current state of language modeling but also paves the way for future improvements in the field.

Check out the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post DataComp for Language Models (DCLM): An AI Benchmark for Language Model Training Data Curation appeared first on MarkTechPost.

Topic modeling is a technique to uncover the underlying thematic structure in large text corpora. Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA), have limitations in terms of their ability to generate topics that are both specific and interpretable. This can lead to difficulties in understanding the content of the documents and making meaningful connections between them. These models also offer limited control over the specificity and formatting of topics, hindering their practical application in content analysis and other fields requiring clear thematic categorization. The paper aims to address these limitations by proposing a new method, TopicGPT, which leverages large language models (LLMs) to generate and refine topics in a corpus.

Traditional topic modeling methods, such as LDA, SeededLDA, and BERTopic, have been widely used for exploring latent thematic structures in text collections. LDA represents topics as distributions over words, which can result in incoherent and difficult-to-interpret topics. SeededLDA attempts to guide the topic generation process with user-defined seed words, while BERTopic uses contextualized embeddings for topic extraction. Despite their utility, these models often fail to produce high-quality and easily interpretable topics.

TopicGPT, a novel framework, stands out from traditional methods in several key ways. It leverages large language models (LLMs) for prompt-based topic generation and assignment, aiming to produce topics that are more in line with human categorizations. Unlike traditional methods, TopicGPT provides natural language labels and descriptions for topics, enhancing their interpretability. This framework also allows for the generation of high-quality topics and offers users the ability to refine and customize the topics without the need for model retraining.

TopicGPT operates in two main stages: topic generation and topic assignment. In the topic generation stage, the framework iteratively prompts an LLM to generate topics based on a sample of documents from the input dataset and a list of previously generated topics. This process encourages the creation of distinctive and specific topics. The generated topics are then refined to remove redundant and infrequent topics, ensuring a coherent and comprehensive set. The LLM used for topic generation is GPT-4, while GPT-3.5-turbo is used for the assignment phase.

In the topic assignment stage, the LLM assigns topics to new documents by providing a quotation from the document that supports its assignment, enhancing the verifiability of the topics. This method has been shown to produce higher-quality topics compared to traditional methods, achieving a harmonic mean purity of 0.74 against human-annotated Wikipedia topics, compared to 0.64 for the strongest baseline. TopicGPT’s topics are also more semantically aligned with human-labeled topics, with significantly fewer misaligned topics than LDA.

The framework’s performance was evaluated on two datasets: Wikipedia articles and Congressional bills. The results demonstrated that TopicGPT’s topics and assignments align more closely with human-annotated ground truth topics than those generated by LDA, SeededLDA, and BERTopic. The researchers measured topical alignment using external clustering metrics such as harmonic mean purity, normalized mutual information, and the adjusted Rand index, finding substantial improvements over baseline methods.

TopicGPT, a groundbreaking advancement in topic modeling, not only overcomes the limitations of traditional methods but also offers practical benefits. By using a prompt-based framework and the combined power of GPT-4 and GPT-3.5-turbo, TopicGPT generates coherent, human-aligned topics that are both interpretable and customizable. This versatility makes it a valuable tool for a wide range of applications in content analysis and beyond, promising to revolutionize the field of topic modeling.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post TopicGPT: A Prompt-based AI Framework that Uses Large Language Models (LLMs) to Uncover Latent Topics in a Text Collection appeared first on MarkTechPost.

Microsoft Research has announced the release of AutoGen Studio, a low-code interface designed to streamline the creation, testing, and deployment of multi-agent AI workflows. Building on the success of the AutoGen framework, this new tool aims to democratize the development of complex AI solutions by reducing the need for extensive coding skills and providing an intuitive, user-friendly environment.

The Evolution of AutoGen

In September 2023, Microsoft Research introduced AutoGen, a versatile, open-source Python-based framework that enables the configuration and orchestration of AI agents to facilitate multi-agent applications. This framework quickly gained traction among researchers, developers, and enthusiasts, who utilized it to develop innovative applications across various domains such as market research, education, and medical data analysis.

AutoGen’s flexibility and robustness laid the groundwork for the development of AutoGen Studio. The new platform inherits all the features of AutoGen. Still, it enhances the user experience with a low-code interface that simplifies the process of building and customizing AI agents and workflows.

Key Features of AutoGen Studio

AutoGen Studio provides several key features that make it a powerful tool for developers & users of all skill levels:

1. Low-Code Environment: AutoGen Studio offers a graphical user interface that allows users to build, test, & deploy multi-agent workflows with minimal coding. Users and Developers can select from a library of predefined agents and compose them into teams to address specific tasks. The graphical interface allows users to further customize these workflows with foundation models, prompts, skills, and workflows.
2. Export and Deployment Options: Users can export agent workflows as JSON configuration files and integrate them into any Python application. These workflows can also be launched as APIs from the command line or deployed on cloud services like Azure Container Apps and Azure Web Apps.
3. Community and Collaboration: AutoGen Studio fosters a collaborative environment by allowing users to share, discover, and learn from each other’s workflows, agents, and skills. This community-driven approach is designed to cultivate expertise and promote technology reuse.
4. Responsible AI: Microsoft Research emphasizes the importance of safe and ethical AI development. AutoGen Studio incorporates features such as support for Docker environments to ensure that agents operate within controlled and secure settings, thereby reducing the risk of unintended or harmful actions.

Early Adoption and Impact

Since its early release, AutoGen Studio has seen significant interest from the AI community. Between January and May 2024, it was downloaded over 154,000 times on PyPI. Feedback from platforms like GitHub, Discord, and YouTube suggests that AutoGen Studio is attracting a new group of users with basic technical capabilities eager to test ideas rapidly without extensive programming skills.

Developers have already used AutoGen Studio to prototype various applications, from travel planning and market research to structured data extraction and video generation. This early adoption indicates the platform’s potential to lower the barrier to entry for developing sophisticated AI applications.

In conclusion, AutoGen Studio represents a significant step forward in enabling the development of multi-agent AI applications. By providing a low-code environment and fostering a collaborative community, Microsoft Research is paving the way for more accessible and responsible AI development. As the platform grows, it is poised to become an indispensable tool for academic researchers and professional developers looking to harness the power of multi-agent AI solutions.

Check out the Autogen Studio. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. 

Join our Telegram Channel and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 44k+ ML SubReddit

The post Microsoft Research Launches AutoGen Studio: A Low-Code Platform Revolutionizing Multi-Agent AI Workflow Development and Deployment appeared first on MarkTechPost.