علماء الذكاء الاصطناعي القابلون للتدريب: NVIDIA NeMo Gym وNeMo RL لأتمتة الاكتشاف العلمي

In the realm of scientific research, the tedious and repetitive nature of tasks such as literature review, experiment management, and data wrangling can significantly hinder progress. This blog post dives into how NVIDIA’s NeMo Gym and NeMo RL frameworks, coupled with the work of Edison Scientific and their Aviary environment, are revolutionizing the field by enabling the creation and training of AI-powered scientific agents. These agents can automate many of the time-consuming tasks, freeing up researchers to focus on creative problem-solving and scientific discovery.

Reinforcement Learning: Extending LLM Capabilities for Science

Traditional Language Learning Models (LLMs) excel at predicting the next token, leading to broad knowledge but often lacking in specific domain skills. Reinforcement Learning (RL) bridges this gap by allowing models to reason and act beyond supervised data. The post highlights the following key aspects:

• Pre-training: Provides a foundational understanding but lacks domain-specific expertise.
• Supervised Fine-Tuning (SFT): Learns from instruction-response pairs but is limited by dataset coverage and only rewards reproducing the reference answer.
• Reinforcement Learning (RL): Uses a reward function to score outputs, enabling models to optimize for specific goals.
  • RLHF (Reinforcement Learning from Human Feedback): Relies on human preferences for reward signals.
  • RLAIF (Reinforcement Learning from AI Feedback): Uses LLMs as judges.
  • RLVR (Reinforcement Learning with Verifiable Rewards): Uses computational checks for objective reward signals, crucial for scientific tasks.
• Scientific RL: Enables agents to design and run experiments, evaluate outcomes, and optimize toward scientific metrics.

NeMo Gym and NeMo RL: Improving Agentic Training

The integration of NeMo Gym and NeMo RL offers a powerful framework for building and evaluating LLM agents for scientific research.

• NeMo RL: Provides training algorithms, manages compute resources, and orchestrates model updates. The latest release supports features like on-policy distillation, asyncRL, advanced RL algorithms, and end-to-end FP8 RL training.
• NeMo Gym: An open-source framework for building RL training environments at scale, handling diverse dependencies and domain-specific requirements.
  • Offers three core server abstractions:
    • Model: Wraps OpenAI-compatible endpoints for reasoning and tool-calling support.
    • Resources: Provides tool implementations and verification logic.
    • Agents: Orchestrates interactions between models and resources.
• Key Functionality: NeMo Gym generates rollouts and rewards, allowing for scalable training and integration with existing systems.

Edison Scientific and Aviary: A Practical Example

Edison Scientific utilizes NeMo Gym and NeMo RL with their Aviary framework to automate scientific discovery across biology, chemistry, and related fields. Aviary provides RL training environments for various tasks, including literature research, bioinformatics data analysis, and molecular cloning.

• Aviary’s Core Methods:
  • reset(): Initializes the environment and returns the first observation.
  • step(): Executes an action and returns new observations, rewards, and termination signals.
• Example Use Case: Edison Scientific trains a Jupyter-notebook data-analysis agent for bioinformatics tasks. They’ve implemented context management features to handle large notebooks, including interaction history dropping and GRPO grouping on individual steps.
• BixBench: A benchmark of verifiable bioinformatics questions built by Edison Scientific to test and validate their system.

Building Agentic Environments with NeMo Gym

The blog post provides a step-by-step guide to building agentic environments in NeMo Gym. The key steps include:

1. Installing NeMo Gym: Cloning the repository and setting up a virtual environment.
2. Configuring the Model: Using a hosted model or deploying one locally with vLLM, enabling tool-calling.
3. Testing an Aviary Environment: Running a simple agent through the GSM8K environment, demonstrating basic functionality.
4. Building a New Environment: Adding the Aviary HotPotQA environment to NeMo Gym, showcasing extensibility.

Best Practices for Building Scientific Agents

The authors offer valuable advice for building effective scientific agents:

• Start Simple: Begin with a basic agent and incrementally add complexity.
• Reward Profiling: Measure reward statistics to create an efficient training environment.
• Monitor Training Metrics: Track metrics to identify issues such as sampling problems or model collapse.
• Train Longer: RLVR-based methods may require extended training periods to achieve significant learning.

In conclusion, the NeMo Gym and NeMo RL frameworks, combined with initiatives like Edison Scientific’s Aviary, provide a powerful and scalable platform for developing AI agents capable of automating scientific discovery. By following best practices and leveraging these tools, researchers can unlock new possibilities and accelerate scientific progress.

Source: DEVELOPER