AllenAI Open Instruct Tulu 3 Post-Training with SFT, DPO, RLVR, GRPO, and Verifier-Based Evaluation

Sicherheitsmeldung mit Schweregrad noch nicht bewertet. Technische Details im Tab „Originaltext“; empfohlene Schritte in der Checkliste.

Originaltext der Quelle (nicht übersetzt) – die deutsche Zusammenfassung findest du im Tab „Kurzfassung".

In this tutorial, we build an end-to-end post-training pipeline for a compact instruction-tuned language model using AllenAI’s Open Instruct framework. We move through three major training stages: Supervised Fine-Tuning, Direct Preference Optimization, and Reinforcement Learning with Verifiable Rewards using GRPO, while adapting the original multi-GPU Tulu 3 stack to fit within a 16 GB runtime. We clone the Open Instruct repository, selectively load its native loss and utility functions, configure LoRA adapters, prepare GSM8K data for each training stage, and use deterministic verifiers to evaluate generated mathematical answers. Throughout the workflow, we preserve the core optimization logic of Open Instruct while replacing distributed components such as vLLM, Ray actors, DeepSpeed, and asynchronous rollout queues with lightweight Hugging Face and PyTorch implementations suitable for Colab.

Copy CodeCopiedUse a different Browserimport os, sys, subprocess, textwrap, json, math, random, re, ast, types, dataclasses, gc, contextlib REPO_URL = "https://github.com/allenai/open-instruct.git" REPO_DIR = "/content/open-instruct" if os.path.isdir("/content") else "./open-instruct" PIP_PKGS = [ "peft", "accelerate", "ray", "wandb", "beaker-py", "langdetect==1.0.9", "immutabledict==1.2.0", "nltk", "absl-py", "sympy", "antlr4-python3-runtime==4.11", "tiktoken", ] def sh(*args): print("$", " ".join(args)) subprocess.run(args, check=False) def setup(): sh(sys.executable, "-m", "pip", "install", "-q", *PIP_PKGS) if not os.path.isdir(REPO_DIR): sh("git", "clone", "--depth", …

Vollständiger Originalartikel ↗

• Betroffenheit im eigenen Stack prüfen: Versionen/Komponenten abgleichen.
• Originalquelle / Hersteller-Advisory lesen: https://www.marktechpost.com/2026/08/12/allenai-open-instruct-tulu-3-post-training-with-sft-dpo-rlvr-grpo-and-verifier-based-evaluation/
• Verfügbaren Patch oder Workaround einspielen und dokumentieren.