Publications

Efficient Construction of Model Family through Progressive Training Using Model Expansion

COLM2025

Foundation model / LLM

By : Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki

Spike No More: Stabilizing the Pre-training of Large Language Models

COLM2025

Foundation model / LLM

By : Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars

COLM2025

Foundation model / LLM

By : Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, Taiji Suzuki