Kyutai and Mirelo Open-Source MuScriptor, a Transformer That Turns Raw Audio Into MIDI

Kyutai has teamed up with Mirelo to release MuScriptor, an open-weight decoder-only Transformer built specifically for multi-instrument music transcription, converting recorded audio directly into MIDI. The model is now available on Hugging Face in three sizes: a 103M-parameter small variant, a 307M-parameter medium version set as the default, and a 1.4B-parameter large model aimed at maximum accuracy.

At its core, MuScriptor works by reading mel-spectrograms extracted from short audio clips and autoregressively generating MIDI-like tokens that encode pitch, timing, and instrument identity. This follows the MT3 tokenization scheme, a format already established in prior transcription research, which lets the model output structured note events rather than raw waveforms or piano-roll images.

Benchmark Gains Over YourMT3+

The headline result comes from the large variant, which posts a Multi F1 score of 48.2 on the D_Test benchmark, a held-out set of 372 tracks. That more than doubles the 21.9 score recorded by YourMT3+, previously one of the strongest open baselines for multi-instrument transcription. The gap suggests MuScriptor isn't just an incremental tuning exercise but a meaningful jump in how well a single model can separate and notate multiple instruments playing simultaneously, a task that has historically been difficult because errors in pitch, timing, and instrument classification tend to compound.

A Three-Stage Training Pipeline

The performance gains trace back to a deliberately staged training process. MuScriptor first pre-trains on D_Synth, a synthetic corpus of roughly 1.45 million MIDI files, giving the model broad exposure to note patterns before it ever hears real recordings. It's then fine-tuned on D_Real, a dataset of 170,000 recordings totaling more than 11,000 hours, each paired with aligned note-level annotations grounded in actual audio.

The final stage adds reinforcement learning on D_RL, a smaller set of 300 manually verified tracks. Kyutai's team used a GRPO-style approach, combining REINFORCE with group-relative advantage normalization, where the reward signal sums three separate F-scores measuring onset, frame, and offset accuracy. This RL pass is designed to sharpen exactly the kind of timing precision that raw supervised fine-tuning tends to leave on the table.

Licensing Split Between Code and Weights

Kyutai and Mirelo are drawing a clear line between the inference code and the model weights. The code ships under the permissive MIT license, meaning developers can freely integrate, modify, and redistribute the inference pipeline. The weights themselves, however, are released under CC BY-NC 4.0, which blocks commercial use without separate licensing. That split lets researchers and hobbyists experiment freely while keeping commercial deployment of the pretrained models under Kyutai and Mirelo's control, a common pattern among labs balancing open research with future monetization plans.

For a task that's long relied on either expensive proprietary transcription software or open models with noticeably weaker multi-instrument accuracy, MuScriptor's benchmark lead over YourMT3+ gives independent developers and audio researchers a substantially stronger open starting point to build on.