Kyutai Releases MuScriptor: An Open-Weight Decoder-Solely Transformer for Multi-Instrument Music Transcription to MIDI

Computerized Music Transcription (AMT) converts an audio recording into symbolic notes, normally MIDI. Single-instrument transcription already works moderately properly. Nonetheless, transcribing a full multi-instrument combine stays troublesome. Kyutai and Mirelo crew now launch MuScriptor to shut that hole. It’s an open-weight mannequin educated on actual, multi-instrument recordings throughout many genres.

This text explains how MuScriptor works, what the benchmarks present, and the right way to run it.

What’s MuScriptor?

At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a brief audio section. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In impact, transcription turns into a language-modeling process, following the MT3 tokenization scheme.

The discharge ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and massive (1.4B). The inference code makes use of the MIT license. The weights use CC BY-NC 4.0, so industrial use is restricted.

How the Three-Stage Pipeline Works

MuScriptor’s major concept is knowledge, not structure. Accordingly, coaching strikes by means of three phases, and every builds on the final.

1. Pre-training makes use of D_Synth, roughly 1.45M MIDI information. An on-the-fly pipeline synthesizes them throughout coaching. Augmentations embody pitch shifting, tempo adjustments, velocity adjustment, and instrument randomization. Over 250 soundfonts plus random detuning yield near-infinite audio realizations.
2. Advantageous-tuning makes use of D_Actual, an inside set of 170,000 recordings. Collectively they whole greater than 11,000 hours with aligned word annotations. Most alignments come from audio-symbolic synchronization utilizing interpolation and dynamic time warping. Poor pairs are filtered by warping distance and a most time-dilation issue.
3. Reinforcement studying post-training makes use of D_RL, 300 manually verified tracks. The crew applies a GRPO-like methodology combining REINFORCE with group-relative benefit normalization. The reward sums three F-scores: onset, body, and offset. In consequence, the mannequin learns to favor cleaner transcriptions.

Efficiency

For analysis, the analysis crew use D_{Take a look at}, 372 held-out tracks with correct annotations. They report instrument-agnostic metrics from the mir_eval library. Amongst them, Multi F1 is strictest, because it additionally requires the proper instrument.

The desk under traces every coaching stage towards the YourMT3+ baseline, utilizing the big (~1.3B) mannequin.

Clearly, each stage improves outcomes, and actual knowledge issues most. Artificial-only coaching reaches aggressive body F1 however weak onset and multi scores. Including D_Actual then lifts all metrics by roughly 20 factors. Lastly, RL post-training reduces false negatives and sharpens onset timing.

Cross-dataset assessments level the identical approach. For instance, body F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset keep decrease on exhausting types like chorals.

Getting Began

Set up takes one command, and inference streams word occasions straight.

# pip set up muscriptor   (or: uv add muscriptor)
from pathlib import Path
from muscriptor import TranscriptionModel

# Downloads the default "medium" variant (additionally accepts "small" / "massive")
mannequin = TranscriptionModel.load_model()

# Stream word occasions; optionally situation on identified devices
for occasion in mannequin.transcribe("audio.wav", devices=["acoustic_piano", "drums"]):
    print(occasion)   # NoteStartEvent / NoteEndEvent / ProgressEvent

# Or write a MIDI file straight
Path("out.mid").write_bytes(mannequin.transcribe_to_midi("audio.wav"))

For the launched fashions, preserve cfg_coef at 1, since they’re already RL post-trained. Moreover, uvx muscriptor serve launches a browser internet UI with a reside piano roll.

Use Circumstances with Examples

As a result of the output is customary MIDI, many workflows open up:

• Producers can extract a MIDI bassline from a mixture, then re-voice it in a DAW.
• Musicologists can convert historic recordings into editable scores for evaluation.
• MIR researchers can feed transcriptions into chord or key recognition programs.
• Educators can construct follow instruments exhibiting a reside piano roll throughout playback.
• Builders can transcribe solely drums by passing instrument conditioning.

Strengths and Weaknesses

Strengths:

• Skilled on 170k actual recordings spanning classical to heavy metallic.
• Open weights plus MIT-licensed inference code, in three dimension variants.
• Multi F1 of 48.2 versus 21.9 for the YourMT3+ baseline on D_{Take a look at}.
• Instrument conditioning customizes output and stabilizes cross-segment predictions.
• A streaming API emits word occasions and MIDI, alongside a browser internet UI.

Weaknesses:

• Weights are CC BY-NC 4.0, so industrial deployment is restricted.
• The tokenizer drops velocity and can’t characterize overlapping same-pitch, same-instrument notes.
• Onset and offset accuracy keep decrease on chorals and comparable types.
• The massive mannequin desires a GPU for sensible velocity.
• The 5-second section dimension limits long-range context and inference velocity.

Take a look at the Paper, GitHub Repo and Mannequin Weights. Additionally, be happy to comply with us on Twitter and don’t overlook to hitch our 150k+ML SubReddit and Subscribe to our E-newsletter. Wait! are you on telegram? now you possibly can be a part of us on telegram as properly.

Must associate with us for selling your GitHub Repo OR Hugging Face Web page OR Product Launch OR Webinar and many others.? Join with us