rogerdickey 8 minutes ago

all the examples seem to be classical music or largely instrumental stuff. how well does it work for things like pop music, rap?

peaseagee 2 hours ago

Congrats on training on open and public domain recordings. Thanks for proving this kind of work can be done in a more ethical way.

  • piltdownman 2 hours ago

    TBF it's by far the best corpus for this use case anyway.

    Classical Music as an academic concern is very pedantic in the best way, and has the lions share of the research hours and funding compared to any other genre. As a result the best 1:1 approximations of multi-instrument Audio into multi-channel MIDI files is done simply by OCR of the established Gesamtausgabe in each case. Computer based scoring and playback has been solid since Finale/Sibelius/Score came on the market, and democratized at a wide consumer level somewhere between Amiga mod-trackers and Macintosh Quicktime 2.0.

epiccoleman 1 hour ago

That is awesome! I'll definitely have to check it out. I was very impressed with muscriptor when I used it in a recent project (a conference talk I gave). It was an AI talk with a musical component, and the final "scene" in the talk involved my own guitar playing. I used Muscriptor to transcribe that playing for categorization as part of a "guitar solo generator" that recut a new solo out of my stock phrases each time. Really fun little demo for a bigger project.

Anyway, it would be awesome to be able to do those sorts of transcriptions better and faster - so I'll definitely give this a whirl. Maybe I can dig up the pipeline scripts from processing my licks for that talk and see how dettato slots in.

gpugreg 1 hour ago

Nice! Worked surprisingly well.

You might want to implement deduplication for your website so impatient users do not convert the same mp3 multiple times (at multiple times the price) when they click "run audio to midi" and the server takes a bit longer to respond.

  • cocodill 11 minutes ago

    I guess impatient users could run audio to midi local.

oever 2 hours ago

Audio-to-MIDI sounds very ambitious. How do you verify that the midi fits the audio? Does it result in useful beat detection? E.g. I'd very much like to have a way to map audio offset in a recording to labels in sheet music. But even beat detection in fairly standard jazz recordings fails often.

Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?

  • gunalx 56 minutes ago

    Its definetly a problem where generating the trivial version of synthetic data is easy though.

eduardoagarcia 42 minutes ago

super interesting, I'll have to check this out tonight when I get home!

jsphweid 2 hours ago

I've been out of the loop since Magenta's MT3 model. This feels a lot better and significantly smaller. Very impressive, dude. And thank you!

gregzem 2 days ago

Congrats with SOTA model. How good is it in recognition of the instruments which are overlapping by frequencies? Does is provide note offset and velocity in the transcribed MIDI? Thanks!

ch_sm 1 hour ago

Just tried it with some home-recorded songs (pop/rock) — this works great!

mycall 2 hours ago

I'm still waiting for someone to convert beatboxing to midi in realtime. That would be a game changer for that art.

  • maxgraf96 2 hours ago

    You should check out https://vochlea.com/ :)

    • mycall 1 hour ago

      > Trained on your own percussive sounds

      This is probably where I would find it limiting range of sounds can be huge, much more than just 6 sounds (10000+) and not all of them percussive. Worth an eval though, thanks!

hongyong 50 minutes ago

nice, this is what i've been looking for

awinter-py 44 minutes ago

at long last we have solved karaoke

JodieBenitez 2 hours ago

I need this in Bitwig !

  • thatmf 6 minutes ago

    my exact thought. hmm. weekend project?

pikann22 2 hours ago

crazy how a 20m param model actually beats the 1,3b muscriptor. awesome work!

IAmGraydon 1 hour ago

Excellent! I'll try this tonight. The timing couldn't be better as I've been researching audio->MIDI models for some personal projects recently.