sysex again, it´s geting complicated...

Hello,

the guys from elektron gave me pdf with the sysex documentation for the machinedrum.

so, i thought, i could fetch the current used machines and add a very important feature to the machinedrumcontroler, the ability to disaply the names of the tracks on the lemur.

however, it´s damn complicated and the data seems to be coded and needs to be decoded. i tried several things but didn´t come close to a solution.


i really hope one of you guys jumps in and gives me a proper hint on how to achieve my goal.

i allowed myself to attach the pdf with the informations, it´s sorta french to me ...

the infos i´d like to retrieve from the machine are on page four, especially "kitname' and 'selected drum model'.

thanks a lot in advance, ANY tip is really apprechiated.


cheers, martin

md1_53-sysex-0_8.zip

Hi
Yeah - sysex can get complicated . . .
And they certainly did not do any favours in there sysex steup - seems slightly non-standard to my eyes . .
The 8-bit encoding stuff is a bit weird but I do recall seeing it previously somewhere, and there checksum/length methods are a bit weird too . . .

Also it is usual that info is provided for 'requests' to the midi device and also for 'returned data' - I don't see any request info?

Once you sort how and when the device sends the 'Kit Message' data - whether by request or automatically from device? -
You need to capture the sysex via an 'OnMIDI - System Exclusive' script matching the MachineDrum header and Kit message data . . .
You then need a subarray of that data from position 0x0A - 16bits long = KitName - which needs to be convert with arraytostring to display as text rather than ASCII numbers

Selected Drum Model seems to be a data structure within the message -16*4byte ints - but I am not sure what the data is meant to represent . . .

Hope that helps

Cheers
MM

Hi,

I´m that far to request the current kit data (which should include all of the kit data from the pdf) and to use the Monitor to display it on the lemur.

Selected Drum Model seems to be a data structure within the message -16*4byte ints - but I am not sure what the data is meant to represent . . .

This data is meant to represent the current loaded machine on all 16 tracks, should look a lot like this:

112,4,76,86...

each of the values are the code for a "machine", there´s a sheet where you see what machine has what number, as soon as i have them numbers accessible i could write a script which outputs the name of the machine (for example: TRX-BD) compared to a number, should be easy.


how do i know what position in the sysexstring holds this values ?

thanks you, your help is very apprechiated!

cheers

Hey,

with the arraytohex(array) script posted by mbncp (thank you), i was able to get the following data, which should represent the sysexdump i needed to get.

00 20 3c 02 00 52 04 01 0B 4E 45 57 20 4B 49 54 00 30 12 00

01 is the end of the header data, according to the elektron sysex sheet 45 should hold 16 values representing my machines on the 16 tracks, if i´m right, once it´s 'unpacked', so they say in the manual.

so, assuming i´m right, how to "unpack" this data?

again, more than thankfull for any hints, code, whatever could help me

cheers,
martin

I think that they don't want you to fool with sysex, so they made it as difficult as possible

Well, first you need to convert the blue part to 8 bits as it is coded in 7 bits. Most machines work with 8 bits (or 16,32,..) but for the MIDI spec all values larger than 7 bits are status byte, so you can't use them. The trick is to convert to 7 bits and back to 8 by using an extra byte that has the 8th bit value for the 7 following bytes.
I think bitwise operators are working on the lemur so that should be ok, so a function script could handle that.

But the problem I see here is the length of the data, I'm not even sure that the Lemur receives or can handle such big sysex message. Remember you figured out that the largest array size is 255 bytes. Are you receiving multiple arrays ?

And how do you get the data, is there some parameter request function ? As it doesn't seem documented on that pdf. Or do you have to trigger some function on the MD ?
On Korg machines you can query a single parameter which would be easier to deal with. Don't know about the MD.

hey, thanks for joining.

Well, first you need to convert the blue part to 8 bits as it is coded in 7 bits. Most machines work with 8 bits (or 16,32,..) but for the MIDI spec all values larger than 7 bits are status byte, so you can't use them. The trick is to convert to 7 bits and back to 8 by using an extra byte that has the 8th bit value for the 7 following bytes.
I think bitwise operators are working on the lemur so that should be ok, so a function script could handle that.

oh, i think that´s what they reffering to on page one, how the 7 bit encoding is generated?

But the problem I see here is the length of the data, I'm not even sure that the Lemur receives or can handle such big sysex message. Remember you figured out that the largest array size is 255 bytes. Are you receiving multiple arrays ?

i´m not sure either, that´s what i´ve read on the web. i will contact the support and ask for clarification on this. would be a major bummer. mail sent.

And how do you get the data, is there some parameter request function ? As it doesn't seem documented on that pdf. Or do you have to trigger some function on the MD ?

i send, what they call, a kit request. it´s a sysey string where i also send the number of the kit i want to have the data from. once i´ve send the request the machinedrum responds with an array of roughly 30 values in an array.

there´s actually an other option called kit sysex dump.

after 0x52 it requests Kit Data Bytes to be sent, any idea what that could be? maybe that´s how we could query single values?

cheers,
martin

edit: i attached a screenshot from the sysexpart in the manual.

sysexscreen.zip

i send, what they call, a kit request. it´s a sysey string where i also send the number of the kit i want to have the data from. once i´ve send the request the machinedrum responds with an array of roughly 30 values in an array.

there´s actually an other option called kit sysex dump.

after 0x52 it requests Kit Data Bytes to be sent, any idea what that could be? maybe that´s how we could query single values?

Do you have the doc concerning these request sys messages ?

Don't know what they mean by after, but 0x52 seems to be the Kit Message, declared in the header.

Subject: Re: sysex again, it´s getting complicated...

Don't know what they mean by after, but 0x52 seems to be the Kit Message, declared in the header.

you´re right! Sadly i don´t have any more documents then the manual with all available sysex commands and the pdf attached in my first post

i tried to send the 0x52 command, the machinedrum receives but doesn´t send back anything then.

how would you go from here?

edit: added a picture of the lemur with the values sent from the MD

Screen.PNG.zip

Ok, got it, all sysex commands are listed in the manual in appendix C. Kit dump command is 0x53 and it replies with 0x52.

Anyway, I will go thru that doc to see what's available, create the 7to8bits and 8to7 bits functions on the lemur and some helpers to extract those data.

As I don't have an MD I will probably need you to send me some stuff. Also it would be good to download a sysex librarian so you can send me the files I need to test.
As I think that you're on mac there is this one : http://www.snoize.com/SysExLibrarian/, should be fairly simple to use.

Once the functions are working, I'll post them here for reference, but in the meantime, I think it's better that we discuss and test this stuff off forum.
I'll pm you when I need something, just be aware that it may take a day or two as I'm a bit busy right now, and pray that I'm done before I get my tempest and little phatty

man, this is awesome!

i´ll supply whatever you need, just let me know and thanks for spending your time with this.

you sure will have some fun with your new toys

cheers,
martin

Hi All,
Couple of comments . . .

The screen pic of SysexFromMachinedrum is incomplete - it is nowhere near long enough and should end in F7 not 7F . . .
It does however contain the Kitname (kind of) '?EW KIT' - first char is 7f -which is 'DEL' so not sure what is going on there . . .Original position 1A = 26 . . .

Kitname should be accessible as a subarray of the sysex data
ex. in OnMIDI System exclusive script

if(subarray(MIDI_ARGS(2,4) == {0x3C, 0x02, 00, 0x52}) // match Elektron Machinedrum ID, Midi channel, and Kit dump ID . . .
kitname = subarray(MIDI_ARGS, 10, 16); // 10 = address 0A, 16 =kitnam subarray length . ..
setattribute(Text, 'content' arraytostring(kitname); // convert ascii data to string and show inText obj . . .

First 5 bytes (after F0) only are the header - 0x53 is the Kit message request command , reply 0x52 (mb mentioned this too)

Not all of the 'blue' section is encoded - only the sections marked (7 bit encoding) - Selected Drum Model, LFO Settings and Trig Groups - the rest is standard except the Kitname is ascii . . .
The 'blue' section is marked out purely to delineate data used for checksum . . .

So where does that leave us . . .

Sysex Kit Request (as per manual) - values in hex unless noted otherwise . . .
Sysex init = 'FO, 00, 20, 3C, 02, 00' - 00 at end is base channel MIDI Ch 1, may be different if MD set to other than MIDI Ch 1 . ..
Kit request = '53'
Kit number = '00 -3F' 0 to 63
Sysex end = 'F7"

so to request Kitnumber 0 - 'FO, 00, 20, 3C, 02, 00, 53, 00, F7'
MD should respond with The kit data - 'FO, 00, 20, 3C, 02, 00, 52, Kit data bytes, F7'
Kit data bytes probably includes from address 0x07 (Version) through to address 0x4CF (Message length low bit)
-: number of Kit Data bytes is 4CF - 07 = 4C8 or 1224 decimal, total message size is 0x4D0

Hope that helps
Cheers
MM

To be readable the array (data byte part)has to be converted to 8 bit first. It works this way.
The first byte holds the 8 th bit value of the next 7 bytes. So if the first byte is
1110001 and the following 7 bytes are all 1111111 then these 7 bytes are actually

11111111
11111111
11111111
01111111
01111111
01111111
11111111
showing in binary as it's easier to read.

The problem I see is the size of the data. I tried to send a huge sysex file to the Lemur and sizeof returns only 128 bytes and the OnSysex callback gets called only once. As the header is 8 byte long that's 120 bytes left. As they are encoded that's 120/8*7 = 105 data byte available. In the case of a Kit Message that gives us the kit name and parameters for about 4 tracks, all the rest is lost.

Is there a way to increase the incomming sysex buffer size ?

mbncp wrote:To be readable the array (data byte part)has to be converted to 8 bit first. It works this way.
The first byte holds the 8 th bit value of the next 7 bytes. So if the first byte is
1110001 and the following 7 bytes are all 1111111 then these 7 bytes are actually

11111111
11111111
11111111
01111111
01111111
01111111
11111111
showing in binary as it's easier to read.

The problem I see is the size of the data. I tried to send a huge sysex file to the Lemur and sizeof returns only 128 bytes and the OnSysex callback gets called only once. As the header is 8 byte long that's 120 bytes left. As they are encoded that's 120/8*7 = 105 data byte available. In the case of a Kit Message that gives us the kit name and parameters for about 4 tracks, all the rest is lost.

Is there a way to increase the incomming sysex buffer size ?

Looks like you've hit the limit of an array on the lemur. I'm doing some pretty extensive sysex work and i had to just send lots of little messages instead of one large one. I realise in your case you cant do this.

Hi
Re: ' To be readable the array (data byte part)has to be converted to 8 bit first. ' - That is only for the 3 sub-sections mentioned earlier . . .
Only the marked 7bit encoded section will need decoding neeThe rest/most of the message is in standard 7bit range for sysex messages . . .

I understand the 7bit encoding - I think KORG use the same system but others do it differently again
If you nibblise it it makes it even more readable - at first I thought you were encoding and had it very wrong . . . .

Decoding from 7 bit encoded sysex data to bytes
Input 0111 111 then 7 0111 1111 = 8 sysex data bytes in total
convert to
1111 1111
1111 1111 etc . = 7 bytes (8bit values)

The Selected Drum Model section that he is interested in in 74 data bytes (64 bytes) long according to manual but it starts over 400 bytes into the message
So unless MIDI_ARGS can hold over 500 bytes it wont be possible until it can, then it will be easy to subarray and process the data . . .

You should certainly be able to get the KitName extracted as I mapped out earlier . . .

Cheers
MM

Macciza, thanks for posting the code, now i have a glimpse on how to access subarrays.

to be able to retrieve the kitname is a good start

i wrote the support yesternight, waiting for an answer and will post it here then.

hopefully they can extend the max size of an array, maybe this was made due to limitations of the Lemur hardware?

cheers, Martin

thanks again to everone for dropping their knowledge, very cool to learn these things.

@ st8_livecontrol:

aren´t you part of the dev team here? are there any plans to implement arrays with more filesize?

please let us know.

cheers,
Martin

Big problem is indeed the sysex message length. I didn't see anything in the MD doc that let's you ask only specific parameters. There is a function to get/set some status values but it seems limited to just a few things.

Now the options are:

1) Get the Lemur to handle much larger sysex message, which would be really cool as I would love to be able to use it somehow as a librarian.
2) Get elektron to allow single parameter query like most korg, yamaha and other brand use. But even then, 128 bytes may still be too short.
3) Last option would be to create an app that will get the sysex from the MD to a mac/pc and transfer them as small chunk to the Lemur. But although I'm currently mainly on mac I haven't done much coding so far and I have no time/interest to really go into that.

Really the best option is to have the Lemur handle larger arrays/sysex as we have plenty of ram on an iPad.

Anyway, I will still write these 2 functions for the 7-8 bit conversion, then using subarrays makes it pretty easy to extract anything.

cheers,
marc (that's the m in mbncp )

Hey Marc,

first option would obviously be the best, let´s see what liine comes up with.

pretty sure elektron won´t change any of this, they´re busy with the Octrtrack right now.

option 3: i thought about building a maxpatch which handles this but then i´d need a computer between the MD and Lemur, which i´d rather avoid. also i would have to learn another application and have even less time for making music then.

thanks for the functions you will write, looking very forward to it.

cheers,
martin

Just got a a nice message from the support team.

there is indeed a built in limit of 127 bytes, they´re are looking into pushing this for the next release.

sounds good, hope they make it

cheers

Any feeling for how much the 127 byte limit may be increased? I have some rather large (well over 2k) messages to deal with.