sysex again, it´s geting complicated...

Discuss Lemur and share techniques.
Macciza
Regular
Posts: 1315
Joined: 07 Dec 2011 04:57
Location: Sydney, Australia.

Re: sysex again, it´s geting complicated...

Post by Macciza »

Hi All,
Couple of comments . . .

The screen pic of SysexFromMachinedrum is incomplete - it is nowhere near long enough and should end in F7 not 7F . . .
It does however contain the Kitname (kind of) '?EW KIT' - first char is 7f -which is 'DEL' so not sure what is going on there . . .Original position 1A = 26 . . .

Kitname should be accessible as a subarray of the sysex data
ex. in OnMIDI System exclusive script

if(subarray(MIDI_ARGS(2,4) == {0x3C, 0x02, 00, 0x52}) // match Elektron Machinedrum ID, Midi channel, and Kit dump ID . . .
kitname = subarray(MIDI_ARGS, 10, 16); // 10 = address 0A, 16 =kitnam subarray length . ..
setattribute(Text, 'content' arraytostring(kitname); // convert ascii data to string and show inText obj . . .

First 5 bytes (after F0) only are the header - 0x53 is the Kit message request command , reply 0x52 (mb mentioned this too)

Not all of the 'blue' section is encoded - only the sections marked (7 bit encoding) - Selected Drum Model, LFO Settings and Trig Groups - the rest is standard except the Kitname is ascii . . .
The 'blue' section is marked out purely to delineate data used for checksum . . .

So where does that leave us . . .

Sysex Kit Request (as per manual) - values in hex unless noted otherwise . . .
Sysex init = 'FO, 00, 20, 3C, 02, 00' - 00 at end is base channel MIDI Ch 1, may be different if MD set to other than MIDI Ch 1 . ..
Kit request = '53'
Kit number = '00 -3F' 0 to 63
Sysex end = 'F7"

so to request Kitnumber 0 - 'FO, 00, 20, 3C, 02, 00, 53, 00, F7'
MD should respond with The kit data - 'FO, 00, 20, 3C, 02, 00, 52, Kit data bytes, F7'
Kit data bytes probably includes from address 0x07 (Version) through to address 0x4CF (Message length low bit)
-: number of Kit Data bytes is 4CF - 07 = 4C8 or 1224 decimal, total message size is 0x4D0

Hope that helps
Cheers
MM
iMac 2.8G i7 12G 10.6.8/10.7.2, Legacy Dexter/Lemur, Liine Lemur/iPad2, KMI SoftStep, 12Step & QuNeo , B-Controls, Mackie C4 etc
MaxMSP, Live Suite, Native Instrument stuff, etc Modified Virtual Guitar System etc All Projects/Modules © CC-BY-NC-SA[*][/b]
mbncp
Regular
Posts: 87
Joined: 08 Dec 2011 07:25

Re: sysex again, it´s geting complicated...

Post by mbncp »

To be readable the array (data byte part)has to be converted to 8 bit first. It works this way.
The first byte holds the 8 th bit value of the next 7 bytes. So if the first byte is
1110001 and the following 7 bytes are all 1111111 then these 7 bytes are actually

11111111
11111111
11111111
01111111
01111111
01111111
11111111
showing in binary as it's easier to read.

The problem I see is the size of the data. I tried to send a huge sysex file to the Lemur and sizeof returns only 128 bytes and the OnSysex callback gets called only once. As the header is 8 byte long that's 120 bytes left. As they are encoded that's 120/8*7 = 105 data byte available. In the case of a Kit Message that gives us the kit name and parameters for about 4 tracks, all the rest is lost.

Is there a way to increase the incomming sysex buffer size ?
st8_livecontrol
Newbie
Posts: 8
Joined: 04 Oct 2010 12:26

Re: sysex again, it´s geting complicated...

Post by st8_livecontrol »

mbncp wrote:To be readable the array (data byte part)has to be converted to 8 bit first. It works this way.
The first byte holds the 8 th bit value of the next 7 bytes. So if the first byte is
1110001 and the following 7 bytes are all 1111111 then these 7 bytes are actually

11111111
11111111
11111111
01111111
01111111
01111111
11111111
showing in binary as it's easier to read.

The problem I see is the size of the data. I tried to send a huge sysex file to the Lemur and sizeof returns only 128 bytes and the OnSysex callback gets called only once. As the header is 8 byte long that's 120 bytes left. As they are encoded that's 120/8*7 = 105 data byte available. In the case of a Kit Message that gives us the kit name and parameters for about 4 tracks, all the rest is lost.

Is there a way to increase the incomming sysex buffer size ?
Looks like you've hit the limit of an array on the lemur. I'm doing some pretty extensive sysex work and i had to just send lots of little messages instead of one large one. I realise in your case you cant do this.
Macciza
Regular
Posts: 1315
Joined: 07 Dec 2011 04:57
Location: Sydney, Australia.

Re: sysex again, it´s geting complicated...

Post by Macciza »

Hi
Re: ' To be readable the array (data byte part)has to be converted to 8 bit first. ' - That is only for the 3 sub-sections mentioned earlier . . .
Only the marked 7bit encoded section will need decoding neeThe rest/most of the message is in standard 7bit range for sysex messages . . .

I understand the 7bit encoding - I think KORG use the same system but others do it differently again
If you nibblise it it makes it even more readable - at first I thought you were encoding and had it very wrong . . . .

Decoding from 7 bit encoded sysex data to bytes
Input 0111 111 then 7 0111 1111 = 8 sysex data bytes in total
convert to
1111 1111
1111 1111 etc . = 7 bytes (8bit values)

The Selected Drum Model section that he is interested in in 74 data bytes (64 bytes) long according to manual but it starts over 400 bytes into the message
So unless MIDI_ARGS can hold over 500 bytes it wont be possible until it can, then it will be easy to subarray and process the data . . .

You should certainly be able to get the KitName extracted as I mapped out earlier . . .

Cheers
MM
iMac 2.8G i7 12G 10.6.8/10.7.2, Legacy Dexter/Lemur, Liine Lemur/iPad2, KMI SoftStep, 12Step & QuNeo , B-Controls, Mackie C4 etc
MaxMSP, Live Suite, Native Instrument stuff, etc Modified Virtual Guitar System etc All Projects/Modules © CC-BY-NC-SA[*][/b]
dbk
Regular
Posts: 98
Joined: 14 Dec 2011 11:35

Re: sysex again, it´s geting complicated...

Post by dbk »

Macciza, thanks for posting the code, now i have a glimpse on how to access subarrays.

to be able to retrieve the kitname is a good start :)

i wrote the support yesternight, waiting for an answer and will post it here then.

hopefully they can extend the max size of an array, maybe this was made due to limitations of the Lemur hardware?

cheers, Martin

thanks again to everone for dropping their knowledge, very cool to learn these things.
dbk
Regular
Posts: 98
Joined: 14 Dec 2011 11:35

Re: sysex again, it´s geting complicated...

Post by dbk »

@ st8_livecontrol:

aren´t you part of the dev team here? are there any plans to implement arrays with more filesize?

please let us know.

cheers,
Martin
mbncp
Regular
Posts: 87
Joined: 08 Dec 2011 07:25

Re: sysex again, it´s geting complicated...

Post by mbncp »

Big problem is indeed the sysex message length. I didn't see anything in the MD doc that let's you ask only specific parameters. There is a function to get/set some status values but it seems limited to just a few things.

Now the options are:

1) Get the Lemur to handle much larger sysex message, which would be really cool as I would love to be able to use it somehow as a librarian.
2) Get elektron to allow single parameter query like most korg, yamaha and other brand use. But even then, 128 bytes may still be too short.
3) Last option would be to create an app that will get the sysex from the MD to a mac/pc and transfer them as small chunk to the Lemur. But although I'm currently mainly on mac I haven't done much coding so far and I have no time/interest to really go into that.

Really the best option is to have the Lemur handle larger arrays/sysex as we have plenty of ram on an iPad.

Anyway, I will still write these 2 functions for the 7-8 bit conversion, then using subarrays makes it pretty easy to extract anything.

cheers,
marc (that's the m in mbncp ;) )
dbk
Regular
Posts: 98
Joined: 14 Dec 2011 11:35

Re: sysex again, it´s geting complicated...

Post by dbk »

Hey Marc,

first option would obviously be the best, let´s see what liine comes up with.

pretty sure elektron won´t change any of this, they´re busy with the Octrtrack right now.

option 3: i thought about building a maxpatch which handles this but then i´d need a computer between the MD and Lemur, which i´d rather avoid. also i would have to learn another application and have even less time for making music then.

thanks for the functions you will write, looking very forward to it.

cheers,
martin
dbk
Regular
Posts: 98
Joined: 14 Dec 2011 11:35

Re: sysex again, it´s geting complicated...

Post by dbk »

Just got a a nice message from the support team.

there is indeed a built in limit of 127 bytes, they´re are looking into pushing this for the next release.

sounds good, hope they make it :)

cheers
LMO
Newbie
Posts: 2
Joined: 09 Jan 2012 07:30

Re: sysex again, it´s geting complicated...

Post by LMO »

Any feeling for how much the 127 byte limit may be increased? I have some rather large (well over 2k) messages to deal with.
Post Reply