inbound MIDI system exclusive processing

[analog604]

[30 Dec 2012 edit update]: earlier this year Liine increased MIDI in to 256 bytes which is the current Vector (array) element limit as of Lemur version 4.0. Some synths that I own, the Waldorf Pulse, AdrenaLinn FX pedals are good with only 256 and I have those working with bidirectional syste exclusive messages in templates.

Greetings !
Ok so I've been trying many tests to get Lemur process as much inbound SysEx (F0) data as possible on iPad.
Tried using iRig with direct MIDI and via Lemur Daemon. Getting basically the same results.

Sysex data receive is almost always incomplete by the time "execution: On MIDI" F0 - System Exclusive / MIDI 0 / 0-127/1 to 1
script finishes or even starts according to my debug monitors and on computer MIDI monitoring (Mac OS X 10.7./USB MIDI via AMT-8)

I can understand that sometimes this has to do with time running out by going beyond the Lemur frame rate (16ms) due to the limits of MIDI transmission speed of a particular device (synth, FX unit..) for the amount of bytes.

[edit: this was caused by a bug that comes up when the AMT-8 USB interface is used with Lemur Daemon on Mac; I can get 127 bytes now every time over Wifi with a different MIDI interface]
As mentioned on the forum in other threads we are getting different sizes of return data into MIDI_ARGS almost always incomplete if greater than about 20bytes. Maximum return I've been able to get is not much more than 50bytes.

Guys are there any tips / programming workarounds that you have been using for this to work? What are the max bytes that you have been able to receive and from what device/connection method?

I really need to be able to get edit buffer preset data from devices to set the Lemur controls.

Liine wizards, a more advanced incoming MIDI listen/receive would be greatly appreciated. Maybe something that has a timeout value and a last expected byte (F7) or array.

[edit: MIDI listen sysex may already work for greater than one 16ms Lemur frame.. but no one at Liine has confirmed]
I think what we might need is a listener capable of paying attention beyond a single frame and only executing it's script when the timeout is reached (and thereby setting an error flag) or at the end of expected bytes(s).

cheers~
Jay

[analog604]

Liine team, I offer my time and scripting abilities to beta test any extended or new system exclusive functions/object tie-ins with my various MIDI machines.

-J

[bxsj]

Funny enough I have started to work on a template to control a Roland synth via sysex. After struggling with stuff like creating the correct checksum byte for a roland device, here is what I got to work:
- bi-derictional control of patch parameters like cutoff frequency etc.
- read out specific patch parameter like parameter name
The sysex messages I used so far were around 12 to 15 bytes. So I did not experience any data loss yet.

Liine clearly stated that there is a limitation in the number of bytes transferred via sysex and the max length of arrays. So for the time being I will keep my sysex messages as short as possible. I'll keep you posted if I run into any troubles.

Btw, what is the best way of receiving sysex messages? With a general on midi expression or with midi on expression on the single objects like knobs or faders? In the first case I would have a "monster" script handling all possible different messages (script length imit!). In the latter I would have a lot of small scripts triggered by each sysex message. Any input would be highly appreciated.

Cheers,
B.

[analog604]

Hi bxsj!

So 12-15 bytes out toward the Roland. Ok.
I haven't extensively pushed the size envelope towards an instrument, but I've been ok sending small less than 15byte sysex requests to update parameters in AdrenaLinn FX pedal. I've done this for days now while debugging my latest template with no problems there.
Also outbound sysex request works requesting the buffer/patch of that and DSI Mopho and Waldorf Pulse.

The devices almost always get the message and respond back correctly at the MIDI interface layer. The responses frequently do not make it from the Lemur app (or IPad??) what I guess to be the Lemur MIDI handling data to the script layer...

So far as I know there is only one way to tie in objects to incoming MIDI and that is to create a script that runs on Execution MIDI + F0- System Exclusive. So one script will need to decode the contents of the message and run additional scripts or alter objects,
there isn't much time to do this so it needs to be as short as possible.. which of course is going to be difficult in decoding complex messages.

Here is a working (system specific to be used only on AdrenaLinn 3 FX pedal) example with lots of debugging variable info assignment.
It works as far as sometimes receiving the response to a universal sysex id request, and telling me that other receives are incomplete (example: drumbeat buffer around 50 bytes, never gets that.. but it sometimes receives enough to tell me that a drum beat was sent)



Note, global variables not shown:
targman={0x00, 0x01, 0x37}
targmodel=0x03
io.VARABLE and mio.VARIABLES from my midi i/o module, various midi related handling and debugging.

Code: Select all

decl sz=sizeof(MIDI_ARGS);
io.outchan=0x01;
if(sz>6) {
 if( !sumof(subarray(MIDI_ARGS,0,4)-{0x7E,io.outchan,0x06,0x02}) ) {//system inquiry response
  if( !sumof(subarray(MIDI_ARGS,4,7)-{targman,0x01,0x00,0x03,0x00}) ) {//Roger Linn Design Adren III
    if(sz>12) mio.inquiry={MIDI_ARGS[11],MIDI_ARGS[12],MIDI_ARGS[13]};//set to software version once a it is ID'd
    mio.mstatmsg='AdrenaLinn III resonds to SyxEx OK';
  } else { 
    mio.inquiry=0; 
    mio.mstatmsg='SysEx Inquiry response not from AdrenaLinn III!';
  }
  return;
 }
}

if( sumof(subarray(MIDI_ARGS,0,5)-{targman,0x03,0x01}) ) { mio.inquiry=-1; return; }

// do not attempt to go over current 3.0.3 array size for MIDI_ARGS
if(sz>5 && sz<127) mio.buf=subarray(MIDI_ARGS,6,sz-6);

if(MIDI_ARGS[5]==0x03){ //Rcv drumbeat
  if(sz<58) mio.mstatmsg={sz,'Incomplete Drumbeat receive!'}; 
  else mio.mstatmsg={sz,'RCV Drumbeat OK!'};
  //TODO: unpack 51 coded bytes and assign buffer.  
}
else if(MIDI_ARGS[5]==0x0B){ //msg11 edit buff
  if(sz<81) mio.mstatmsg={sz,'Incomplete Edit Buffer receive!'}; 
  else mio.mstatmsg={sz,'RCV Preset edit buffer OK!'};
  //TODO: unpack 64 coded bytes and assign buffer.  
}
else if(MIDI_ARGS[5]==0x0F){ //msg15 sys param rcv
  if(sz<58)  mio.mstatmsg={sz,'Incomplete RCV  System Paramaters'};
  else  mio.mstatmsg={sz,'RCV System Param OK!'};
  //TODO: unpack and assign buffer.  
}

Funny enough I have started to work on a template to control a Roland synth via sysex. After struggling with stuff like creating the correct checksum byte for a roland device, here is what I got to work:
- bi-derictional control of patch parameters like cutoff frequency etc.
- read out specific patch parameter like parameter name
The sysex messages I used so far were around 12 to 15 bytes. So I did not experience any data loss yet.

Liine clearly stated that there is a limitation in the number of bytes transferred via sysex and the max length of arrays. So for the time being I will keep my sysex messages as short as possible. I'll keep you posted if I run into any troubles.

Btw, what is the best way of receiving sysex messages? With a general on midi expression or with midi on expression on the single objects like knobs or faders? In the first case I would have a "monster" script handling all possible different messages (script length imit!). In the latter I would have a lot of small scripts triggered by each sysex message. Any input would be highly appreciated.

Cheers,
B.

[bxsj]

Thanks for all the information! I will run some tests with incoming sysex data to see were the limits of my setup are. Guess it's dependent on the interface setup and off course on how busy the template is.
Cheers,
B.

[analog604]

you're welcome!
please let us know how your testing goes and request message type details.
-J

Thanks for all the information! I will run some tests with incoming sysex data to see were the limits of my setup are. Guess it's dependent on the interface setup and off course on how busy the template is.
Cheers,
B.

[analog604]

I did some more testing with MIDI sysex in, this time I connected a Waldorf Blofeld directly to computer via USB and routed it via Lemur Daemon to a template.

I did a patch dump from Blofeld's front panel:
392Bytes sent at computer MIDI USB layer and the Template function received 128. Far better than 50 for some work!

[6 March Edit: After switching MIDI interfaces, I've been able to receive 70 bytes all day long when requesting presets/edit buffers.. but it is all less than 100bytes] There remains a bug somewhere between the MIDI USB layer at the computer and the Lemur Daemon that shows up when sending the data along to Lemur iPad.
This can cause random application crashing and different number of received bytes every send (always less than 50 in my case with the other MIDI interface).

cheers~
Jay

[bxsj]

Here are some results from my side.

Sysex transfer in was ok up to 126 byte. The 127th byte of the incoming message is always 0xF0 (?!). I tested this with the attached template. It's a quick and dirty tool to request sysex data from a Roland V-Synth XT. The size of the data dump can be changed via a slider. Not sure if it can be used for other gear (sysexhdr_vsynt needs to be changed and the checksum byte calculation might need some tweaking).

I did not experience any data loss in my setup. I tested midi via USB and via a midi cable). Not sure what you mean with "only standard/old midi"?

For my purposes, designing a template to tweak parameters on my v-synth, the 127 byte limit on sysex is ok.
Cheers,
B.

[analog604]

Thanks for the testing and the template tool!

I'll try some other MIDI interfaces and direct USB connect via the camera connection kit to see if I can come up with better numbers. 127 bytes would cover everything that I need to complete my Adrenalinn project.

To be more specific about my comment on standard MIDI, I meant that the older hardware transport's data rate is very slow (31.25kbps!) verses modern transports such as USB or ethernet.

[analog604]

Loaded up your template and just send data towards it for sysex in length counting, the hex display function is a nice addition.

new test: connected midiman 2x2 via USB to mac 10.7.3 with the FX pedal and I got all expected bytes up to 127.
Also connected the DSI Mopho to it and initiated patch dumps from front panel, got 127 every time. All good..
Also used a couple of my templates and these worked for reception as well of the above.

So my main emagic AMT-8 (i know, legacy!) 8-port studio MIDI interface and Lemur/Daemon do not play well together at all for system exclusive messages. Works fine for CC and notes though.

The iRig MIDI on the iPad produces inconsistent results it will cause Lemur to lock up when trying these tests but other apps can still be accessed and I have to power cycle the iPad.
So far the only instance of Lemur locking up is related to me having the iRig plugged directly in.

Haven't been able to test much with the camera connection kit yet but will do later.

I have enough to map edit buffers in/out of the AdrenaLinn now though so I'm pleased!
[edit: and this has worked out well in practice as I build the template and run it all day long during development!]
cheers~
-J