New serial code induces instability? SOLVED

Hamburger · Post by **Hamburger** » Fri Dec 09, 2011 4:12 pm

On my test copter I now have a FreeIMUv035MS. Connected to 5V. It has 3.3V regulator for sensors but no LLC. the sensorboard's pullups are activated.) This gives 0 i2c errors and much smoother sensor readings than with the previous sensor. To me that is the best I have ever seen on this copter. ( I know I should use LLC but that is different story)

For software I use Alex r411 (that is v1.9p2 with Serial fixes for Arduino IDE 1.00) together with some mods to strings to reduce memory usage. (as in my Hamburger branch on google rep)

This combo works kindof stable in acro while holding in hand and connected via usb cable to GUI . But when I remove usb and activate lcd telemetry over BT, I can feel the copter becoming jerky. So I am assuming the Serial handling does somehow influence the main loop or the motor control. telemetry on -> jerky, telemetry off -> stable.
With telemetry data are sent only approx. every 0.1s (10 times per second). That is different from using GUI, which constantly requests data.

to investigate further, I did add a reset to the cycle time min/max variables whenever the telemetry page 'C' (displays cycle time current/min/max) is activated. And every time I do that I can see the max. cycle time jumping to ~45.000 ! The current cycle time usually is around 2.800. Since the only thng different with LCD telemetry active is serial data transmission I am assuming the Serial code (interrupts?) does somehow influence main loop behaviour and/or i2c sensor communication and/or motor output generation.

Ideas how to investigate further

Niclas Hedlund · Post by **Niclas Hedlund** » Fri Dec 09, 2011 7:11 pm

I have experienced exactly the same thing.
I used the PC GUI over BT-link and my quad was very jerky. I thought it was due to i2c errors again, but it could be the Serial data then.
The search continues!

Hamburger · Post by **Hamburger** » Fri Dec 09, 2011 7:23 pm

Hm,
I am pretty sure it is not related to the sram memory issue -

Program: 25346 bytes
(.text + .data + .bootloader)

Data: 1694 bytes
(.data + .bss + .noinit)

If you use the GUI over Serial-BT, then that should give a rather constant Serial load. That contrdicts my theory of the periodic (10Hz) telemetry transmission. But it could well be the Serial still. Just curious why this does not happen when using the usb cable as a serial link.

mr.rc-cam · Post by **mr.rc-cam** » Fri Dec 09, 2011 7:41 pm

It certainly could be that the serial and servo interrupts are competing for time critical CPU access. I suggest you poke a stick at this to confirm the issue is serial activity related.

There is a LCD_TELEMETRY define in config.h that sets the 10Hz LCD refresh rate. Increase this to achieve a 1Hz rate and see if the twitching is less. If the timer's value impacts the twitches then that will confirm your suspicions.

BTW, is the BT connected LCD soft serial or hardware serial? If it is soft serial then I don't think the LCD's serial's activity will be the issue. But you never know till you know.

- Thomas

Hamburger · Post by **Hamburger** » Fri Dec 09, 2011 7:49 pm

very good idea. Will do tomorrow.

ziss_dm · Post by **ziss_dm** » Fri Dec 09, 2011 9:24 pm

Hi Hamburger,

I think, this is because currently sending can block in case ring buffer is full. Try to increase ring buffer size.

regards,
ziss_dm

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 5:15 pm

thanks for your ongoing support.

I can report some progress: I set

Code: Select all

#define LCD_TELEMETRY 4100917

that is 4 second interval for serial transmission and now I can feel each transmission as a single disturbance in motors. It conincides exactly with the display update on the LCD!

So I take it as proven that the serial transmission is somehow responsible for the disturbance. This also matches my other finding that upon activating of telemetry transmission (press button on LCD) the first cycle time measured is ~45.000us instead of the average 2.800us. Maybe some buffer has to be taken care of initially?

Next step will be to proceed as suggested by ziss_dm

Try to increase ring buffer size

So I will have to figure out how to do this.

mr.rc-cam · Post by **mr.rc-cam** » Sat Dec 10, 2011 6:16 pm

The only serial ring buffer I can think of is RX_BUFFER_SIZE (128 bytes) for receiving hardware based serial data. Maybe that is what ziss_dm means?

While you work with ziss_dm on that, I'm off looking in a different direction as a backup plan. A few more questions before I toss more troubleshooting ideas at you:
1. Is your BT LCD interface using 9600 baud soft serial? If not, then describe the MWC serial method you are using.
2. Is it two lines of text? {2 lines = LOG_VALUES enabled}
3. Does the glitching problem seem to improve if you disable LOG_VALUES?
4. Does the glitching problem seem to improve if you disable LOG_VALUES *and* POWERMETER?

- Thomas

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 6:22 pm

The arduino's ring buffer sizes are fixed to rx=64,tx=128 bytes. I think I found the buffer definitions in Alex' own interrupt driven implementation in Serial.ino.

Code: Select all

#define SERIAL_RX_BUFFER_SIZE 64
#define SERIAL_TX_BUFFER_SIZE 128

Well, I tried both increasing and decreasing these buffer sizes. Both not with the desired result. I do not fully understand Alex' implementation but I think the buffer gets written via the interrupt magic in one big chunk.

Maybe this needs to be split in tiny portions so as not to distract cpu for a big blob of time in one piece? (once upon a time I experienced something similar with a slow pc+audio hardware. Increasing buffers made output ever more jittery, fix was to continuously emptying the (smaller) buffer in tiny portions in frequent steps)

anyway I am out of my depth here - can anyoe take over, please?

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 6:44 pm

I tried to limit the buffer sizes down to 4/8 bytes - still no difference.

mr.rc-cam wrote:A few more questions before I toss more troubleshooting ideas at you:
1. Is your BT LCD interface using 9600 baud soft serial? If not, then describe the MWC serial method you are using.
2. Is it two lines of text? {2 lines = LOG_VALUES enabled}
3. Does the glitching problem seem to improve if you disable LOG_VALUES?
4. Does the glitching problem seem to improve if you disable LOG_VALUES *and* POWERMETER?

Thomas,
good points.
1.
no arduino-BT<~~~>BT-LCD needed to experiment this. I can run the copter with cable to computer, in Arduino IDE's terminal at 115k enter 'C' while copter is active in acro with motors running almost at hover speed (keep it in hand tightly). Each time output shows in terminal at same time you feel glitch in copter!
Alternatively, yes my BT is connected at 115k to arduino serial.

2.
yes, that gives two lines of output (on arduino IDE terminal no CRLF visible, only 32 chars).

3. & 4.
my code basis for tests is the r412 code as in the google rep in my branch http://code.google.com/p/multiwii/sourc ... n412&r=412
I currently test with
#undef FAILSAFE
#undef VBAT
#undef POWERMETER
#undef LCD_TELEMETRY_AUTO

and have set
LCD_CONF
LCD_TEXTSTAR
LCD_TELEMETRY 4100917 // ~ 4seconds
LOG_VALUES

to be safe on the memory side I have also removed the memory hogging logMotorsPower() in Output.ino. My memory footprint is down to 23056/1500 bytes.

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 6:55 pm

not sure, does Alex' implementation still rely on arduino's predefined internal RX/TX buffers (64/128 bytes)?
I found an old thread on that handmade serial topic here http://www.arduino.cc/cgi-bin/yabb2/YaB ... 223363/all, maybe it will help somehow.

mr.rc-cam · Post by **mr.rc-cam** » Sat Dec 10, 2011 7:08 pm

Too bad you are using hardware serial methods. That kills another suspicion I had about the problem.
1. If you were using serial soft 9600 mode then I would have had some things to try.
2. Using two lines would have endorsed my suspicions about a serial soft being involved in the problem.
3. LOG_VALUES will turn on LCD line 2, which means more chars need to be sent.
4. I was not at all concerned about excess ram/memory usage. Instead, I was concerned about the number of char's send with 9600 baud serial soft. Why? The more text that is sent the more it stalls the system's cycle time.

The arduino's ring buffer sizes are fixed to rx=64,tx=128 bytes. I think I found the buffer definitions in Alex' own interrupt driven implementation in Serial.ino.

That explains some of my confusion. I'm using the source from V1.9 and I don't see both of these buffers in it. All I can find is a 128 byte Rx buffer; If yours is currently 64 bytes then changing it to 128 would be more like the old way.

I'm not going to be of any further help since I'm not using the new V1.0 IDE or the revised MCW V1.9p2 dev code. But as a parting comment, I recall that some interrupt related changes were made with the dev release to allow compatibility with the newest Arduino IDE/compiler. For comparison, it might be useful to try the Nov-12 V1.9 (with your ram diet measures) and compile it on the old 022 IDE. If it works better then you could use the old V1.9 code to compare against the latest V1.9p2 dev changes. You could also try the new dev release on the old compiler. Experiments like these will help narrow things down.

not sure, does Alex' implementation still rely on arduino's predefined internal RX/TX buffers (64/128 bytes)?

Doubtful since you reported seeing similar buffers in the latest MWC source code. This may be part of the fix to use the new Arduino IDE, but I'm of no use about confirming anything since I'm still using the old V1.9 and the old trusty IDE. When you brave souls get all the new stuff to work I will then use it too. So good luck!

- Thomas

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 8:09 pm

just to verify I downloaded the v1.9 zip (Nov.12) again, applied my config and removed the large ampere array and tested again, now using IDE-022.
- feels way smoother in acro in hand even with stock settings
- no glitch in motor upon every 4 seconds' write of telemetry page C via serial to terminal at 115k.

So the cause of the glitches (either in sensor read or motor output) is now very likely related to the serial handling.

Hamburger · Post by **Hamburger** » Sat Dec 10, 2011 8:32 pm

Hamburger wrote:not sure, does Alex' implementation still rely on arduino's predefined internal RX/TX buffers (64/128 bytes)?

ah, it seems it does not.
a. the SerialWrite() implements a ring buffer (sizes are again 64/128 for rx/tx)
b. actual writing is done via interrupt by writing one char at a time (into UDR0). (why writing a char into UDR0 is equal to sending serial, is beyond me)

That contradicts my original theory of better using a steady trickling stream (if there is no other buffer beyond this UDR0 thing). Maybe it is just too many interrupts? Maybe offloading all pending chars upon one interrupt would be better, if possible?

mr.rc-cam · Post by **mr.rc-cam** » Sat Dec 10, 2011 8:55 pm

why writing a char into UDR0 is equal to sending serial, is beyond me

UDR0 is the CPU's hardware UART I/O register. Any char put there gets serial sent via the chip's TxD pin. As soon as it is empty (time depends on baud rate) you can put the next char into it. You do this over and over again, until your Tx data is all sent. In typical applications all the chars to send are sitting in a data buffer (of our making) and the mundane work of sending each char is handled by the hardware interrupts as a background task.

- Thomas

mr.rc-cam · Post by **mr.rc-cam** » Sun Dec 11, 2011 1:48 am

I downloaded the new serial.ino file and saw the blocking state issue that ziss_dm mentioned. If you want to see if this is involved with the problem then here is some replacement code for SerialWrite():

Code: Select all

void SerialWrite(uint8_t port,uint8_t c){
  uint8_t i = (serialHeadTX[port] + 1) % SERIAL_TX_BUFFER_SIZE;
  while (i == serialTailTX[port]) {
     i2c_errors_count++; // We borrow the I2C error count var because it is convenient for the test. Uses debug2 in the GUI.
  }
  serialBufferTX[serialHeadTX[port]][port] = c;
  serialHeadTX[port] = i;
  switch (port) {
    case 0: UCSR0B |= (1<<UDRIE0); break;      /* Enable transmitter UDRE interrupt */
    #if defined(MEGA)
    case 1: UCSR1B |= (1<<UDRIE1); break;
    case 2: UCSR2B |= (1<<UDRIE2); break;
    case 3: UCSR3B |= (1<<UDRIE3); break;
    #endif
  }

This is very primitive code. The debug2 value will change (ignore the count, just observe if it changes) every time the Tx buffer becomes full. I don't expect a full buffer while sending the LCD's ~35 chars. It should be able to remain empty if the serial code is working correctly. Untested, so keep that in mind.

- Thomas

Hamburger · Post by **Hamburger** » Sun Dec 11, 2011 8:51 pm

mr.rc-cam wrote:This is very primitive code. The debug2 value will change (ignore the count, just observe if it changes) every time the Tx buffer becomes full. I don't expect a full buffer while sending the LCD's ~35 chars. It should be able to remain empty if the serial code is working correctly. Untested, so keep that in mind.

Thomas,
I am glad you are still with me. Just to be sure I checked again this disturbance upon Serial transmission happens with both Arduino-022 and Arduio-100.

Then I applied your simple experiment. When connected to the GUI, right after hitting Start, the counter skyrocketed together with the first few graph items showing, then stops at some value. I could repeat this only once, on all other occasions it stays at zero. So this is not easy to interpret. The jumps I saw may be due to some initialization thing and not be really relevant.

Hamburger · Post by **Hamburger** » Sun Dec 11, 2011 9:18 pm

well, I am lost.
Now I use Arduino-022, Alex' version with the new serial, reduced memory consumption and replaced his implementation with this

Code: Select all

void SerialOpen(uint8_t port, uint32_t baud) { Serial.begin(baud); }
void SerialEnd(uint8_t port) { Serial.end(); }
uint8_t SerialRead(uint8_t port) {return Serial.read() ; }
uint8_t SerialAvailable(uint8_t port) {  return Serial.available(); }
void SerialWrite(uint8_t port,uint8_t c){ Serial.write(c); }

which should basically give us back the old style arduino serial code.
Tested again with

Code: Select all

telemetry='Z'

connect to the IDE's Serial monitor and -
same effect - can feel jumpy copter on each push of serial data every 4 seconds! If it was as simple as I thought this must not happen!

I can only say it again - I am lost for now and will take a break and get some sleep.

Alexinparis · Post by **Alexinparis** » Mon Dec 12, 2011 12:13 am

Hi,

Could you try to replace this function in the last code: (just this mod, nothing more)

Code: Select all

void SerialWrite(uint8_t port,uint8_t c){
  while (!(UCSR0A & (1 << UDRIE0))) ;
  UDR0 = c;
}

This should give the same behavior as what was done in previous versions of Arduino IDE.
ie a non interrupt based Serial.write().

It's still ok with the GUI, even if there was a special trick to use interrupts for it.

If it's fine for you, there's maybe no need to switch to an interrupt based method to transmit serial data.

Hamburger wrote:well, I am lost.
Now I use Arduino-022, Alex' version with the new serial, reduced memory consumption and replaced his implementation with this
Code: Select all
void SerialOpen(uint8_t port, uint32_t baud) { Serial.begin(baud); }
void SerialEnd(uint8_t port) { Serial.end(); }
uint8_t SerialRead(uint8_t port) {return Serial.read() ; }
uint8_t SerialAvailable(uint8_t port) {  return Serial.available(); }
void SerialWrite(uint8_t port,uint8_t c){ Serial.write(c); } 
which should basically give us back the old style arduino serial code.
Tested again with
Code: Select all
telemetry='Z'
connect to the IDE's Serial monitor and -
same effect - can feel jumpy copter on each push of serial data every 4 seconds! If it was as simple as I thought this must not happen!

I can only say it again - I am lost for now and will take a break and get some sleep.

mr.rc-cam · Post by **mr.rc-cam** » Mon Dec 12, 2011 4:37 am

ust to be sure I checked again this disturbance upon Serial transmission happens with both Arduino-022 and Arduio-100.

Ok, thanks for checking that. Sometimes different compiler versions introduce different program behaviors, so it was worth checking.

Then I applied your simple experiment. When connected to the GUI, right after hitting Start, the counter skyrocketed together with the first few graph items showing, then stops at some value. I could repeat this only once, on all other occasions it stays at zero. So this is not easy to interpret. The jumps I saw may be due to some initialization thing and not be really relevant.

That is good news. It appears the buffer is not getting full during normal GUI operation. That means the buffer size is fine for the GUI's Tx data. But the LCD communication is where you reported the problem, so please repeat the test with the LCD.

This should give the same behavior as what was done in previous versions of Arduino IDE.
ie a non interrupt based Serial.write().
It's still ok with the GUI, even if there was a special trick to use interrupts for it.
If it's fine for you, there's maybe no need to switch to an interrupt based method to transmit serial data.

Alex, there is something unusual about the symptoms. From the information in the first post, using the GUI does not cause the problem. It only occurs with the LCD. The GUI sends well over 100 bytes, but the LCD is about 35 bytes. So it is very strange that the problem does not appear when using the GUI. So maybe it is too soon to abandon the interrupt driven Tx? That is to say, perhaps there may be other things involved in the problem.

Hamburger · Post by **Hamburger** » Mon Dec 12, 2011 9:58 am

mr.rc-cam wrote:Alex, there is something unusual about the symptoms. From the information in the first post, using the GUI does not cause the problem. It only occurs with the LCD. The GUI sends well over 100 bytes, but the LCD is about 35 bytes. So it is very strange that the problem does not appear when using the GUI. So maybe it is too soon to abandon the interrupt driven Tx? That is to say, perhaps there may be other things involved in the problem.

I am not sure the problem does not appear when using GUI. Maybe we do not see the sympton that clearly with the GUI because the frequency of serial transmission is so high that it induces its disturbance all the time and we perceive it as general noise. Whereas with the LCD code and especially my trimmed down to 1/4 Hz version, the transmissions occur so seldom compared to main loop cycle, that it stands out from normal operation and becomes recognizable. But it may always be there.

Also when I have hardwired the unconditional transmission by setting telemetry='C' and LCD_TELEMETRY at 1/4Hz and then run the GUI I do not see any obvious spikes in the sensor graphs. So I tend to believe the problem is not with the sensors but with the motor outputs. On the other hand maybe the GUI is just not fast enough to catch this.

Hamburger · Post by **Hamburger** » Mon Dec 12, 2011 2:30 pm

Alexinparis wrote:Could you try to replace this function in the last code: (just this mod, nothing more)
Code: Select all
void SerialWrite(uint8_t port,uint8_t c){
  while (!(UCSR0A & (1 << UDRIE0))) ;
  UDR0 = c;
}
This should give the same behavior as what was done in previous versions of Arduino IDE.
ie a non interrupt based Serial.write().

It's still ok with the GUI, even if there was a special trick to use interrupts for it.

If it's fine for you, there's maybe no need to switch to an interrupt based method to transmit serial data.

ok, so here is what I did:
pull latest non_shared version from trunk (with your own serial implementation), use arduino-100.
configure minimal features, use

Code: Select all

      analogWrite(PWM_PIN[i], ((motor[i]>>2) - 250) + 2);

for the special ESCs at this copter. All this is like with all my previous tests.
as always, my changes to config.h are in additional TRI60.h included after config.h to override your settings where neccessary.

test 1:
Then upload, run GUI, configure ACC, stop GUI, run arduino-IDE terminal, enter 'A' - should give 1/4Hz glitches when ~35chars printed? YES.
test2:
change SerialWrite(), same procedure as before - glitches? YES.

test3 (with same codebase as test2):
ok, now I additionally preset telemetry='A' and run GUI. Cannot see the telemtry chars but maybe can see glitches in either sensor graphs or motor outputs? maybe not, because these chars disturb the 'M' routine of the GUI? - nothing to recognize in GUI, signals already are too jumpy with motor running.But: I cannot clearly feel the 1/4Hz glitches any more! maybe the continuous transmission to the GUI already supersedes everything?
just to be safe we are talking about the same code I append the code of test3.

test4 (with test1 code again):
change GUI code poll 'M' with delay set to 5seconds like this:

Code: Select all

  if (init_com==1 && (time1-time2)>5000 && graph_on==1) {

feel glitches? - NO

maybe the amount of data for 'M' is closer to the buffer size of 128 than the ~35chars for telemetry - does it matter?
test5: (change test1 code)
extend line1,line2 and send approx 100chars on each telemetry transmission at 1/4Hz - run from terminal. Glitches? YES
so it is not that the buffer needs to be almost full.

Can anyone verify the effect of the glitches? Should be possible, because no LCD is needed, just copter and arduino IDE.
out of ideas atm.
Hamburger

mr.rc-cam · Post by **mr.rc-cam** » Mon Dec 12, 2011 6:05 pm

Also when I have hardwired the unconditional transmission by setting telemetry='C' and LCD_TELEMETRY at 1/4Hz and then run the GUI I do not see any obvious spikes in the sensor graphs. So I tend to believe the problem is not with the sensors but with the motor outputs. On the other hand maybe the GUI is just not fast enough to catch this.

High on my list would be that the serial interrupts cause latency on the R/C pulse decoding interrupt, which is a time critical task. To check for that, when you run the GUI do you see stable R/C pulse values or is there some jitter in them? They should be as stable as the old working code.

But you said that when you used Alex's non-interrupt serial write function you also had the same problem. So that puts a very strange twist to this problem! At the moment I am lost too.

maybe the amount of data for 'M' is closer to the buffer size of 128 than the ~35chars for telemetry - does it matter?

If the test code's debug2 value is not changing during normal use then the Tx buffer size is fine.

Can anyone verify the effect of the glitches? Should be possible, because no LCD is needed, just copter and arduino IDE.

I'm not ready to install the new IDE and dev code. I will leave such things to you brave early adopters.

Hamburger · Post by **Hamburger** » Mon Dec 12, 2011 6:44 pm

ah, but you can use IDE ß22 and just rename the one ino file?
About my test with the buffer - I was trying to minimize differences between M command from GUI and telemetry transmission. As M is sending more data (close to buffer size) I tried to emulate that and padded the telemetry data to same amount.with no succcess.

By now I guess this glitches error may ell have been present for long time, just not visible because normal frequency of telemetry transmission is high enough to blur visibility.

Couuld test to increase freq?

Hamburger · Post by **Hamburger** » Mon Dec 12, 2011 6:52 pm

or you could test with olde IDE and v1.8 or early v1.9 code with telemetry freq set to 1/5Hz and connected via cable to IDE TERMINAL.

Hamburger · Post by **Hamburger** » Mon Dec 12, 2011 6:54 pm

We should resolve this issue. Osd is using serial as well, so ignoring is not really an option.

mr.rc-cam · Post by **mr.rc-cam** » Mon Dec 12, 2011 7:10 pm

Do you see stable R/C servo pulses in the GUI? Compare the old working version to the new/bad version.

I looked at your zip file set and saw that the Tx's ISR is still enabled in the serial.pde. I think that this will interfere with Alex's non-interrupt (polled) Tx method. So try the test again with the ISR(USART_UDRE_vect) function commented out.

or you could test with olde IDE and v1.8 or early v1.9 code with telemetry freq set to 1/5Hz and connected via cable to IDE TERMINAL.

Before I do that I'd like you try the bad code on a Quad. In case a Tri acts differently than the Quad, what are the most obvious symptoms I should look for on the Quad?

We should resolve this issue. Osd is using serial as well, so ignoring is not really an option.

Agreed. Any little bug today will often be a much bigger bug problem tomorrow.

- Thomas

Hamburger · Post by **Hamburger** » Tue Dec 13, 2011 3:48 pm

mr.rc-cam wrote:Do you see stable R/C servo pulses in the GUI? Compare the old working version to the new/bad version.

yes, I can see the input sliders and values in the GUI to be stable if I do not move the sticks. But I can only test this when I have no telemtry transmission possibly interfering. So I am not sure what to deduce from that.
Just went back in time and did the telemetry @ 1/4Hz transmission test with an old v1.8px version. No glitches there. So this has crept into the code somehow after r252.

I looked at your zip file set and saw that the Tx's ISR is still enabled in the serial.pde. I think that this will interfere with Alex's non-interrupt (polled) Tx method. So try the test again with the ISR(USART_UDRE_vect) function commented out.

commented that out. But then mwc does not initialize, no servo movement and no alarm-LED flash upon reset after upload or power on, no reaction to GUI or to TX input. So this test I can not make the way you wanted.

or you could test with olde IDE and v1.8 or early v1.9 code with telemetry freq set to 1/5Hz and connected via cable to IDE TERMINAL.

Before I do that I'd like you try the bad code on a Quad. In case a Tri acts differently than the Quad, what are the most obvious symptoms I should look for on the Quad?

I do not have a quad. But if you set the telemetry interval to 4 seconds, hook up via cable and run the terminal within Arduino IDE, enter 'A' to start telemetry transmission. Fly copter in hand at hover throttle. For me I can feel almost every serial transmission as it becomes visible in the terminal, because the copter makes one considerable twitch at same time.

We should resolve this issue. Osd is using serial as well, so ignoring is not really an option.

Agreed. Any little bug today will often be a much bigger bug problem tomorrow.

yes.

mr.rc-cam · Post by **mr.rc-cam** » Tue Dec 13, 2011 6:32 pm

yes, I can see the input sliders and values in the GUI to be stable if I do not move the sticks.

Good to know.

... commented that out. But then mwc does not initialize, no servo movement and no alarm-LED flash upon reset after upload or power on, no reaction to GUI or to TX input. So this test I can not make the way you wanted.

Try it again with this code in place of existing ISR(USART0_UDRE_vect) function. Your serial.ino must use Alex's serial write function patch.

Code: Select all

ISR(USART0_UDRE_vect) {
    UCSR0B &= ~(1<<UDRIE0);     // Kill transmitter interrupt 
    serialHeadTX[0] = 0;   // Init Buffer Pointer
    serialTailTX[0] = 0;  // Init Buffer Pointer
}

I do not have a quad. But if you set the telemetry interval to 4 seconds, hook up via cable and run the terminal within Arduino IDE, enter 'A' to start telemetry transmission. Fly copter in hand at hover throttle. For me I can feel almost every serial transmission as it becomes visible in the terminal, because the copter makes one considerable twitch at same time.

The interrupt structure is more relaxed on a Quad so I don't expect to feel the twitch like you do on your Tri. But in case it helps I will try your test. I have IDE 022, what code do you want me to try that will compile on it? It would be great if you tested the code on 022, confirmed the issue, then zipped it up and posted it here (don't use dev file area, I have problems downloading the code from there).

copterrichie · Post by **copterrichie** » Tue Dec 13, 2011 7:24 pm

I wonder if the usage of a digital servo and modifying the Tri code to use a digital servo verse an analog servo will help.

I am still using 1.7 however what I found out was, I needed to power the servo separately from the FC. The servo introduced too much noise on the power line.

mr.rc-cam · Post by **mr.rc-cam** » Tue Dec 13, 2011 7:40 pm

I wonder if the usage of a digital servo and modifying the Tri code to use a digital servo verse an analog servo will help.

The Tri's servo code is another interrupt burden for sure. As soon as we identify which interrupt(s) is causing the issue we can come up with the best game plan. Or we may find a solution that does not involve changing the interrupt usage. At this point it is all a mystery.

BTW, if the Tri's servo is disconnected does the twitch feeling go away? If the twitch is only from the yaw servo then that will be good to know.

mr.rc-cam · Post by **mr.rc-cam** » Tue Dec 13, 2011 11:50 pm

I took your trunk-modifield-SerialWrite.zip test code and configured it for the Quad configuration. This is the version with Alex's non-interrupt serial write. Compiled it under IDE 022.

I cannot feel any twitch at each LCD broadcast (4-sec update rate). Seems normal under various throttle ranges. Nothing unusual to report. If you have another version you want me to try then please zip and post it.

I verified that my ISR(USART0_UDRE_vect) patch worked with this version. So please try it.

Lastly, do you feel the twitch if you unplug your servo?

Alexinparis · Post by **Alexinparis** » Wed Dec 14, 2011 12:20 am

Hi,
I've just committed a new mod.

Some explanations:
There are 2 parts about Serial com: TX and RX which are handle differently and independently.

on the previous version of Arduino IDE:
- RX was handled with interrupts
- TX was handled with no interrupt for direct Serial.write().

with Arduino 1.0:
- RX is handled with interrupts (like before)
- TX is now also handled with interrupts.

MIS introduced a custom interrupt function for GUI communication. (used initially only for MIS OSD)
The new TX interrupt implementation of Arduino 1.0 appears to be less efficient than the custom interrupt function from MIS (5 or 6 times slower)

So I reintroduced the custom interrupt function which ensure a total delay to write all Arduino to GUI com less than 150 microseconds (was 870microseconds before). + some tricks to avoid /10 or /3 costly divisions.

The SerialWrite(port,c) (equivalent to Serial.Write(c)) function is handled like before with no interrupts.
This point is important, because a single char needs 8/115200 s to be sent with a direct write approach.
ie for a string of 35 chars, 2.4ms.
As the serialcom function and Telemetry functions are called inside the annexCode() function, the delay is far beyond the recommended 650 microseconds.
This fact is true since the introduction of telemetry => the current implementation of telemetry is not clean from this point of view and might explain some side effects.

@Hamburger, I suggest you to try this last mod (with the updated GUI). If the problem still persists, there is probably something new in Arduino 1.0 we still don't think about.

dramida · Post by **dramida** » Wed Dec 14, 2011 12:54 am

I tried to compile the GUI from dev 20111214 in Processing 1.5.1 on Win XP SP3 and i got this error:

['siteTracker._setAccount', 'UA-18071-1'],
"Badly formed character constant"

Note that previous versions of gui are compiling OK.

Latest edit:

I figure it out, i downloaded as PDE the HTML page

)) of corse, when you compile an html text with pde extension, this is what you get

)

ziss_dm · Post by **ziss_dm** » Wed Dec 14, 2011 1:32 am

Hi Alex,

Can I make couple of notes?

1) Currently we have packet buffer s[200] and all variables which we serialize. I think, to economise SRAM it is better to assemble all varibles into struct and use it for sending. Additionally all transformations should be moved to the GUI side.
2) It is quite strange to have 64 byte receive buffer, taking into account that we are mostly receiving 1byte packets.

3) And if I implement MAVLINK as a main comm. protocol, will you consider to adopt it?

regards,
ziss_dm

Alexinparis · Post by **Alexinparis** » Thu Dec 15, 2011 12:07 am

Hi,

sure you can

1) I think I see what you have in mind. Is it something like the structure in the eeprom part to store parameters ?
We have to be careful because the data will be accessed inside an interrupt function.
It's maybe not important to be careful about this as data would be read only.
But I agree it would be a nice code optimization.
We must think also about the java GUI part. I don't know if it's possible to map structs easily.

2) It's true except for the W command, when multiwii waits for 25+2*CHECKBOXITEMS chars

3) I will adopt it if:
- it's globally equivalent in term of transmitted byte size efficiency
- it's globally equivalent in term of computation time
- it's as easy as now to add a new param or remove an existing one
- if the current GUI could be also adapted.
- if the code size does not grow too much
If you think it's ok, be sure I will switch to it

dramida · Post by **dramida** » Thu Dec 15, 2011 12:19 am

I just flew my octo with latest dev 20111214 on a mega board, obviously. I still got short cut off all engine moments some times,copter drops straight down 2-3 cm, it's scary.

Hamburger · Post by **Hamburger** » Thu Dec 15, 2011 12:21 am

Hi Alex,

thank you for taking this Serial seriously.

Alexinparis wrote:MIS introduced a custom interrupt function for GUI communication. (used initially only for MIS OSD)
The new TX interrupt implementation of Arduino 1.0 appears to be less efficient than the custom interrupt function from MIS (5 or 6 times slower)

so this might be what for me makes the difference between an symptom free late v1.8p1 version and the v1.9 problems.

So I reintroduced the custom interrupt function which ensure a total delay to write all Arduino to GUI com less than 150 microseconds (was 870microseconds before). + some tricks to avoid /10 or /3 costly divisions.

The SerialWrite(port,c) (equivalent to Serial.Write(c)) function is handled like before with no interrupts.
This point is important, because a single char needs 8/115200 s to be sent with a direct write approach.
ie for a string of 35 chars, 2.4ms.
As the serialcom function and Telemetry functions are called inside the annexCode() function, the delay is far beyond the recommended 650 microseconds.
This fact is true since the introduction of telemetry => the current implementation of telemetry is not clean from this point of view and might explain some side effects.

very good that we are addressing it now for real

@Hamburger, I suggest you to try this last mod (with the updated GUI). If the problem still persists, there is probably something new in Arduino 1.0 we still don't think about.

done that with Arduino-IDE 100. (and to answer one of Mr.RC-Cam meaningful requests I also removed the servo plug to disable TRI's servo)
test 1
Way better, but occasionally a glitch is still there. While it was there with the former version on >50% of the serial telemetry writes, with the new version it is down to approx 10% (meaning I can feel it 10% of the time, not that a disturbance does not occur in 90% of the time).
test 2
For experiment I trimmed down the write to 9 bytes. -> symptoms gone (not feelable over 2 minutes).

To complete test scenario, I compile with Arduino IDE 022 (just rename main file)
test 3
repeat test 1 - same result, can feel occasional glitches
test 4
repeat test 2 - same result, symptoms gone.

So I guess we are on the right track with the Serial and the time consumed. Since Mr.RC-Cam could not reproduce symptoms with a quad, maybe we should also consider whether creating servo output signal would also compete for output time. The way I see it we have four candidates for time and/or interrupt handling
- sensor reading (i2c)
- motor output
- servo output
- serial i/o

test 5 (IDE 022 still):
remove SERVO definition from beginning of Output.pde for TRI.
repeat test 1 - same result, can feel occasional glitches
repeat tests 2 - same result, symptoms gone.

test 6:
change GUI to send M-request 1/5Hz - no symptoms, though ~100 bytes get transferred

my conclusion:
(from test 5) servo output by itself is not the bottleneck,
(from tests 2 and 4) the amount of serial data transmission for telemetry does matter (maybe only in terms of making the symptom extreme enough to become recognizable),
but from test 6 - amount does not matter for M-command. contradiciton ?!?
(from test 1) the way of transmission seems to be relevant to the symptom, 'old style' serial is faster, faster is better.
(from test 6) the amount of data (and thus transmission time alone, does not cause the disturbance.
the choice of IDE 022 vs 100 does not make the difference

how to proceed?
I could trim down the telemetry transmissions to only update those parts on lcd-screen that have actually changed; on average that would reduce from 2*(3+16)=38 down to 4+5=9 per changed value (with telemetry pages ranging form 1 to 4 values).
For the questionable benefit of masking an existing problem, quite some effort. I'd rather help debugging this further (although I am getting kinda worn out on this).

What amazes me:
the M-command transfers so much more data, why is the disturbance not recognizable there? Or is it masked by the high frequency (not if my test 6 holds)? Do we know how frequent OSDs do request data?

the contradiciton about size does or does not matter needs inspection. Obvious difference is that for M-command handling the mwc has to handle some incoming data (a 'M' char) as an RX event before sending again. I cannot test the hypothesis easily, because sending constant stream of chars from terminal or lcd is not possible.

what do you think?

Hamburger · Post by **Hamburger** » Thu Dec 15, 2011 12:24 am

dramida wrote:I just flew my octo with latest dev 20111214 on a mega board, obviously. I still got short cut off all engine moments some times,copter drops straight down 2-3 cm, it's scary.

does this occur while you have running serial transmission of data?
Else I would assume your effect - as critical as it is - belongs in another thread?

Hamburger · Post by **Hamburger** » Thu Dec 15, 2011 12:26 am

mr.rc-cam wrote:
I wonder if the usage of a digital servo and modifying the Tri code to use a digital servo verse an analog servo will help.

The Tri's servo code is another interrupt burden for sure. As soon as we identify which interrupt(s) is causing the issue we can come up with the best game plan. Or we may find a solution that does not involve changing the interrupt usage. At this point it is all a mystery.

BTW, if the Tri's servo is disconnected does the twitch feeling go away? If the twitch is only from the yaw servo then that will be good to know.

NO, unfortunately it is not.
I removed servo plug _and_ disabled SERVO macro in OUtput.pde. This should remove all servo related code and interrupt handling. But No change for the symptoms. good point and we know that now.

Hamburger · Post by **Hamburger** » Thu Dec 15, 2011 12:39 am

mr.rc-cam wrote:
... commented that out. But then mwc does not initialize, no servo movement and no alarm-LED flash upon reset after upload or power on, no reaction to GUI or to TX input. So this test I can not make the way you wanted.

Try it again with this code in place of existing ISR(USART0_UDRE_vect) function. Your serial.ino must use Alex's serial write function patch.
Code: Select all
ISR(USART0_UDRE_vect) {
    UCSR0B &= ~(1<<UDRIE0);     // Kill transmitter interrupt 
    serialHeadTX[0] = 0;   // Init Buffer Pointer
    serialTailTX[0] = 0;  // Init Buffer Pointer
}

I
ok, IDE 100.
same effect. does not initialize (no servo movement, no alarm-led, no reaction to TX input).
Is this test (whatever you intended to investigate, superseeded by Alex' new serial code (old MIS code)?

Hamburger · Post by **Hamburger** » Thu Dec 15, 2011 1:04 am

mr.rc-cam wrote:The interrupt structure is more relaxed on a Quad so I don't expect to feel the twitch like you do on your Tri. But in case it helps I will try your test. I have IDE 022, what code do you want me to try that will compile on it? It would be great if you tested the code on 022, confirmed the issue, then zipped it up and posted it here (don't use dev file area, I have problems downloading the code from there).

Thomas,
this is with Alex' newest code (r415). I created symlink for MultiWii.ino to MultiWii.pde. Then compiles with IDE 022.
keep copter connected via serial cable for test
- config for your h/w in TRI60.h, compile, upload,
- run GUI to calb ACC, set switches so ACC is off
- run terminal from IDE,
- hover copter in hand at short below hover throttle,
- send capital 'A' from terminal
- see telemetry chars come in at 1/4Hz,
- feel copter jerking shortly in hand, coincides with serial transmission showing in terminal.

I can feel this at least at 10% of the transmissions.
If that works for you, then try with sending a capital 'C'. That will send shorter string only; for me that gives fewer glitches, but still sometimes recoglnzable.

I hope you can verify the symptom! I would not know how to make a film sequence out of that. And yes, I am not insane to make it all up (at least not yet, I think

)

mr.rc-cam · Post by **mr.rc-cam** » Thu Dec 15, 2011 4:56 am

The way I see it we have four candidates for time and/or interrupt handling
- sensor reading (i2c)
- motor output
- servo output
- serial i/o

+ R/C pulse decoding.

On a TRI configuration, I expect that servo output, serial I/O, and R/C pulse decoding are the items to watch if this is strictly a typical interrupt latency issue.

test 5 (IDE 022 still):
remove SERVO definition from beginning of Output.pde for TRI.
repeat test 1 - same result, can feel occasional glitches
repeat tests 2 - same result, symptoms gone.

If you removed SERVO and still had glitches, then I would say that it leaves serial I/O interrupts competing with R/C Pulse decoding. But you used Alex's first patch (see post Sun Dec 11, 2011 3:13 pm), which does not use serial interrupts, and you mentioned that you still had the problem. That leaves us with just R/C pulse decoding and no shared interrupt conflicts. So perhaps we've missed something along the way.

What amazes me:
the M-command transfers so much more data, why is the disturbance not recognizable there?

+1.

I removed servo plug _and_ disabled SERVO macro in OUtput.pde. This should remove all servo related code and interrupt handling. But No change for the symptoms. good point and we know that now.

Just for clarity, what exactly did you disable? Did you do comment out these lines?:

Code: Select all

//#if defined(BI) || defined(TRI) || defined(SERVO_TILT) || defined(GIMBAL) || defined(FLYING_WING) || defined(CAMTRIG)
//  #define SERVO
//#endif

I asked:

Try it again with this code in place of existing ISR(USART0_UDRE_vect) function. Your serial.ino must use Alex's serial write function patch.

You replied:

same effect. does not initialize (no servo movement, no alarm-led, no reaction to TX input).

Very strange. I used the same code in dev 20111203 with Alex's first patch (see post Sun Dec 11, 2011 3:13 pm) with success.

Is this test (whatever you intended to investigate, superseeded by Alex' new serial code (old MIS code)?

My ISR code was to ensure that that Tx interrupts were disabled while you used Alex's first patch. We did not want to have a Tx interrupt or his test would be invalid. We may want to come back to this later.

this is with Alex' newest code (r415). I created symlink for MultiWii.ino to MultiWii.pde. Then compiles with IDE 022.
keep copter connected via serial cable for test
- config for your h/w in TRI60.h, compile, upload,
- run GUI to calb ACC, set switches so ACC is off
- run terminal from IDE,
- hover copter in hand at short below hover throttle,
- send capital 'A' from terminal
- see telemetry chars come in at 1/4Hz,
- feel copter jerking shortly in hand, coincides with serial transmission showing in terminal.

I can feel this at least at 10% of the transmissions.
If that works for you, then try with sending a capital 'C'. That will send shorter string only; for me that gives fewer glitches, but still sometimes recognizable.

I hope you can verify the symptom!

I edited the TRI60.h and configured it for a QuadX and my sensors. Other than killing the EXT_MOTOR_RANGE, all else was the same. I cannot feel any twitch. I really tried to feel it, but the Quad was smooth as usual with a variety of throttle positions.

But two things to comment on:

(1) There is something evil about the provided file set. My ACC's sensitivity was insane with it. It was like it was 5X more sensitive than usual. I made no attempt to discover why. My configs were correct.

(2) I think that it would help if you went through your configuration and striped it down into a generic TRI model. For example, the EXT_MOTOR_RANGE pissed off my ESC's. Things like it need to be avoided during the testing. Likewise, I recommend you suspend your use of the TRI60.h file and just edit the normal config file during the testing. I'm not saying any one of these things are bad to do, but simplification is a necessary troubleshooting tool. Once you have a basic system that works, free of any "special touches," you can proceed to expand on it until it breaks. At that point you might be able to ID the culprit.

Unfortunately I don't have anything to report that will solve your TRI's glitch problem. But this nasty thing will get solved, so stick with it!

- Thomas

ziss_dm · Post by **ziss_dm** » Thu Dec 15, 2011 7:38 am

Hi Hamburger,

Can you comment SerialWrite(0, i ) in LCDprint()? Or you already tried that?

regards,
ziss_dm

Hamburger · Post by **Hamburger** » Fri Dec 16, 2011 12:50 am

Thomas,

mr.rc-cam wrote:Just for clarity, what exactly did you disable? Did you do comment out these lines?:
Code: Select all
//#if defined(BI) || defined(TRI) || defined(SERVO_TILT) || defined(GIMBAL) || defined(FLYING_WING) || defined(CAMTRIG)
//  #define SERVO
//#endif

yes.

I asked:
Try it again with this code in place of existing ISR(USART0_UDRE_vect) function. Your serial.ino must use Alex's serial write function patch.

You replied:
same effect. does not initialize (no servo movement, no alarm-led, no reaction to TX input).

Very strange. I used the same code in dev 20111203 with Alex's first patch (see post Sun Dec 11, 2011 3:13 pm) with success.
Is this test (whatever you intended to investigate, superseeded by Alex' new serial code (old MIS code)?

My ISR code was to ensure that that Tx interrupts were disabled while you used Alex's first patch. We did not want to have a Tx interrupt or his test would be invalid. We may want to come back to this later.

yes, strange.

I edited the TRI60.h and configured it for a QuadX and my sensors. Other than killing the EXT_MOTOR_RANGE, all else was the same. I cannot feel any twitch. I really tried to feel it, but the Quad was smooth as usual with a variety of throttle positions.

But two things to comment on:

(1) There is something evil about the provided file set. My ACC's sensitivity was insane with it. It was like it was 5X more sensitive than usual. I made no attempt to discover why. My configs were correct.

(2) I think that it would help if you went through your configuration and striped it down into a generic TRI model. For example, the EXT_MOTOR_RANGE pissed off my ESC's. Things like it need to be avoided during the testing. Likewise, I recommend you suspend your use of the TRI60.h file and just edit the normal config file during the testing. I'm not saying any one of these things are bad to do, but simplification is a necessary troubleshooting tool. Once you have a basic system that works, free of any "special touches," you can proceed to expand on it until it breaks. At that point you might be able to ID the culprit.

ok, I did cleanup the files and moved my settings into config.h for now.
But I do need the EXT_MOTOR_RANGE handling, otherwise with these special ESCs the motors will not run.

Hamburger · Post by **Hamburger** » Fri Dec 16, 2011 12:57 am

ziss_dm,

ziss_dm wrote:Can you comment SerialWrite(0, i ) in LCDprint()? Or you already tried that?

yes I tried, but I think the symptom is still there. But then it is hard to identify the twists and movements of copter as being clearly related to the 1/4Hz telemetry invocations, when you can not see those! I tried by counting in my head.

So I repeated an earlier experiment and shortened the telemetry string to be transferred for 'C' down to 5 chars. Then the symptom is not there any more (or it is not strong enough anymore to be recognizable.)

The saga continues.

Hamburger · Post by **Hamburger** » Fri Dec 16, 2011 1:44 am

I tried to film the computer screen with terminal window, one running rotor with phone camera to demonstrate what I experience. Bu the film does not clearly show the glitche.
following tests each based on r415, with one modification as described, then reverted back.

next test:
Limiting the sensor definitions down to only 2 sensors of the 4 on the FreeIMU035_ms does not help.

next test:
disabling serial auxpin12 does not help

next test:
simplify the call structure like this (less stack activity?)

Code: Select all

void LCDprintChar(const char *s) {
  while (*s) SerialWrite(0,  *s++); //was LCDprint(*s++);
}

_and_ slowing down the access to registers by inserting delayMicroseconds(1) like this (read it somewhere)

Code: Select all

void SerialWrite(uint8_t port,uint8_t c){
  switch (port) {
    case 0: while (!(UCSR0A & (1 << UDRIE0))) delayMicroseconds(1); UDR0 = c; break;

DOES help. Run for several minutes, cannot feel a single glitch anymore.
make sense? it is most probably not short on memory

Program: 22468 bytes Data: 1533 bytes

So, would either one of the two modifications alone do the trick?
test
do only the call LCDprint simplification - no luck

test
do only the delaymicro(1) - no luck.

Scary. I have no clue what is going on here. Does this make sense to you?

Hamburger · Post by **Hamburger** » Fri Dec 16, 2011 1:55 am

fromt the arduino documentation

As of Arduino 0018, delayMicroseconds() no longer disables interrupts.

it does not say that for the micros() function.
Maybe we have too many calls to micros() (order of 10) in the main loop and in annexCode() and those interfere with timing/interrupts/internal whatever?

mr.rc-cam · Post by **mr.rc-cam** » Fri Dec 16, 2011 3:18 am

it does not say that for the micros() function.
Maybe we have too many calls to micros() (order of 10) in the main loop and in annexCode() and those interfere with timing/interrupts/internal whatever?

The micros() function briefly disables interrupts. Typically very few clock cycles would be missed by the dozen or so calls since the function is short. But something to look into, especially in some of the micros() based wait loops.

The delay() function can be a bigger threat because it hassles the interrupts for an extended period of time. There are some delays() here and there, even in LCD.pde. But I did a quick look in the pde's and did not see anything that would explain your problem. But someone else should check it too in case I missed something.

So it could be useful to play with the code's micros() and delay() to see if any are involved in your problem. For example, delayMicroseconds() doesn't interfere with interrupts so you could try it instead of delay().

- Thomas

Hamburger · Post by **Hamburger** » Fri Dec 16, 2011 11:30 am

Thomas,

we currently have some open ends to clear.
Did you see that one of my tests (with simplified call structure to LCDprint() and inserted delayMicroseconds(1) to SerialWrite() _did_ fix the symptoms? Symptoms is all I can perceive, not sure if it resolves the cause. Seems we do not have direct access to arduino's innards via debugger or some such. (reducing amount of chars reduces symptoms but surely does not fix cause)

I will experiment with removing the micros() calls as far as possible. Not sure if this will lead anywhere. Actually, what ticked that thought was that after my test with delayMicroseconds() the term micros popped up in my mind and I remember that at one point (early v1.8 or even pre 1.8) we had an inflation of calls to micros() in main loop and annexCode. In former versions this had been dealt with differently. I had asked Alex about this change but nobody tracked that issue.

Should I pursuit the test with changes to the serial code you suggested - so far it did not work for me but I could continue trying. Or is the new finding superseeding that?

The fact that you cannot reproduce the symptom has me worried still. It could be because of different physical layout of copters (arm length, motor/rotor strength & size) etc. Or it could be related to my specific copter/cabling/wire routing/whatnot . Unfortunately I do not see an easy way how we could transfer this 'feel the copter glitch in hand' into a 'hard fact' like a spike in whatever graph.

what do you think? Hamburger