v1.9 hang with Magneto of FreeIMUv1 enabled - i2c related?

This forum is dedicated to software development related to MultiWii.
It is not the right place to submit a setup problem.
Software download
User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

v1.9 hang with Magneto of FreeIMUv1 enabled - i2c related?

Post by Hamburger »

UPDATE
sensor problem actually diagnosed as memory starvation problem. to be discussed in the other thread. Thanks for all the support, learning something new all the time.
/UPDATE

Hi,
I just upgraded one copter from 1.8p1 (plus some development features, based on r252) to latest v1.9.
It does not work but hangs almost instantly - see attached gui.
This happens with official v1.9 and with Alex' latest trunk r376.
Switching back to my v1.8p1 version gives working copter again.

So I think it might be i2c problem. But error counter in GUI is no help?
When I manually configure the freeIMUv1 items without magneto - voila, it works again.
Adding magneto defs and oops, not working.

So copter is a TRI with freeimuV1:

Code: Select all

//#define FREEIMUv01
//#define FREEIMUv1
/* fax8 freeimu 0.1 */
#undef INTERNAL_I2C_PULLUPS
#define ITG3200 // gyro
#define ADXL345 // acc
//#define HMC5843 // magneto
#define ADXL345_ADDRESS 0xA6 // taken from fabio's library and his post http://wbb.multiwii.com/viewtopic.php?f=8&t=262
  #define ACC_ORIENTATION(X, Y, Z)  {accADC[ROLL]  =  -Y; accADC[PITCH]  = X; accADC[YAW]  = Z;}
  #define GYRO_ORIENTATION(X, Y, Z) {gyroADC[ROLL] =  X;  gyroADC[PITCH] = Y; gyroADC[YAW] = Z;}
  #define MAG_ORIENTATION(X, Y, Z)  {magADC[ROLL]  =  X;  magADC[PITCH]  = Y; magADC[YAW]  = Z;}
  #define ADXL345_ADDRESS 0xA6
  #undef INTERNAL_I2C_PULLUPS


For pullups I use 2.7k (not absolutely sure, cannot visually identify value, but that was what Fabio recommended.
I also tried the debug version of waitTransmissionI2C() function, but did not get any more info.
Only once during my hourlong test runs did I see an i2c error count of 3.

For now I am at my wits end - any pointers most welcome.
Attachments
v1.9 r376 freeze
v1.9 r376 freeze
Last edited by Hamburger on Thu Dec 08, 2011 3:48 pm, edited 2 times in total.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto enabled - i2c related?

Post by mr.rc-cam »

This happens with official v1.9 and with Alex' latest trunk r376.

Perhaps it would be useful to zip up your existing MWC file set and post it here.

There are three "official" V1.9 versions. You want the latest one that has the NOV-12 date stamp. So, are you absolutely sure you have the latest release? If any doubt then download it again and start fresh. Also, I looked through the trunks and the only r376 I found was for the LEDRing feature. If you have installed the LEDRing then disconnect it and remove the support file and all related edits.

So I think it might be i2c problem. But error counter in GUI is no help?

If the GUI is locked up then the I2C error counter won't work. The GUI has to be running in order for it to show the debug2 values.

For pullups I use 2.7k (not absolutely sure, cannot visually identify value, but that was what Fabio recommended.

It is important to know the exact value. So you will need to investigate and then report what values you have installed.

Also, post a detailed schematic of how the freeIMUv1 is wired in your installation. These little details matter.

Lastly, it would help to hear from other freeIMUv1 users to see if they had success with V1.9.

- Thomas

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto enabled - i2c related?

Post by Hamburger »

Thomas,
thanks for your considerate thoughts.
mr.rc-cam wrote:
This happens with official v1.9 and with Alex' latest trunk r376.

Perhaps it would be useful to zip up your existing MWC file set and post it here.

now I am at r377, same effect. You can find it in googlerep. Find my config attached - it contains deviations from default config.h; I include this file in MultiWii.pde after #include config.h

There are three "official" V1.9 versions. You want the latest one that has the NOV-12 date stamp. So, are you absolutely sure you have the latest release? If any doubt then download it again and start fresh. Also, I looked through the trunks and the only r376 I found was for the LEDRing feature. If you have installed the LEDRing then disconnect it and remove the support file and all related edits.

I understand. What I meant was I used Alex' trunk after updating my copy to r376, so in effect using his latest version - not the LEDring variant.
And yes, I know about the differing v1.9 variants. I had downloaded it yesterday, so it was latest as well. I have same effect with latest official v1.9.

So I think it might be i2c problem. But error counter in GUI is no help?

If the GUI is locked up then the I2C error counter won't work. The GUI has to be running in order for it to show the debug2 values.

I know about the GUI not working. That is why I posted the screenshot. The only info I can derive from it is the i2c error count as it was until GUI hung. What can be seen is that some cycles have passed, because one can see some data in the graph. So at least up to that point the i2c error count was 0.

For pullups I use 2.7k (not absolutely sure, cannot visually identify value, but that was what Fabio recommended.

It is important to know the exact value. So you will need to investigate and then report what values you have installed.

ok, will disassemble and measure tomorrow.
Also, post a detailed schematic of how the freeIMUv1 is wired in your installation. These little details matter.

schematic as recommended by Fabio here http://www.varesano.net/files/FreeIMU_v ... ctions.jpg , Vcc is taken from arduino nano 3.3V output.

Lastly, it would help to hear from other freeIMUv1 users to see if they had success with V1.9.

yes, please. (I am afraid we are only a few, though). I am not afraid to test further, so I am open to suggestions.
Thanks, Hamburger
Attachments
TRI60.h.zip
my config deviations from default config.h
(1.21 KiB) Downloaded 198 times

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

If you are using the latest V1.9 code then I doubt that the GUI freeze is caused by I2C errors. But I think I have an idea why it is occurring (just a gut feeling).

Your schematic shows that you are using the 3.3V output from your Arduino. This voltage is taken from its USB chip and so it is shared with the serial communication circuitry. If this voltage is corrupted the GUI may not work well. If you have a o-scope then use it to review the 3.3V. If it has "noise" and/or reduced voltage then that means the 3.3V output cannot support the load. You may ask why such a problem would only occur when you enable all the sensors. Well, as each sensor is involved the I2C buss gets more use and so the current and supply noise will increase. Plus, V1.9 has reduced cycle time so that contributes to the sensor activity as well.

Also, I see that you do not have an LLC. In your simple interface you have 3.3V I2C signals being used with the 5V CPU. This is not always a reliable solution since the CPU expects 5V logic levels. So there is a chance that after you solve the GUI lockup you may experience I2C problems. Be prepared for that; if things work fine (zero I2C errors) without the LLC then consider yourself blessed.

From what I have seen, there are a lot of frustrated builders out there due to V1.9. The main problem is that older MWC code masked I2C communication problems. The latest code has improved I2C drivers, plus it shows I2C errors so we can fix the previously hidden I2C hardware problems rather than ignore them. For example, some of the complaints about drift and other imperfect performance characteristics were partially due to I2C errors that the user did not know about.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:Your schematic shows that you are using the 3.3V output from your Arduino. This voltage is taken from its USB chip and so it is shared with the serial communication circuitry. If this voltage is corrupted the GUI may not work well. If you have a o-scope then use it to review the 3.3V. If it has "noise" and/or reduced voltage then that means the 3.3V output cannot support the load. You may ask why such a problem would only occur when you enable all the sensors. Well, as each sensor is involved the I2C buss gets more use and so the current and supply noise will increase. Plus, V1.9 has reduced cycle time so that contributes to the sensor activity as well.

Unfortunately I do not have an o-scope. But I found an unused lm1117 3.3V regulator and now use that to power the sensorboard. Same effect, works without magneto but freezes when using all three sensors. Again, no visible i2c errors.

Also, I see that you do not have an LLC. In your simple interface you have 3.3V I2C signals being used with the 5V CPU. This is not always a reliable solution since the CPU expects 5V logic levels. So there is a chance that after you solve the GUI lockup you may experience I2C problems. Be prepared for that; if things work fine (zero I2C errors) without the LLC then consider yourself blessed.

since using all 3 sensors still does not work, I need not worry about that.
Also I can remove the extra 3.3V regulator and rebuild to using the Nano's 3.3V output.

just for clarity I also ran a combo with only gyro and magneto configured. It sure works, values are visible in the GUI, again no i2c errors.
I2C_SPEED does not seem to have impact on my config working or not working. So short of getting an LLC again I am stuck here.

From what I have seen, there are a lot of frustrated builders out there due to V1.9. The main problem is that older MWC code masked I2C communication problems. The latest code has improved I2C drivers, plus it shows I2C errors so we can fix the previously hidden I2C hardware problems rather than ignore them. For example, some of the complaints about drift and other imperfect performance characteristics were partially due to I2C errors that the user did not know about.


well, it sounds like a clean approach and I can understand the incentive. For the moment I can not wholeheartedly welcome it.
Unless some further ideas pop up it seems I will have to live with an only partially working sensorboard for this copter.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

This is an odd one. But there is a reason for everything.

I do not believe this is a I2C issue, but I'll go back and look at the code specific to your sensors to see if there is something that could cause it. For sure, if you can find a buddy with a scope then you can see exactly what is going on.

I2C_SPEED does not seem to have impact on my config working or not working.

I2C_SPEED is strictly for the WMP and is ignored by the other sensors (which are hard coded to 400HKz). However, it would be interesting to try the slower I2C speed, which requires editing the sensor.pde file. To do this you would locate the "TWBR = ((16000000L / 400000L) - 16) / 2;" statement for **each** of your three sensors and replace it with "TWBR = ((16000000L / I2C_SPEED) - 16) / 2;" This will allow you to use I2C_SPEED to set the SCL clock.

BTW, if you haven't confirmed the pullup resistor value then it would still be a good idea to do that. It would also be great to hear from another FREEIMUv1 user to see if they report the same problem as you. Maybe you can contact the developer and have him try V1.9 using the exact same components that you are using. If he can duplicate it them he may be able to find the fix.

Unless some further ideas pop up it seems I will have to live with an only partially working sensor board for this copter.

Hopefully you can solve it; These sort of problems usually have other hidden issues that come back to haunt us, sometimes at very bad times.
Last edited by mr.rc-cam on Sun Nov 27, 2011 1:25 am, edited 1 time in total.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:This is an odd one. But there is a reason for everything.

I2C_SPEED does not seem to have impact on my config working or not working.

I2C_SPEED is strictly for the WMP and is ignored by the other sensors (which are hard coded to 400HKz). However, it would be interesting to try the slower I2C speed, which requires editing the sensor.pde file. To do this you would locate the "TWBR = ((16000000L / 400000L) - 16) / 2;" statement for **each** of your three sensors and replace it with "TWBR = ((16000000L / I2C_SPEED) - 16) / 2;" This will allow you to use I2C_SPEED to set the SCL clock.

will do that tomorrow.
BTW, if you haven't confirmed the pullup resistor value then it would still be a good idea to do that.

they both are 2.7k.
It would also be great to hear from another FREEIMUv1 user to see if they report the same problem as you. Maybe you can contact the developer and have him try V1.9 using the exact same components that you are using. If he can duplicate it them he may be able to find the fix.

yes, will do.
I see another thread has sprung up with wmp+bma020 problems of same kind. I am afraid I can relate to that as wel once I touch that other copter's running software.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

I see another thread has sprung up with wmp+bma020 problems of same kind. I am afraid I can relate to that as wel once I touch that other copter's running software.

There's been more than one V1.9 user like that. But the issues are being solved by using the latest V1.9 code, updating the WMP/BMA020 sensors for 5V operation, and using the correct pullup values. We just need to find the extra little thing that your installation needs so that you can have success to. I'd love to get my scope on your board to chase down this interesting problem. Too bad you can't email it to me. :)

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

will do that tomorrow

I did the test with slow i2c speed for all sensors.
Still, no success.
- only small part of graph in GUI, then hangs
- no i2c errors up until then
- READ button in GUI still works
- MWii does not react to yaw input from TX, servo on Tricopter does not move.

With only two sensors cycle time was up to ~4500, so reverted to higher value.

Too bad you can't email it to me

yes. so Fabio should come to the rescue.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

Still, no success.
- only small part of graph in GUI, then hangs
- no i2c errors up until then

I went back through the latest V1.9 I2C hardware specific code and cannot see anything that would directly cause the hang. The earlier V1.9 code would cause a I2C hang, but that issue had been fixed with the Nov-12 code release.

-READ button in GUI still works

For a moment that bit of news got me excited. But I understand that the values that are shown when the GUI's READ button is pressed are pre-read in the background with all the other data. Since your GUI ran for a short time it collected these values in its data buffer. When you pressed the READ button they became visible even though the GUI is no longer receiving data.

I have a feeling that the solution to the stalled GUI will be something quite unexpected. So a fresh approach would be best. Maybe Alex and the other experts will have some ideas. For sure, if you notice any other little things about the problem, even if it seems trivial, then be sure to report about it here.

BTW, the V1.9 release uses some new tricks to simplify the compiling efforts. It no longer requires the user to state the Arduino board type in the def.h file; The compiler does this for you. So please check a couple things:
1. Open the IDE's Help->About menu. What Arduino compiler version does it show?
2. Open Tool->Board menu. Exactly which CPU is selected?

Also, V1.9 has the FreeIMUv1 defined in it, so maybe it would be best to omit your TRI60.h file and configure your model using the traditional methods (via config.h). BTW, I noticed two "#define ADXL345_ADDRESS 0xA6" in your TRI60.h file so one of them should be deleted.

FWIW, it would be helpful to zip up your entire file set and post it here. I can load your exact code into my model and see if I get a GUI hang. Of course I don't use the same IMU board, but I still think it would be helpful to see if the problem can be duplicated.

EDIT: I read elsewhere that you have made improvements to the GUI. So please verify that you are using the unmodified MultiWiiConf_1_9 that was distributed with V1.9. This will help reduce the unexpected surprises.


- Thomas
Last edited by mr.rc-cam on Sun Nov 27, 2011 7:35 am, edited 1 time in total.

timecop
Posts: 1880
Joined: Fri Sep 02, 2011 4:48 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by timecop »

This is where having a processor that can be in-circuit debugged would prove invaluable. Instead of a bunch of experts suggesting potential solutions, breakpoint on i2c code, confirm what the problem is, and fix it. It's fun to watch though.

edit: before I get flamed I do realize I"m not offering any new solution to the problem.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

dongs wrote:edit: before I get flamed I do realize I"m not offering any new solution to the problem.

sad but true.

Do you have an alternative h/w readily available ? You did a port of MWii code to another architecture, right?
Do you have corresponding h/w plug and play ready for purchase or can point us to one?
If that was an option the developers interested could buy that, swap it against the arduino board for critical examination and benefit from its diagnosis' capabilities.
IMO that would be an offer in the direction of solution.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:I went back through the latest V1.9 I2C hardware specific code and cannot see anything that would directly cause the hang. The earlier V1.9 code would cause a I2C hang, but that issue had been fixed with the Nov-12 code release.

what comes to mind that my adxl345 has the same address as the default wmp. Even though the adxl345 is configured in, does the MWii somehow probe for something using the wmp address, which is now the adxl address and thus creating a conflict?

-READ button in GUI still works

For a moment that bit of news got me excited. But I understand that the values that are shown when the GUI's READ button is pressed are pre-read in the background with all the other data. Since your GUI ran for a short time it collected these values in its data buffer. When you pressed the READ button they became visible even though the GUI is no longer receiving data.


Eventually we should alter the GUI code in such a way that for a READ it invalidates the variables used after the values have been pushed into the GUI representation.

BTW, the V1.9 release uses some new tricks to simplify the compiling efforts. It no longer requires the user to state the Arduino board type in the def.h file; The compiler does this for you. So please check a couple things:
1. Open the IDE's Help->About menu. What Arduino compiler version does it show?
2. Open Tool->Board menu. Exactly which CPU is selected?

Also, V1.9 has the FreeIMUv1 defined in it, so maybe it would be best to omit your TRI60.h file and configure your model using the traditional methods (via config.h). BTW, I noticed two "#define ADXL345_ADDRESS 0xA6" in your TRI60.h file so one of them should be deleted.

cool. On mac the about does not state anything useful. But I followed my arduino 022 down into its guts. Guess what I found:

Code: Select all

mcx:bin js$ cat avr-gcc
#!/bin/sh
if readlink "`dirname "$0"`/../etc/options/gcc-version" | grep 3 >/dev/null 2>&1; then
exec -a "$0" "`dirname "$0"`/../avr-3/bin/gcc" "$@"
else
exec -a "$0" "`dirname "$0"`/../avr-4/bin/gcc" "$@"
fi

Now my arduino.app(on mac) is located in a directory 03_Arduino! Seems I have been using avr-3 all the time because of a too simplistic avr script?
Update: just renamed the directory, does not seem make a difference for me. So avr-3 or avr-4 probably does not matter here for my problem.

Board type is set to 'Nano/Atmega328.

About my private #include tri60.h: as I am working on the code way too often to add a feature I have come to using this approach. Benefits are
- can leave config.h untouched
- have each copter's specific settings all in one place
- can switch between my copters by way of changing this one include

I agree this current TRI60.h has become a mess; I pasted stuff over from defs.h for testing purposes.
With current compiler settings, the redefinition does not even produce a warning. The latest define wins. No problems there.

FWIW, it would be helpful to zip up your entire file set and post it here. I can load your exact code into my model and see if I get a GUI hang. Of course I don't use the same IMU board, but I still think it would be helpful to see if the problem can be duplicated.

EDIT: I read elsewhere that you have made improvements to the GUI. So please verify that you are using the unmodified MultiWiiConf_1_9 that was distributed with V1.9. This will help reduce the unexpected surprises.
[/quote]
I concurr, it is good to limit side effects. for the tests I did use the latest official v1.9.
Attachments
MultiWiiConf_1_9.pde.zip
GUI to go with TRI60 sensor problems with v1.9
(10.26 KiB) Downloaded 187 times
TRI60-sensor-problem-MultiWii_1_9.zip
TRI60 sensor problems with v1.9
(56 KiB) Downloaded 188 times

timecop
Posts: 1880
Joined: Fri Sep 02, 2011 4:48 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by timecop »

> does the MWii somehow probe for something using the wmp address, which is now the adxl address and thus creating a conflict?

If my memory is correct, WMP code is ONLY running under if !GYRO, so if you have some other gyro configured none of the WMP code will even be built, including wmp_init() and other related stuff. So, theoretically, it shouldn't be touching anything at that address (looks like 0xA6, or 0x53 as a proper 7bit address).

> swap it against the arduino board for critical examination and benefit from its diagnosis' capabilities.
not really. trying to solve hardware-dependent code on a different platform doesn't make sense. infact, for 99% of the situations, i2c boils down to a simple transfer to an address with read/write bit set or not, and everything else after that (subadress/register/etc) is simply data. having a platform specific implementation of that simply makes everything else work in terms of filling a buffer with data and running the transfer. this is applicable to everything in sensors.pde except BMP085 which uses some ridiculous bus-keeping techniques to do things most likely in the wrong way, but since I don't have that sensor I'm not going to try figuring out what its doing wrong (if anything besides abusing i2c protocol). i'm not a fan of I2C anyway, and nothing that I make uses that bus for anything, so the problem doesn't even affect me.

another unfortunate (or fortunate?) byproduct of only catering to my own requirements when designing things is lack of non-PPM rc input, which automatically rules out probably 99% of multiwii users who will save 0.50 to get a 6channel HK-6A receiver and then spend the difference from a real one on female-female servo cables.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

what comes to mind that my adxl345 has the same address as the default wmp. Even though the adxl345 is configured in, does the MWii somehow probe for something using the wmp address, which is now the adxl address and thus creating a conflict?

In early MWC releases, such a thing could cause a GUI hang because of the methods used to mask bad I2C. But with the latest {Nov 12} V1.9, as long as Arduino/USB power is good, it is VERY unlikely that an I2C corruption will freeze the GUI. For example, now you can short out the SDA / SCL signals, specify wrong chips in config.h, and even have duplicated or missing I2C addresses. The GUI should still run! The only side affect, besides useless data, is long cycle times. And lots and lots of I2C errors will be shown in the debug2 window.

I use the same 0022 Arduino IDE (I think AVR GCC is 4.3.2). I loaded up your files, hoping to duplicate your hang, and unfortunately the GUI ran fine. With all three FreeIMUv1 sensors enabled, cycle times were about 15000 (because I don't have any of your sensors in my installation).

More things to try:
1. Maybe the problem is in the MAC GUI, not the MWC. I suppose a simple test would be to enable the three sensors and observe the status LED. What does it do? If you pitch/roll the model, does the LED ever flash?
2. It would be too much to expect it to arm, but does the model respond (buzzer noise and LED flash) if you perform an ACC or Gyro Cal from the R/C Tx?
3. Just as a sanity test, enable all of your sensors and confirm your GUI hangs. Then remove power, unsolder the FreeIMUv1 from your model, and try it again. Ignoring the horrible data and cycle times, does the GUI run now?

The problem you have is a tough one, but you will beat this.

- Thomas

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

i am away from bench and copters this week. Will try upon return and report back here.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

also I will change the 2.7k pullups against 2.2k.
Should be worthy to try?

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

also I will change the 2.7k pullups against 2.2k. Should be worthy to try?

I wouldn't worry about the resistor values at this point. It will be worth your time to experiment with them after you get the GUI to work.

BTW, the 2.2K value has been proven to be the holy grail for a MWC that uses a 5V I2C buss. But your buss is 3.3V, so an equivalent pullup value would be lower. And as I mentioned before, a 3.3V I2C buss on a 5V CPU is a marginal arrangement and might not be reliable in every situation. But these worries come later; first on the list is to solve the GUI lockup.

BTW, here's some information on I2C pullup resistors and how they affect signal voltage levels and the waveform's rise times:
http://dsscircuits.com/articles/effects ... stors.html

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:More things to try:
1. Maybe the problem is in the MAC GUI, not the MWC. I suppose a simple test would be to enable the three sensors and observe the status LED. What does it do? If you pitch/roll the model, does the LED ever flash?

it takes some time of hectic tail servo movement and alarm blinking (I run LEDs for alarm, not buzzer), eventually system settles down. Then no reaction from tail servo on TX yaw input, also no status LED activity upon model tilting.
2. It would be too much to expect it to arm, but does the model respond (buzzer noise and LED flash) if you perform an ACC or Gyro Cal from the R/C Tx?

cannot do this test as model does not respond to TX anymore.
3. Just as a sanity test, enable all of your sensors and confirm your GUI hangs. Then remove power, unsolder the FreeIMUv1 from your model, and try it again. Ignoring the horrible data and cycle times, does the GUI run now?

So I unsoldered Vcc from the sensor board.system does some init upon powerup, then no reaction on GUI->Start nor on GUI->Read. Alternatively I connected LCD to serial port and try to get telemetry data. but system does not respond to LCD keypress. Also no tail servo movement upon TX yaw input.

I will get a variety of pullup resistors to play with. 2k2, 1k8 and see if that changes anything. If you are not yet tired of all this, I am all open to suggestions.
Thanks, Hamburger

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

it takes some time of hectic tail servo movement and alarm blinking (I run LEDs for alarm, not buzzer), eventually system settles down. Then no reaction from tail servo on TX yaw input, also no status LED activity upon model tilting.

Sounds like the model is completely lost. I think this needs some debug code to see what it is doing during this time. You can write some test code and use the LED to show you what is running.

So I unsoldered Vcc from the sensor board.system does some init upon powerup, then no reaction on GUI->Start nor on GUI->Read.

Disconnect ALL four sensor board wires (Vcc, Gnd, SDA, SCL) and pullup resistors. Does this allow the GUI to work? V1.9 does NOT need any sensors for the GUI to run. So if it refuses to run without the sensors connected (but while they are enabled in the config.h) then that will help direct your troubleshooting efforts.

I will get a variety of pullup resistors to play with. 2k2, 1k8 and see if that changes anything.

Changing the existing 2K7 pullups to 2K2 or 1K8 will not solve the GUI lockup if you are using the latest V1.9 (Nov-12, unmodified). So no need to play with them now; That will come later when you can get the GUI to work.

Have you gone back with the voltmeter to ensure that a the various system voltages are still ok when the model is dead? For example, the 3.3V, and 5V, and D12 outputs of the Arduino board are a good place to check for voltage troubles. It's also a good idea to unplug the servo during troubleshooting so that you can eliminate its load from the power supply.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

not sure I mentioned this before.
With disabling any of acc or magneto (only two active sensors instead of full three), system works. So it is something like 'load' on i2c bus that is crucial. Since I also powered the sensor board via a separate lm1117 3.3V with same results it is most probably not a Vcc power issue.

I have not tried with disabling gyro but guess it would give same result.

fax8
Posts: 61
Joined: Mon Feb 14, 2011 5:29 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by fax8 »

Lower the pullups to 1K.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

not sure I mentioned this before. With disabling any of acc or magneto (only two active sensors instead of full three), system works. So it is something like 'load' on i2c bus that is crucial. Since I also powered the sensor board via a separate lm1117 3.3V with same results it is most probably not a Vcc power issue.

I have not forgotten that. The main thing that contradicts the pullup resistor theory is that Nov-12's V1.9 no longer locks up if the i2c buss is in trouble. Your symptoms are very unusual so that is why I have been trying to encourage you to explore possibilities beyond the I2C pullups. The problem could be caused by the hardware, power supply, or hidden V1.9 software bug that is triggered by your particular installation. That said, if you are interested in trying other pullup values then I'm the last guy that will stop you.

Lower the pullups to 1K.

That is a reasonable choice for this installation's 3.3V i2c buss (would be similar to a 1.5K on a 5V i2c buss). But don't use lower values or the i2c spec will be violated.

- Thomas

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

changed pullups to 1k. No change - works with v1.8; does _not_ work with v1.9 (all sensors enabled)

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

That is sad news, but not unexpected. I still think it would be wise to look at all the other possibilities we've discussed. But mainly I'm here to cheer you on to beat this devil.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:That is sad news, but not unexpected. I still think it would be wise to look at all the other possibilities we've discussed. But mainly I'm here to cheer you on to beat this devil.


thanks for the ongoing support! I really appreciate it.

I just did desolder all 4 wires from FreeIMUv0.1 _and_ enabled all 3 sensors in stock v1.9 MWii. Like you predicted: loop time is in the 5 digit range, tail servo responds to TX yaw commands but jitters (like slowly, with intermittent errors, i2c errors count skyrockets, but overall keeps running and responding.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

I just did desolder all 4 wires from FreeIMUv0.1 _and_ enabled all 3 sensors in stock v1.9 MWii. Like you predicted: loop time is in the 5 digit range, tail servo responds to TX yaw commands but jitters (like slowly, with intermittent errors, i2c errors count skyrockets, but overall keeps running and responding.

It's alive, that is good to hear. I suggest you connect the sensor board's Vcc and Gnd (leave SCL/SDA unconnected). But leave the two pullups (use 1K5 sourced from 3.3V) connected to the CPU's SLC/SDA. Does the GUI still run for several minutes?

Can you post a clear close up photo of your hardware that shows the CPU and sensor board?

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

now it is getting really weird. Bear with me.
The original FreeIMUv1 is desoldered. I want to try different sensor board FreeIMUv035MS, so
_no_ sensor board soldered, configure in FreeIMUv035MS, upload -> hangs!

Now this does contradict the previous experience where it did run with all sensors of FreeIMUv1 enabled but no board soldered.

To continue I manually configure in only gyro and acc of FreeIMUv035MS -> system runs, servo responds to TX yaw, GUI can connect and display i2c error count (no sensor graphs, because no sensor attached.)

So I add the baro sensor to config and upload -> system runs, servo responds to TX yaw input, but GUI connect -> hangs.

So I configure gyro+acc+magneto, upload -> system hangs
Try again, this time runs after upload, can connect GUI, runs, display ok.

It seems when I get reset with servo movement after upload, chances are it will run afterwards.
---
all this was without sensor board, just sensors configured in.
Contradicting:
a)
in former post 3 sensors configured in, no sensor board soldererd -> runs (so it is the sensor board?)
here 3 sensors configured in, no sensor board soldererd -> hangs (not the sensor board?!?)

b)
same config 3sensors two times - once runs, other time hangs (random, ouch)

Now I do not really know what to try next.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

saga continues:
no sensor board, eeprom cleared, lots of stuff disabled (lcd, telemetry), 4 sensors enabled (binary file way smaller), upload -> runs!
can connect with GUI, ->Start, see 3 sensors with green background (ok, no visual for gyro, so that equals my 4 sensors), hangs!

before and after voltages are 3.3V and 5V with multimeter.

If it did not hang I would have thought it was the sketch size. But with the hang upon connect?
So maybe it is with this arduino? Well, I did solder everything directly to the nano's pins; swapping the nano against other will take time.
Other ideas, please?

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

no sensor board, eeprom cleared, lots of stuff disabled (lcd, telemetry), 4 sensors enabled (binary file way smaller), upload -> runs!
can connect with GUI, ->Start, see 3 sensors with green background (ok, no visual for gyro, so that equals my 4 sensors), hangs!

Is the MWC still running when the GUI hangs? This may require some debug code to blink the LED (not much work to do that).

When the GUI hangs, does pressing the CPU reset button cause the GUI to work for a moment, then hang again? Or does the GUI not show any life when you try the reset resuscitation?

Does it work better if you unplug all the servos and repeat the tests?

Do you only have a MAC? Any chance you have access to a PC with Win7?

What sensor chips are soldered on your 035MS board? {visually inspect, don't trust the config.h}. I know it is not connected, but let's confirm what you have in case it matters later on.

before and after voltages are 3.3V and 5V with multimeter.

Where exactly are you measuring these? For example, there is a tiny 5.0V VReg on the Arduino that must be checked (you can use pin D12 as a more convenient substitute for measuring it). And 3.3V should be found on the USB chip (it has an internal Vreg) / this voltage can be measured on the unused "3.3V" pin of your Arduino CPU board because I recall yours has the USB chip on it.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

A short detour when you have a moment:
Please substitute V1.9's Output.pde with V1.8p2's Output.pde. Try with both sensor board definitions (but with the sensor boards disconnected). With this hack, do the symptoms change in ANY way? {Even if the GUI does not work, does the model continue to respond?}
Restore the original file after the test.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

no sensor board; old v1.8 output.pde.
Works with both definitions. Can connect to GUI, see graphs, can start/stop/read.
attached the output.pde that works.
A first diff against the v1.9 version shows formatting, and A0_A1_PIN_HEX pin5 pin6 handling.
Hm, not sure what this does exactly.

As next step I will solder the old sensor board and test again.
Attachments
Output.pde.zip
the v1.8 variant of output.pde used for testing.
(3.64 KiB) Downloaded 103 times

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

no sensor board; old v1.8 output.pde.
Works with both definitions. Can connect to GUI, see graphs, can start/stop/read.
attached the output.pde that works.

That is very promising. Hopefully this leads us down a useful path. So I'm looking forward to some good news when you install the sensors. But ...

I am very nervous about your 3.3V I2C buss (not the best choice for a 5V CPU), so we may have some hardware to fix. Fortunately, V1.9's debug2 error inspection value will help us validate your I2C's reliability. For external pullups, I expect 1K5 to 1K8 will be fine. But if you have 2K2's on it now then try them.

The output.pde is where all the serious interrupt code is handled. There was some code cleanup and minor updates that were made to it in V1.9; Maybe we now have an interrupt vector handler that is being a bad boy. This would certainly explain all the unexplained problems. Interrupts are an awesome way to deal with external I/O, but nasty little devils when the code is not perfect.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

runs for minutes with FreeIMUv1 attached, no i2c errors.

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

runs for minutes with FreeIMUv1 attached, no i2c errors.

You made my day! Congrats.

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

mr.rc-cam wrote:You made my day! Congrats.


YOU made my early night! A big thank you.
Just what made you think of that?

I am all for exploring further, but I do not have a clue how to proceed. Waiting for input?

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

Just what made you think of that?

You started to report a bunch of unrelated symptoms. I knew it wasn't an I2C problem, but I had a hunch it was power supply or code related (hence all the endless and strange questions from me). Interrupt bugs are notorious for weird operational issues. Given that the Output.pde is the mother of all interrupt handling in the MWC, and you had success with V1.8, it seemed like it was time to skip ahead and try the file swap hack. From there it was all a matter of good luck.

I am all for exploring further, but I do not have a clue how to proceed. Waiting for input?

I quickly looked over the Nov-12 V1.9's output.pde and cannot see anything obvious that caused the problem. So perhaps the compiler is doing something weird to the code (I've run into that a few times over the years). I think the best thing to do is to undo the V1.9's output.pde code changes, one line at a time, until it breaks your copter. That trick will sometimes provide a hint to what went wrong. It shouldn't take more than 30 minutes to do this, so not a bad thing to try.

The source of all this frustration needs to be identified and permanently fixed. Otherwise some other dude will be back here with equally strange issues to fix. And everything we can do to solve the bugs will give Alex more time for some sleep. :)

- Thomas

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

culprit seems to be in the function void logMotorsPower() . So it is not interrupt handling, maybe?

Changing

Code: Select all

      amp = amperes[(motor[i] - 1000)>>4] / (uint32_t) vbat; // range mapped from [1000:2000] => [0:1000]; then break that up into 64 ranges; lookup amp

into

Code: Select all

      amp = 12345L; 

seems to fix the hangs - but breaks the logging. Not sure what is wrong here.
anyway, time to sleep now.

ziss_dm
Posts: 529
Joined: Tue Mar 08, 2011 5:26 am

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by ziss_dm »

Hi Hamburger,

No free SRAM left, maybe? ;)

regards,
ziss_dm

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

No free SRAM left, maybe?

That is high on the list too. Does the Arduino tool set offer a convenient way to check ram usage?

- Thomas

ziss_dm
Posts: 529
Joined: Tue Mar 08, 2011 5:26 am

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by ziss_dm »

Hi,

Not a convinient, but you can use avr-size.exe utility

Code: Select all

C:\Users\xxx\AppData\Local\Temp\build8362760963851104549.tmp>"C:\Program Files (x86)\arduino-0022\hardware\tools\avr\bin\avr-size.exe" -C MultiWii.cpp.elf


Code: Select all

AVR Memory Usage
----------------
Device: Unknown

Program:   22016 bytes
(.text + .data + .bootloader)

Data:       1992 bytes
(.data + .bss + .noinit)


with:
#define ITG3200
#define BMA180
#define BMP085
#define HMC5843
#define POWERMETER 1

And this is only static data..

regards,
ziss_dm

mr.rc-cam
Posts: 457
Joined: Wed Jul 27, 2011 11:36 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by mr.rc-cam »

Data: 1992 bytes
(.data + .bss + .noinit)

Yikes, perhaps we've grown too big for our pants!

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

good finding!

with original code I get

Code: Select all

avr-size -C MultiWii_1_9.cpp.elf 
AVR Memory Usage
----------------
Device: Unknown

Program:   21386 bytes
(.text + .data + .bootloader)

Data:       1911 bytes
(.data + .bss + .noinit)


whereas with the simple replacement amp=0L this gives

Code: Select all

avr-size -C MultiWii_1_9.cpp.elf 
AVR Memory Usage
----------------
Device: Unknown

Program:   20986 bytes
(.text + .data + .bootloader)

Data:       1655 bytes
(.data + .bss + .noinit)


Quite a difference? What are the limits for the little arduino?

ziss_dm
Posts: 529
Joined: Tue Mar 08, 2011 5:26 am

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by ziss_dm »

Hi,

atmega328 - 2k

regards,
ziss_dm

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

arduino specs say:
Flash Memory 16 KB (ATmega168) or 32 KB (ATmega328) of which 2 KB used by bootloader
SRAM 1 KB (ATmega168) or 2 KB (ATmega328)
EEPROM 512 bytes (ATmega168) or 1 KB (ATmega328)

so 2KB probably means 2*1024 =2048 bytes. While 1911 is pretty close it is still lower. What else might grab sram at runtime that the compiler does not take into account when computing the Data: size?

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

ziss_dm wrote:atmega328 - 2k

ah ziss_dm,
you are always faster than me.
Thanks, Hamburger

ziss_dm
Posts: 529
Joined: Tue Mar 08, 2011 5:26 am

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by ziss_dm »

Dynamic memory, memory manager, stack, local varibles, etc...

User avatar
Hamburger
Posts: 2578
Joined: Tue Mar 01, 2011 2:14 pm
Location: air
Contact:

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by Hamburger »

any chance to figure out the amount of mem all that takes?

ziss_dm
Posts: 529
Joined: Tue Mar 08, 2011 5:26 am

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by ziss_dm »

Hi
Not really... But reserving 200-300 bytes is usually good idea.

regards,
ziss_dm

User avatar
shikra
Posts: 783
Joined: Wed Mar 30, 2011 7:58 pm

Re: v1.9 hang with Magneto of FreeIMUv1 enabled - i2c relate

Post by shikra »

Good catch guys. Easy to forget as bits get added over time...

Post Reply