Page 1 of 1

What's all this I2C failure stuff, anyhow?

Posted: Fri Dec 09, 2011 1:45 am
by mr.rc-cam
There have been sensor related problems reported after upgrading to V1.9. A common reaction from the users is that there is a software problem in the new release. But I would like to offer another perspective on this. It is more likely that your V1.9 problem is NOT due to a new software bug. I'm not implying that V1.9 is perfect. But some of the reported sensor issues are caused by long standing hardware problems that have been ignored.

Wait a minute, you say. You've used several different versions and everything has been fine; Your MWC flew perfectly, until V1.9! So you are confident that there's no way your model has a hardware problem. You may be surprised to hear that the solution involves fixing your I2C related hardware. That's right, some of us have discovered that the existing I2C wiring techniques are part of the problem.

A little background information:

In the earlier MWC software releases the I2C communication was not 100% reliable in every installation. But the I2C problems weren't noticed because software masked them. On installations with these hidden problems the model's stability and drift performance can be affected (sometimes not even noticed by the pilot). No doubt this has caused Alex some headaches; Throughout the various releases, users have been reporting random performance problems that kept him busy providing support and experimental software fixes. But, the software can only do so much when the hardware has problems.

A bit of MWC history:

During early TriCopter development Alex had problems with some variants of the WMP board. Sometimes the WMP would occasionally fail to respond to I2C R/W access. Part of this may have been caused by how the start/stop bits were compacted together during I2C communication. Regardless of why it occurred, some I2C software tricks were incorporated to mask the issue. For example, the Arduino's D12 output pin was used to Hard Power-Reset the WMP board. It's also the reason for the neutralizeTime task, which temporarily tosses out sensor data after each I2C reset event. Although the resets can occur often, you cannot detect them. But they can cause reduced stability and/or more drift.

In addition, to simplify our installations there were some tricks to reduce the project's parts count. For example, some installations used the high value (20K-50K ohm) internal/CPU I2C pullups instead of carefully chosen external resistors. Also, some installations had 3.3V sensors that eliminated the LLC board (5V->3.3V Logic Level Converter) by using external pullups that were sourced by 3.3V. Hardware tricks like these are gloriously simple; Unfortunately they are tempermental and not reliable for everyone.

Long story short, the older software was allowing models to fly that actually had problems. Some may think that this is acceptable but I doubt everyone wants to continue flying models that are suffering from reduced performance.

Then came V1.9:

Things changed for the better when Alex released V1.9. Unfortunately, there are three different V1.9 releases that have identical version names. These three V1.9's are as follows:

Release 1: Introductory release, Version 1.9 {Nov-06}. It had a bug in the I2C code that caused a fatal CPU lockup in some installations. It occurred ONLY if the I2C buss had problems. Some models would not boot and others would run for awhile, then lockup. Like some of you, I experienced the boot lockup, which after some troubleshooting resulted in the introduction of the I2C error counter in the GUI: viewtopic.php?f=8&t=884

Release 2: Emergency patch, Version 1.9 {Nov-11} was released which fixed the I2C lockup problem. But more importantly, Alex reworked the I2C code and he was able to solve the old WMP problems. The need for the D12 Hard Reset was finally eliminated. But this version is missing the important I2C Error count value in the GUI.

Release 3: Emergency patch, Version 1.9 {Nov-12} was released which added the I2C error counter in the GUI (debug2 location). The value will increment with each I2C error. At about 32K errors it will become negative and count backwards.


Which brings us to the purpose of this i2c discussion:

With the third V1.9 release the I2C related software was vastly better than before. But that did not mean everything was solved because some installations have I2C wiring that needs improvement. Fortunately, with the I2C error counter in the GUI, the hardware problems can be detected and fixed. The Debug2 value is all you need to troubleshoot V1.9, fix the I2C, and improve your model's performance. So please use the Debug2 feature!

I2C Hardware Fix List / RC-CAM's i2c Recipe (subject to revision)
------------------------------------------------------------------------------
Here are some of my suggested changes to solve I2C problems. These have helped many installations. Don't cheat, do them all!
  • Use the V1.9 release that is dated NOV-12. Not sure which one you have? Then download it again from the web and re-install it.
  • The recommended Arduino CPU's are 5V devices. So for reliable operation your I2C pullups on the CPU must be sourced from 5V, not 3.3V. Installations with 3.3V sensors will need an LLC (to prevent damaging the sensor).
  • Disable internal pullups and use external 2.2K ohm pullups. Higher pullup values may not work for you so I recommend 2.2K ohm. Please note that some MWC shields have pre-installed pullups sourced by 3.3V. These must be disabled before you can use the 2.2K pullups sourced by 5V.
  • 5V compatible sensors should be powered directly from 5V. Do not use the D12 signal and remove any "voltage drop" diodes.
  • Observe the GUI's debug2 window. Verify that this value remains zero during a long (>10 minutes) bench test. Repeat for at least one minute with the motors running. If the value is not zero at the end of the tests then you have I2C wiring or sensor related problems.

Keep in mind that the I2C error counter is a primitive debugging tool and will not report *every* I2C problem that may occur. However, if you follow the I2C Hardware Fix list you should be able to eliminate most of the potential problems. Of course some of you are going to report that you ignored the list and all is well. Consider yourself blessed; for everyone else I suggest following the recipe. :)

Related topics with useful information to solve I2C problems:
viewtopic.php?f=8&t=872
viewtopic.php?f=8&t=884
viewtopic.php?f=6&t=935
viewtopic.php?f=8&t=932
viewtopic.php?f=8&t=887
viewtopic.php?f=8&t=942
http://dsscircuits.com/articles/effects ... stors.html

By the way:

Yes, I have seen the "ram shortage" discussion. If I have not convinced you that your I2C problem is due to your hardware, then please feel free to run the avr-size tool and confirm you have sufficient ram. Honestly, if you have a ram shortage your problems will be much more devious than some I2C errors in debug2.

For those that have "tried everything" and still see I2C errors in Debug2, then don't despair. Post an *accurate* schematic of your MWC and perhaps we can offer some advice. Please understand that Without a clear and concise schematic (of YOUR model) useful help will likely be impossible.

- Thomas

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Dec 09, 2011 11:21 am
by UndCon
This is a good sum-up.

To add to things that seems to work with current(latest) code... (all my boards are powered by separate UBEC)

A common hardware list:
5V Arduinos, Genuine WM+ & NK's

http://www.multiwiicopter.com
Both of my Paris v.4 boards deliver 0 errors (1 board has a BMP085 installed)

http://www.flyduino.net
Flyduino WMC tested yesterday gave lots of errors but I used 3.3 instead of D12 on the shield by mistake, will resolder to D12 and try again.

www.viacopter.eu (Jussi's shields)
1.0
1.1 (sold)
1.3BT
(none tested due to lack of sensors but both 1.0 and 1.3 are populated with arduino's)

//UndCon

Re: What's all this I2C failure stuff, anyhow?

Posted: Sat Dec 31, 2011 4:18 am
by jessestr
Thanks for telling us.

I got that problem. I got the latest build from 20 december, aduino is powered from 5 volt, every sensor is powered from 5V i'm using an original WM+ and a BMA020 from Paul, now he said I just got to use the internal pullups. So I did, it worked in 1.8p2, but then I tried 1.9 it didnt work, then I saw that use internal pullups was commented in 1.9 so I activated it, it worked !

After that I disconnected everything to try my quadcopter in my room. Connected the battery and the led Of the arduino stayed on.. So I went to my computer to check, and it gave I2C errors again, and a few minutes therefore it worked.. What the!? So I'm stuck with my dirst new quad, reading topics about the problem but no decent solution with photo's or something else, please could you show me how to use the pullups from what point to what point they have to be soldered? Or another working solution?

I spent over 300 euros for this quad so I really dont wan't to just let it go..

Please help, I asked this so much already on different forums! If you need photos ask!

Re: What's all this I2C failure stuff, anyhow?

Posted: Sat Dec 31, 2011 5:04 am
by mr.rc-cam
Please help, I asked this so much already on different forums! If you need photos ask!

Information for solving common I2C problems are in the first post. But if the information has not helped then it will be necessary for you to post an accurate schematic of YOUR wiring/circuitry. All the little details matter so make sure the drawing is accurate.

Re: What's all this I2C failure stuff, anyhow?

Posted: Sat Dec 31, 2011 12:17 pm
by jessestr
Image

There you go

Re: What's all this I2C failure stuff, anyhow?

Posted: Sat Dec 31, 2011 6:59 pm
by mr.rc-cam
The block diagram that was posted does not provide the details I would need to offer advice beyond what was discussed in the first post (I need a detailed schematic). But maybe someone else that is familiar with your commercially made MWC boards can offer some advice.

Before you take the time to create a detailed schematic, I suggest you thoroughly read the first post a few more times. Information on solving common I2C problems are explained in it; If you perform ALL the suggestions, and the I2C problem is not solved, then that would mean you have an unusual problem that will involve review of your circuitry/wiring. To provide the additional help we would need to see the detailed schematic of your MWC.

Re: What's all this I2C failure stuff, anyhow?

Posted: Mon Jan 02, 2012 12:51 pm
by Th0rsten
Do you get reading from ACC but not from the Gyro? Sounds to me like the same problem I have:
viewtopic.php?f=15&p=7302#p7302

Re: What's all this I2C failure stuff, anyhow?

Posted: Mon Jan 02, 2012 7:22 pm
by mr.rc-cam
Th0rsten wrote:Do you get reading from ACC but not from the Gyro? Sounds to me like the same problem I have:
viewtopic.php?f=15&p=7302#p7302

I read that you found your board had the wrong value I2C pullup resistors (a common problem). Glad to hear you solved it.

Re: What's all this I2C failure stuff, anyhow?

Posted: Mon Jan 02, 2012 8:06 pm
by Th0rsten
To be correct: The board from Paul doesn't have any pull-ups than the internal ones but should have 2k2 external pull-ups.

Re: What's all this I2C failure stuff, anyhow?

Posted: Tue Jan 03, 2012 8:46 pm
by Ben_der
Thanks for this !
This helped a lot !

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Jan 05, 2012 12:47 pm
by Th0rsten
Here is a nice text on effects of varying pull-ups on the i2c:
http://www.dsscircuits.com/articles/eff ... stors.html

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Jan 05, 2012 5:47 pm
by mr.rc-cam
Here is a nice text on effects of varying pull-ups on the i2c:
http://www.dsscircuits.com/articles/eff ... stors.html

Thanks. That link is also listed in the first post under the "Related topics with useful information to solve I2C problems" section. It demonstrates why many installations have I2C errors. It is good information and worth repeating.

I was hoping the I2C problems would be under control at this point. But I still see posts at the various R/C forums where a user is reporting problems after upgrading to V1.9 and they have no idea why. In nearly every case their screenshot shows debug2 error counts. So maybe the GUI needs to prominently display the I2C error count rather than hide it at the bottom of the screen. The FAQ should mention the I2C error feature too.

- Thomas

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Jan 05, 2012 6:34 pm
by Th0rsten
Hi mr.rc-cam

Made a test on the i2c on my new Mega with WM+ and BMA020.

This picture shows SCL only with the BMA020 10k Pull-up.
This is not working.
IMAG002.png
(4.27 KiB) Not downloaded yet

This picture shows SCL with the internal 20k Mega and 10k BMA020 pull-up = 6k7
This is ok for 1.8.2 but not for 1.9
IMAG004.png
(4.2 KiB) Not downloaded yet

This picture shows SCL with external 2k2 pull-ups and the 10k from the BMA020 = 1k8
This is completely ok.
IMAG003.png
(3.98 KiB) Not downloaded yet

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Jan 05, 2012 8:36 pm
by mr.rc-cam
Made a test on the i2c on my new Mega with WM+ and BMA020.

Thanks for the scope measurements. Another eye-opener is to put your scope probe on a MWC that uses 3.3V sensors without an LLC. A watchful signal observer will see that regardless of the 3.3V sourced pullup value, they don't provide the proper I2C levels for reliable use with the 5V powered CPU. Details to this mixed logic level scenario are explained in table 29-15 (Two-wire Serial Bus Requirements) in the ATMEGA328 manual.

Just to make things perfectly clear to those following this discussion, the I2C problem is NOT caused by V1.9. Installations that worked on prior versions, but not on V1.9, had the problem all along. And the problem reduced PID performance.

FWIW, V1.9's I2C error feature is just telling us what we should have known long ago. I honestly think it would have saved Alex some gray hairs if MWC builders knew this a year ago.

- Thomas

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Jan 20, 2012 4:48 pm
by copterrichie
I just wanted to confirm adding 2.2k pullups or correcting their values does work. I added two 2.2k pullups to my bench testing united that did not work with 1.9 before and now works flawlessly. Zero Debug errors where as before, uncountable.

Thank you.

Image


Image

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Mar 01, 2012 11:37 am
by Rick
Hi
I've been reading through this trying to make sense of it. I've asked for some help on the Paris forum on RCG but no replies yet so thought I'd ask here.

This is where I'm at.

Had two boards (Paris V3) with genuine WM+ and NK running ok. I upgraded to the latest 1.9 and it seemed ok. No debug errors noticed on short tests.

I also decided to update my esc's so flashed them with the SimonK update. Put them on the tri and it flew, but was a bit twitchy, so I thought I'd put it back on the UI to see what I could tweak. But now the board won't run up the gyro or acc on the gui. I can write and read to it. I reuploaded a completely fresh 1.9 through the arduino and tried again - same thing.

The UI is now showing debug2 errors so there are obviously I2C problems. Now forgive me - I have red through as much of this and related posts as I can but I get lost it in it all. My questions at this point would be - how do I find out what the debug errors actually are - do they get logged anywhere? Where can I actually read and understand about pull ups? I can never seem to understand what there role is and whether or not I need to do anything with them?

I am running a standard Paris V3 board (no solder changes on the board) with WM+ and NK (both genuine). With the software as standard - I've only changed it to be TRI - nothing else.

With this recent recognition of I2C errors etc I'm trying to figure out if I need to change something to do with pullups now?

So confused - any advice would be gratefully received.

thanks for all your excellent work and for sharing with us all.

Rick

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Mar 01, 2012 9:12 pm
by mr.rc-cam
Had two boards (Paris V3) with genuine WM+ and NK running ok. I upgraded to the latest 1.9 and it seemed ok. No debug errors noticed on short tests.
{snip}
But now the board won't run up the gyro or acc on the gui.

Since it was working, the newly observed failure may be a bad solder connection or intermittent wiring issue. So time to check over your sensor wiring with a magnifying lens and fresh eyes.

The UI is now showing debug2 errors so there are obviously I2C problems. Now forgive me - I have red through as much of this and related posts as I can but I get lost it in it all. My questions at this point would be - how do I find out what the debug errors actually are - do they get logged anywhere?

The debug2 error reports that the communication with the sensor has failed because a data acknowledgement was not received by the CPU. Every time the count changes another failed communication has occurred. There's no formal log with details about the specific error. There's no way for the software to know why it failed, that part is up to your troubleshooting. For example, it could be a bad solder connection, wiring error, bad sensor, power supply issue, pullup resistor problem, and the list goes on.

Where can I actually read and understand about pull ups?

http://www.dsscircuits.com/articles/eff ... stors.html

I can never seem to understand what there role is and whether or not I need to do anything with them?

That is the reason for this topic. The first post summarizes the issue and explains how to deal with the I2C problems that are caused by poorly chosen pullup values.

With this recent recognition of I2C errors etc I'm trying to figure out if I need to change something to do with pullups now?

First confirm that your board is not damaged and has no wiring errors by reverting back to an earlier MWC version that was known to work for you. If the GUI/sensors work with the old MWC version, but reports I2C errors in V1.9, then it is a symptom that your pullup values are wrong. So if that is the case, then to use V1.9 my recommendation is to do all the things discussed in the first post under the "I2C Hardware Fix List / RC-CAM's i2c Recipe" section.

- Thomas

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Mar 01, 2012 10:09 pm
by Rick
Thomas
thank you for your reply - I'm very grateful.

I tried to read through all that stuff but I got lost in the threads somewhere. I'll try again. I't seems odd that both boards are doing the same thing so that's what made me wonder if I needed to change something in the settings?

I have since disabled the internal pullups in the code - and soldered the bridge that connects the lines between A4 and SDA and A5 and SCL. It still is the same.

If I do this do I need to stop taking the feed from pin 12 and take it form somewhere else?

I'll go back through and try it all again - I've taken off the NK to eliminate that for the moment but I'll recheck the soldering on the other bits.

Sorry for not grasping this stuff too well. I try, but the threads cover so much ground so quickly - when there are 22 pages of it I'm not sure which bits I should be trying and which bits are not relative?

I always appreciate any assistance though - so thank you for your time. It is very much appreciated.

Rick

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Mar 01, 2012 10:45 pm
by mr.rc-cam
If both boards are failing, and you are sure that the wiring is correct, then thoroughly re-read post #1 again + several more times. Read it until it the problems/solutions are clear to you.

Regarding what to do, it is summarized in the Fix List section, reprinted as follows:
I2C Hardware Fix List / RC-CAM's i2c Recipe (subject to revision)
------------------------------------------------------------------------------
Here are some of my suggested changes to solve I2C problems. These have helped many installations. Don't cheat, do them all!

1. Use the V1.9 release that is dated NOV-12. Not sure which one you have? Then download it again from the web and re-install it.
2. The recommended Arduino CPU's are 5V devices. So for reliable operation your I2C pullups on the CPU must be sourced from 5V, not 3.3V. Installations with 3.3V sensors will need an LLC (to prevent damaging the sensor).
3. Disable internal pullups and use external 2.2K ohm pullups. Higher pullup values may not work for you so I recommend 2.2K ohm. Please note that some MWC shields have pre-installed pullups sourced by 3.3V. These must be disabled before you can use the 2.2K pullups sourced by 5V.
4. 5V compatible sensors should be powered directly from 5V. Do not use the D12 signal and remove any "voltage drop" diodes.
5. Observe the GUI's debug2 window. Verify that this value remains zero during a long (>10 minutes) bench test. Repeat for at least one minute with the motors running. If the value is not zero at the end of the tests then you have I2C wiring or sensor related problems.


Before making any hardware changes be sure to re-load an earlier {working} software version and confirm your two boards works with it. If the earlier version does not work then you have hardware problems. This is all part of the troubleshooting process so don't skip this step.

I have since disabled the internal pullups in the code

If you needed internal pullups enabled in previous versions in order for it to work then that suggests that you are missing the required external pullups (or have incorrect pullup values). See the Fix list #3 above.

and soldered the bridge that connects the lines between A4 and SDA and A5 and SCL. It still is the same.

You'll need to provide more details about the "bridge." If it's 3.3V pullup bridge pads on the the Paris board then such things must NOT be enabled (because you need 5V sourced pullups). See the Fix list #2 above.

If I do this do I need to stop taking the feed from pin 12 and take it form somewhere else?

See Fix List #4.

I've taken off the NK to eliminate that for the moment

Good idea, Simplify the installation.

the threads cover so much ground so quickly - when there are 22 pages of it I'm not sure which bits I should be trying and which bits are not relative?

Then just observe the bits discussed in this thread's Page 1/ Post #1.

If it is still not clear, and you need technical help, then post an accurate schematic of your Paris board and its sensors. If the board was built/assembled by the vendor then ask them for a detailed drawing and post it here. If you built it then you should be able to create a drawing of EVERYTHING you have wired up. The drawing needs to be complete and show the CPU wiring to/from the sensors, sensor power, and include the LLC if installed.

- Thomas

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Mar 01, 2012 11:29 pm
by Rick
thanks once again Thomas

I will endeavour to go over this again and again until it makes sense.

thank you for your patience.

Rick

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Mar 16, 2012 3:52 pm
by aidensere07
Iv just been reading throught this thread, i just upgraded to v2 preversion 2 and i get lots of i2c errors, i can see that people say i need to add a pullup resistor to the flydusense v1.2 as shown in this picture
Image

is this correct? should i place them exactly where they are in the picture, or somewhere else, and can someone confirm the values i should be using in which place or are they the same value?

thanks for your help
Aiden

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Mar 16, 2012 7:24 pm
by Th0rsten
All the pull-up resistors in sum should be near 2,2 kOhm. R sum = 1/(1/R1 + 1/R2)
I disabled the internal pull-ups and placed the resistors the way you can see on my picture.
Of course you can place the pull-ups everywhere between VCC and SDA and VCC and SCL.
For me the way on the pic was the easiest.

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Mar 16, 2012 8:23 pm
by aidensere07
Thanks for that mate and no traces need to be cut if u do it exactly like your picture? Just trying to get my head round it. What do you mean by the sum of the 2, that well and truely puzzled mek

Sorry to be a pain, thanks
Aiden

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Mar 16, 2012 9:49 pm
by Th0rsten
Even with disabling the internal pull-ups there are also the ones from the BMA.
So in parallel with the 2k2 you get 1,8 kOhm. This is still ok - you don't need to cut any traces.
You can of course disable the pull-up pin from the BMA by cutting it.

Connect the resistors the same way as shown on the picture and the errors are gone.

Re: What's all this I2C failure stuff, anyhow?

Posted: Fri Mar 16, 2012 10:21 pm
by aidensere07
Yeah I see, I'm just going to nip down to maplins and grab a few 2k2 resistors in the morning then and solder as you have

Thanks very much for taking the time to help me out
Aiden

Re: What's all this I2C failure stuff, anyhow?

Posted: Sat Mar 17, 2012 4:09 pm
by aidensere07
Th0rsten wrote:Even with disabling the internal pull-ups there are also the ones from the BMA.
So in parallel with the 2k2 you get 1,8 kOhm. This is still ok - you don't need to cut any traces.
You can of course disable the pull-up pin from the BMA by cutting it.

Connect the resistors the same way as shown on the picture and the errors are gone.


just soldered on the resistors and it works, no more i2c errors before it was counting like crazy now, been running 10mins and nothing, zero, zilch.
hopefully this may help my twitchy copter

http://vimeo.com/38685229

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Sep 05, 2013 10:23 am
by natael
Hi,

I'm trying to fix some i2c issues after trying to connect a baro bmp085. I have the V3 board for Pro Micro (not mini) from flyduino (see picture of both sides), with wmp+ and ACC nunchk (they alone never caused any i2c issues though), and i'd like to follow the recommended wiring/setup exposed in this thread.
IMG_2874.jpg

IMG_2874.jpg


After reading this thread i'd like to set the 2k2 pull ups, as i was using the internal ones so far. I'm not sure of what i'm doing with this board : am i meant to solder 2k2 resistor between the SDA and VCC, and another one between SDL and VCC, whose deported pins can be seen on the right of the first picture ?

I have 3k3 resistor lying around, and it might be long to get 2k2 resistors (no shops around). 3k3 resistor won't do the job, would they ?

On the 2nd picture, there are 5v and 3.3v pull up. I'm reading in this thread that if using 5v sensor makes unnecessary to use LLV, is that correct ? My bmp085 can be powered by both 5V and 3.3V (http://www.drotek.fr/shop/en/home/11-bm ... ensor.html). Will it be enough to power things with my 5v pro Micro and these external pull ups soldered as asked ? Am i meant to simply solder the 5v patch with the one in the middle, or should i leave it as it is ?

Thank you

Re: What's all this I2C failure stuff, anyhow?

Posted: Thu Sep 05, 2013 1:18 pm
by natael
I finally found 2k2 resistors and tried to guess according to the almost-invisible tracks from the pull-up patches up to the Pro Micro pins, in order to find how to position the resistors :

IMG_2876.jpg


I re-commented the internal pull ups in config.h, uncommented the bmp085 and all problems were solved ! I've been trying to fix that problem for days !
Very useful thread.

Cheers