What's all this I2C failure stuff, anyhow?
Posted: Fri Dec 09, 2011 1:45 am
There have been sensor related problems reported after upgrading to V1.9. A common reaction from the users is that there is a software problem in the new release. But I would like to offer another perspective on this. It is more likely that your V1.9 problem is NOT due to a new software bug. I'm not implying that V1.9 is perfect. But some of the reported sensor issues are caused by long standing hardware problems that have been ignored.
Wait a minute, you say. You've used several different versions and everything has been fine; Your MWC flew perfectly, until V1.9! So you are confident that there's no way your model has a hardware problem. You may be surprised to hear that the solution involves fixing your I2C related hardware. That's right, some of us have discovered that the existing I2C wiring techniques are part of the problem.
A little background information:
In the earlier MWC software releases the I2C communication was not 100% reliable in every installation. But the I2C problems weren't noticed because software masked them. On installations with these hidden problems the model's stability and drift performance can be affected (sometimes not even noticed by the pilot). No doubt this has caused Alex some headaches; Throughout the various releases, users have been reporting random performance problems that kept him busy providing support and experimental software fixes. But, the software can only do so much when the hardware has problems.
A bit of MWC history:
During early TriCopter development Alex had problems with some variants of the WMP board. Sometimes the WMP would occasionally fail to respond to I2C R/W access. Part of this may have been caused by how the start/stop bits were compacted together during I2C communication. Regardless of why it occurred, some I2C software tricks were incorporated to mask the issue. For example, the Arduino's D12 output pin was used to Hard Power-Reset the WMP board. It's also the reason for the neutralizeTime task, which temporarily tosses out sensor data after each I2C reset event. Although the resets can occur often, you cannot detect them. But they can cause reduced stability and/or more drift.
In addition, to simplify our installations there were some tricks to reduce the project's parts count. For example, some installations used the high value (20K-50K ohm) internal/CPU I2C pullups instead of carefully chosen external resistors. Also, some installations had 3.3V sensors that eliminated the LLC board (5V->3.3V Logic Level Converter) by using external pullups that were sourced by 3.3V. Hardware tricks like these are gloriously simple; Unfortunately they are tempermental and not reliable for everyone.
Long story short, the older software was allowing models to fly that actually had problems. Some may think that this is acceptable but I doubt everyone wants to continue flying models that are suffering from reduced performance.
Then came V1.9:
Things changed for the better when Alex released V1.9. Unfortunately, there are three different V1.9 releases that have identical version names. These three V1.9's are as follows:
Release 1: Introductory release, Version 1.9 {Nov-06}. It had a bug in the I2C code that caused a fatal CPU lockup in some installations. It occurred ONLY if the I2C buss had problems. Some models would not boot and others would run for awhile, then lockup. Like some of you, I experienced the boot lockup, which after some troubleshooting resulted in the introduction of the I2C error counter in the GUI: viewtopic.php?f=8&t=884
Release 2: Emergency patch, Version 1.9 {Nov-11} was released which fixed the I2C lockup problem. But more importantly, Alex reworked the I2C code and he was able to solve the old WMP problems. The need for the D12 Hard Reset was finally eliminated. But this version is missing the important I2C Error count value in the GUI.
Release 3: Emergency patch, Version 1.9 {Nov-12} was released which added the I2C error counter in the GUI (debug2 location). The value will increment with each I2C error. At about 32K errors it will become negative and count backwards.
Which brings us to the purpose of this i2c discussion:
With the third V1.9 release the I2C related software was vastly better than before. But that did not mean everything was solved because some installations have I2C wiring that needs improvement. Fortunately, with the I2C error counter in the GUI, the hardware problems can be detected and fixed. The Debug2 value is all you need to troubleshoot V1.9, fix the I2C, and improve your model's performance. So please use the Debug2 feature!
I2C Hardware Fix List / RC-CAM's i2c Recipe (subject to revision)
------------------------------------------------------------------------------
Here are some of my suggested changes to solve I2C problems. These have helped many installations. Don't cheat, do them all!
Keep in mind that the I2C error counter is a primitive debugging tool and will not report *every* I2C problem that may occur. However, if you follow the I2C Hardware Fix list you should be able to eliminate most of the potential problems. Of course some of you are going to report that you ignored the list and all is well. Consider yourself blessed; for everyone else I suggest following the recipe.
Related topics with useful information to solve I2C problems:
viewtopic.php?f=8&t=872
viewtopic.php?f=8&t=884
viewtopic.php?f=6&t=935
viewtopic.php?f=8&t=932
viewtopic.php?f=8&t=887
viewtopic.php?f=8&t=942
http://dsscircuits.com/articles/effects ... stors.html
By the way:
Yes, I have seen the "ram shortage" discussion. If I have not convinced you that your I2C problem is due to your hardware, then please feel free to run the avr-size tool and confirm you have sufficient ram. Honestly, if you have a ram shortage your problems will be much more devious than some I2C errors in debug2.
For those that have "tried everything" and still see I2C errors in Debug2, then don't despair. Post an *accurate* schematic of your MWC and perhaps we can offer some advice. Please understand that Without a clear and concise schematic (of YOUR model) useful help will likely be impossible.
- Thomas
Wait a minute, you say. You've used several different versions and everything has been fine; Your MWC flew perfectly, until V1.9! So you are confident that there's no way your model has a hardware problem. You may be surprised to hear that the solution involves fixing your I2C related hardware. That's right, some of us have discovered that the existing I2C wiring techniques are part of the problem.
A little background information:
In the earlier MWC software releases the I2C communication was not 100% reliable in every installation. But the I2C problems weren't noticed because software masked them. On installations with these hidden problems the model's stability and drift performance can be affected (sometimes not even noticed by the pilot). No doubt this has caused Alex some headaches; Throughout the various releases, users have been reporting random performance problems that kept him busy providing support and experimental software fixes. But, the software can only do so much when the hardware has problems.
A bit of MWC history:
During early TriCopter development Alex had problems with some variants of the WMP board. Sometimes the WMP would occasionally fail to respond to I2C R/W access. Part of this may have been caused by how the start/stop bits were compacted together during I2C communication. Regardless of why it occurred, some I2C software tricks were incorporated to mask the issue. For example, the Arduino's D12 output pin was used to Hard Power-Reset the WMP board. It's also the reason for the neutralizeTime task, which temporarily tosses out sensor data after each I2C reset event. Although the resets can occur often, you cannot detect them. But they can cause reduced stability and/or more drift.
In addition, to simplify our installations there were some tricks to reduce the project's parts count. For example, some installations used the high value (20K-50K ohm) internal/CPU I2C pullups instead of carefully chosen external resistors. Also, some installations had 3.3V sensors that eliminated the LLC board (5V->3.3V Logic Level Converter) by using external pullups that were sourced by 3.3V. Hardware tricks like these are gloriously simple; Unfortunately they are tempermental and not reliable for everyone.
Long story short, the older software was allowing models to fly that actually had problems. Some may think that this is acceptable but I doubt everyone wants to continue flying models that are suffering from reduced performance.
Then came V1.9:
Things changed for the better when Alex released V1.9. Unfortunately, there are three different V1.9 releases that have identical version names. These three V1.9's are as follows:
Release 1: Introductory release, Version 1.9 {Nov-06}. It had a bug in the I2C code that caused a fatal CPU lockup in some installations. It occurred ONLY if the I2C buss had problems. Some models would not boot and others would run for awhile, then lockup. Like some of you, I experienced the boot lockup, which after some troubleshooting resulted in the introduction of the I2C error counter in the GUI: viewtopic.php?f=8&t=884
Release 2: Emergency patch, Version 1.9 {Nov-11} was released which fixed the I2C lockup problem. But more importantly, Alex reworked the I2C code and he was able to solve the old WMP problems. The need for the D12 Hard Reset was finally eliminated. But this version is missing the important I2C Error count value in the GUI.
Release 3: Emergency patch, Version 1.9 {Nov-12} was released which added the I2C error counter in the GUI (debug2 location). The value will increment with each I2C error. At about 32K errors it will become negative and count backwards.
Which brings us to the purpose of this i2c discussion:
With the third V1.9 release the I2C related software was vastly better than before. But that did not mean everything was solved because some installations have I2C wiring that needs improvement. Fortunately, with the I2C error counter in the GUI, the hardware problems can be detected and fixed. The Debug2 value is all you need to troubleshoot V1.9, fix the I2C, and improve your model's performance. So please use the Debug2 feature!
I2C Hardware Fix List / RC-CAM's i2c Recipe (subject to revision)
------------------------------------------------------------------------------
Here are some of my suggested changes to solve I2C problems. These have helped many installations. Don't cheat, do them all!
- Use the V1.9 release that is dated NOV-12. Not sure which one you have? Then download it again from the web and re-install it.
- The recommended Arduino CPU's are 5V devices. So for reliable operation your I2C pullups on the CPU must be sourced from 5V, not 3.3V. Installations with 3.3V sensors will need an LLC (to prevent damaging the sensor).
- Disable internal pullups and use external 2.2K ohm pullups. Higher pullup values may not work for you so I recommend 2.2K ohm. Please note that some MWC shields have pre-installed pullups sourced by 3.3V. These must be disabled before you can use the 2.2K pullups sourced by 5V.
- 5V compatible sensors should be powered directly from 5V. Do not use the D12 signal and remove any "voltage drop" diodes.
- Observe the GUI's debug2 window. Verify that this value remains zero during a long (>10 minutes) bench test. Repeat for at least one minute with the motors running. If the value is not zero at the end of the tests then you have I2C wiring or sensor related problems.
Keep in mind that the I2C error counter is a primitive debugging tool and will not report *every* I2C problem that may occur. However, if you follow the I2C Hardware Fix list you should be able to eliminate most of the potential problems. Of course some of you are going to report that you ignored the list and all is well. Consider yourself blessed; for everyone else I suggest following the recipe.
Related topics with useful information to solve I2C problems:
viewtopic.php?f=8&t=872
viewtopic.php?f=8&t=884
viewtopic.php?f=6&t=935
viewtopic.php?f=8&t=932
viewtopic.php?f=8&t=887
viewtopic.php?f=8&t=942
http://dsscircuits.com/articles/effects ... stors.html
By the way:
Yes, I have seen the "ram shortage" discussion. If I have not convinced you that your I2C problem is due to your hardware, then please feel free to run the avr-size tool and confirm you have sufficient ram. Honestly, if you have a ram shortage your problems will be much more devious than some I2C errors in debug2.
For those that have "tried everything" and still see I2C errors in Debug2, then don't despair. Post an *accurate* schematic of your MWC and perhaps we can offer some advice. Please understand that Without a clear and concise schematic (of YOUR model) useful help will likely be impossible.
- Thomas