SD Card Logger

This forum is dedicated to software development related to MultiWii.
It is not the right place to submit a setup problem.
Software download
renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

SD Card Logger

Post by renes »

Hello everyone,

I had some issues with my quad and I needed some in flight data logging. I did not want to change the MW code - I wanted an non intrusive approach.

So I created a data logger using the serial port of my AIO board and the multiwii serial protocol.
An arduino pro mini + a SDCard reader did the job quite well. So all you need is connect serial port of the flight controller to the arduino and connect the arduino to your card reader (SPI interface) and its done.

The logging is done to csv files and records the IMU data, angles and stick inputs so far.

If anyone needs something like that I put it on github:

https://github.com/renes/multiwiiDataLogger

Still room for improvements, its just the first version - but its working. I already did some recording of data with my own quad. Example files are also on github.

Feedback is welcome of course. If anyone needs some more features just let me know.

best regards,
rene
Last edited by renes on Mon Jun 23, 2014 7:46 pm, edited 1 time in total.

copterrichie
Posts: 2261
Joined: Sat Feb 19, 2011 8:30 pm

Re: SD Card Logger

Post by copterrichie »

Nice Job.

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

Looking nice.

Will it work on OpenLog HW ?

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

I dont have an open log HW but what I figured out is that it is just a logger that reads serial input and writes it to a SD card.
this wont work so far, because to get the data from MW you have to send a command to the MW and then you will get back the data to log.
so the openlog hw wont get any data to log, because it does not send any commands to the MW board.

an other problem is, that you need some processing of the data. otherwise you get the binary protocol of the MW logged - no csv files.

one reason why I log to csv files (I know its not that efficient ) is that everyone can use them very easily in various programs like excel to create charts do some analysis or for any other custom application

im personally working on an application which replays the captured data by the logger and visualizes the quad on my pc (some little 3d animation of the quad), besides the RC inputs, etc.

so I can go through the flight step by step and see what I did on my rc and what was the reaction of my quad.

by the way a pro mini + sd card logger is cheaper than the openlog hw as far as I could see on my little research.

best regards
rene

copterrichie
Posts: 2261
Joined: Sat Feb 19, 2011 8:30 pm

Re: SD Card Logger

Post by copterrichie »

There is the option of softserial and turn down the baud rate on the Openlog to 9600 but I agree, for this application, it is not a very good. It would be better to create a sub-routine in the Wii Protocol to write directly to the Openlog with the desired variables.

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

yes that was one way I was thinking of, but i did not want to modify the MW code.
wanted it to be more "plug & play".

best regards
rene

copterrichie
Posts: 2261
Joined: Sat Feb 19, 2011 8:30 pm

Re: SD Card Logger

Post by copterrichie »

Just a FYI, I like the way you structured your send MSP code classes. That should make other applications very easy to implement.

Thanks again and a great job..

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:I dont have an open log HW but what I figured out is that it is just a logger that reads serial input and writes it to a SD card.
this wont work so far, because to get the data from MW you have to send a command to the MW and then you will get back the data to log.
so the openlog hw wont get any data to log, because it does not send any commands to the MW board.

an other problem is, that you need some processing of the data. otherwise you get the binary protocol of the MW logged - no csv files.

one reason why I log to csv files (I know its not that efficient ) is that everyone can use them very easily in various programs like excel to create charts do some analysis or for any other custom application

im personally working on an application which replays the captured data by the logger and visualizes the quad on my pc (some little 3d animation of the quad), besides the RC inputs, etc.

so I can go through the flight step by step and see what I did on my rc and what was the reaction of my quad.

by the way a pro mini + sd card logger is cheaper than the openlog hw as far as I could see on my little research.

best regards
rene


I know open log, and its limits.

The question was if openLog HW was supported by your code, since it is looking it is at first look.

The advantage of openLog HW is to have a ultra compact module, almost same size of a microSD ... that just need 4 wires: 2 power + 2 serial ... and can take log data automatically ... no need to hack anything in the MultiWii code.
So, if your firmware run into OpenLog, as I do suspect, then it is an excellent alternative to "stock" firmware of OpenLog in the MultiWii log job.

PS: I guess you did took somewhere the SD-FAT16/32 library , can you tell which one you pick up ?

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

ok I got it wrong what you wanted to do.

this way - it maybe works. I will order one and give it a try. I will post the results and push the modifications (if needed to the repo) as soon as I got the logger.

yes the fat lib is used but not directly.

I use the standard arudino SD lib (http://arduino.cc/de/Reference/SD) which includes the fat lib. You can look in ur arudino libs folder which fat lib is actually used on your system

best regards
rene

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:ok I got it wrong what you wanted to do.

this way - it maybe works. I will order one and give it a try. I will post the results and push the modifications (if needed to the repo) as soon as I got the logger.

yes the fat lib is used but not directly.

I use the standard arudino SD lib (http://arduino.cc/de/Reference/SD) which includes the fat lib. You can look in ur arudino libs folder which fat lib is actually used on your system

best regards
rene


I have a OpenLog HW since quite a while ... it did cost me a bit, including the shipping from US to EU, but ... it is a nice little hardware ... overprice for what it is, but, there is no alternative, unless a bulky solution sticking a pro-mini to a microSD module.

Which Arduino do you use ? 1.0.5 ?

I can give it a go

I did not know arduino had SD library on it, open log use quite some messed up SD+FAT libraries ....

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

I just ordered one 30€ incl shipping , little bit expensive but yes its small and the solution with the pro mini and the sd card works but an all in one board maybe the better solution

ok great - yes I use 1.0.5 (dont try the 1.5.6, it wont work)

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:I just ordered one 30€ incl shipping , little bit expensive but yes its small and the solution with the pro mini and the sd card works but an all in one board maybe the better solution

ok great - yes I use 1.0.5 (dont try the 1.5.6, it wont work)


Thanks

Going to give it a try soon ;-)

Sorry for have been costing you 30 Euros :roll:

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

dont worry - I think its a great idea to use the open log with a custom firmware - so its something worth trying out

argotera
Posts: 17
Joined: Sun Oct 20, 2013 1:40 pm
Location: Athens, Greece

Re: SD Card Logger

Post by argotera »

Hi Renes, thanks a lot for your contribution. I ll try to implement it to a small quad I have.

Is it possible to record gps data too?



Edit: I am trying to add these I found in the logger sketch.
MSP_RAW_GPS
MSP_COMP_GPS




Vassilis

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

yes it is possible.

but not out of the box, you have to make some minor modifications to the code. I hope i will find some time and push them to the git repo this week end

I also go my openlog hardware today so as far as i find a micro SD card in my stuff I will see if the code will work on the open data logger.

best regards rene

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:yes it is possible.

but not out of the box, you have to make some minor modifications to the code. I hope i will find some time and push them to the git repo this week end

I also go my openlog hardware today so as far as i find a micro SD card in my stuff I will see if the code will work on the open data logger.

best regards rene


Very interested on your results.

I personally like to be able to keep, also, the OpenLog original firmware ... so it is good to be sure it is easy & possible to flash back and force the two different firmware (yours and the original one)

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

it is no problem to go back

open log firmware is open source

https://github.com/sparkfun/OpenLog

so you can always flash the firmware back or update to the newst version

best regards
rene

argotera
Posts: 17
Joined: Sun Oct 20, 2013 1:40 pm
Location: Athens, Greece

Post by argotera »

Thanks Rene, I did try to change the code but I still do not know programming. I ll try a few things again just to experiment but realistically I ll have to wait for your sketch. Thanks again.


P.S. Hello e_lm_70. I hope you are fine.

copterrichie
Posts: 2261
Joined: Sat Feb 19, 2011 8:30 pm

Re: SD Card Logger

Post by copterrichie »

renes, is all that is required to use your code is a SD Card Modular and any Mega328?

Thank you.

SDCardModule.JPG

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

All you need is either a openLog module or, a atmega328+sd adapter ... the pin to be connected in the second case are shown on the gitHub ... mainly since it is a spi interface you need pin 11, 12 and 13 ... plus some more

Using openLog HW you can have a solid and ultrlight/ultra small solution.

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

yes its all you need

pins used depend on the board you use:

e.g. pro mini

** MOSI - pin 11
** MISO - pin 12
** CLK - pin 13
** CS - pin 4

on other boards you may find this site usefull

http://arduino.cc/de/Reference/SPI

openlog is a quite good alternative, but i did not test it until now - does anyone give it a try? is it working? I dont know if I have time this we to give it a try

------------------------------------------

I also added the requested GPS log support - just the GPS raw data so far so no MSP_COMP_GPS for this round.

adding the MSP_COMP_GPS will be easy - all methods to e.g. decode unit16 are there - so if anyone wants to add this just send me a pull request - otherwise you have to wait for me to implement it.

the code is on the github repo and ready for use.

best regards
rene

argotera
Posts: 17
Joined: Sun Oct 20, 2013 1:40 pm
Location: Athens, Greece

Post by argotera »

Thanks again Rene for your time and effort.

Vassilis

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:yes its all you need

pins used depend on the board you use:

e.g. pro mini

** MOSI - pin 11
** MISO - pin 12
** CLK - pin 13
** CS - pin 4

on other boards you may find this site usefull

http://arduino.cc/de/Reference/SPI

openlog is a quite good alternative, but i did not test it until now - does anyone give it a try? is it working? I dont know if I have time this we to give it a try

------------------------------------------

I also added the requested GPS log support - just the GPS raw data so far so no MSP_COMP_GPS for this round.

adding the MSP_COMP_GPS will be easy - all ethods to e.g. decode unit16 are there - so if anyone wants to add this just send me a pull request - otherwise you have to wait for me to implement it.

the code is on the github repo and ready for use.

best regards
rene


cs on pin 4 is possibly the only change to be made for make it run on openLog

I was hoping on you for try this out , but .. maybe I will try on mine in the next days and I will report

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

ok I tested it with my open logger.

You just have to change the chipselect to 10 as it was said earlier.

file cardlogger.cpp
const int chipSelect = 10;

nothing else to change.

works perfect

I also adapted the readme on github to provide information about howto flash it etc.

best regards
rene

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

Excellent

Using OpenLog as HW, now I have no excused to try it out, and use as fly logger.

Hypermobile
Posts: 94
Joined: Mon Jan 13, 2014 8:53 pm

Re: SD Card Logger

Post by Hypermobile »

Anyone a Clue which ConverterProgram should be best to generate a GPX File, (or other format).

GPS Babel perhaps?
Can anyone provide me some CSV files made by the SD Logger??

I really like the Idea of Onboard Logging.
I did some Logging with EZGui from Ezio....but it looks like my modified Bluetooth (70mtr range) doesn't provide me enough data. But does provide me GPX Files.

Sample of posibilities: http://youtu.be/95YqNlYtouw
A combination of Speed, GPS, Heading would be very. to have in a GPX File

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

there are example files on the git repo of the logger

converting it to GPX should not be a problem - maybe someone can write a little tool for it

best regards rene

Hypermobile
Posts: 94
Joined: Mon Jan 13, 2014 8:53 pm

Re: SD Card Logger

Post by Hypermobile »

Was looking for it in the Log_1 ...

But the GPS.TXT CSV, had only got 2 lines of info.

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

yes but actually thats the format of the file
if you want a longer example file maybe someone can share his actual flight data

brewski
Posts: 483
Joined: Tue Apr 29, 2014 12:04 am
Location: Cleveland Qld Australia

Re: SD Card Logger

Post by brewski »

Hi Renes,

My Crius AIOP V2 has onboard 16Mb Dataflash chip. Apparently this provides auto datalogging with MegaPirates.
Do you know how I can utilise this 16Mb for datalogging with MW 2.3 Navi b7? It would be cool to be able to view a mission & also do analysis when something goes wrong.

Hypermobile
Posts: 94
Joined: Mon Jan 13, 2014 8:53 pm

Re: SD Card Logger

Post by Hypermobile »


renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

Only if he installs the firmware above on it

Ad aiop: I 've one myself - there is no way so far to use the internal memory as far as you make changes to the multi Wii software

brewski
Posts: 483
Joined: Tue Apr 29, 2014 12:04 am
Location: Cleveland Qld Australia

Re: SD Card Logger

Post by brewski »

Ok I'll forget the onboard datalogging chip (maybe can use with MW2.4?). Surely if MP can write to it we can just add the section of code from MP?
I have a micro SD logger from my Arduino projects that uses same interface I.e. Mosi, Miso, Clk, CS. Do I just connect to the SPI pads on my Crius AIOP V2 & connect 5V?

Edit. no that would,t work as SPI is programming only port on these boards?

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

u cant just connect a sd logger because the loggers firmware needs to send some commands to the multiwii to send data to the logger
if u just connect e.g. a openlog ur board will send nothing that it can log

so u have to flash it with a custom firmware e.g. https://github.com/renes/multiwiiDataLogger which sends the commends to the multiwii board to trigger that some data is send back to the logger

how to connect this sd logger see on the github read me page

best regards rene

brewski
Posts: 483
Joined: Tue Apr 29, 2014 12:04 am
Location: Cleveland Qld Australia

Re: SD Card Logger

Post by brewski »

Hi Renes,

I looked at Read Me on github page & couldn't see where it shows how to connect. From what I have read above the datalogger uses serial interface to connect to FC.
On my AIOP V2 I have Bluetooth connected to FTDI/Serial0 & GPS connected to Serial port2. Which serial port on FC is your program expecting to see the datalogger?

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

I disconnected the bluetooth because its not needed in flight anyway

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

an other options is

http://www.hobbyking.com/hobbyking/stor ... 863X42.pdf

see page 2

you have a couple of serial interfaces on your AIOP board so u just need one for a logger
then u can use bluetooth and a logger

e_lm_70
Posts: 297
Joined: Fri Aug 09, 2013 8:35 pm

Re: SD Card Logger

Post by e_lm_70 »

renes wrote:an other options is

http://www.hobbyking.com/hobbyking/stor ... 863X42.pdf

see page 2

you have a couple of serial interfaces on your AIOP board so u just need one for a logger
then u can use bluetooth and a logger


AIOPv2 has on board 16Mbit memory ... so ... in theory the SDlogger code could be implemented inside multiwii .. without the need to use an external SDlogger ... even if ... using OpenLog HW it is extremely small , and it has the advantage to use less CPU power from the control board in charge to stabilize and guide the copter ...

brewski
Posts: 483
Joined: Tue Apr 29, 2014 12:04 am
Location: Cleveland Qld Australia

Re: SD Card Logger

Post by brewski »

That's what I have been suggesting. If APM 2.6 can log to similar flash memory chip (only 4Mb on APM) using same AtMega 2560 MC Multiwii should be able to do this also.
From research I have done the only difference between APM 2.6 & Crius AIOP V2 (besides Crius having 16Mb datalogging dataflash chip) is the MPU6000 which has SPI bus @ up to 20MHz against MPU6050 with I2C @ 400Hz Max. In real world flight reaction times the difference between the two bus types will be negligible, but will definitely prevent running APM firmware on the Crius AIOP FC. If anyone is interested here is an article describing the differences/pros & cons of 1C2 & SPI
http://components.about.com/od/Theory/a ... nd-Spi.htm

User avatar
alduxvm
Posts: 40
Joined: Thu Apr 25, 2013 2:25 pm
Contact:

Re: SD Card Logger

Post by alduxvm »

I'm trying to upload this on my openlog, but so far, it has not being possible...

So, what I did is this:

- I have a FTDI cable, connect the board to it, and crossover the tx/rx
- Download the firmware, and compile/upload it

and so far, I always got this error:

avrdude: ser_recv(): programmer is not responding
avrdude: stk500_recv(): programmer is not responding

I tried selecting “Arduino Uno”, “Arduino Pro or Pro Mini 5V/16MHz w/ ATmega328”, and also reverse the crossover and again nothing... has anyone got into this problem???

User avatar
alduxvm
Posts: 40
Joined: Thu Apr 25, 2013 2:25 pm
Contact:

Re: SD Card Logger

Post by alduxvm »

My mistake!!!

I was just plugging in 4 cables... vcc, gnd, tx and rx... but we need to plug all of them, even the DTR and CTS...

https://github.com/sparkfun/OpenLog/blo ... ctions.jpg

Liriel
Posts: 10
Joined: Fri Dec 26, 2014 8:46 pm

Re: SD Card Logger

Post by Liriel »

i bring it back to the top.

has someone convert the gps log from renes datalogger to google earth or other gps map programms?
but how?

renes
Posts: 26
Joined: Sat Mar 08, 2014 10:50 am

Re: SD Card Logger

Post by renes »

the logger output is just a csv so you can use any tool to convert the csv to e.g. KML which can be imported in e.g. google earth

use a tool like

http://www.mapsdata.co.uk/online-file-converter/

to do that

best regards
rene

argotera
Posts: 17
Joined: Sun Oct 20, 2013 1:40 pm
Location: Athens, Greece

Re: SD Card Logger

Post by argotera »

I use this converter, lots of graphical ways to output your data

http://www.gpsvisualizer.com/

szakacs
Posts: 18
Joined: Wed Dec 11, 2013 11:53 pm
Location: Sydney, NSW Australia

Re: SD Card Logger

Post by szakacs »

Newbee help please ...
Could somebody please confirm that the connections from the flight controller to the pro mini would be a crossed over tx - rx serial connection.
i.e. HK multiwii pro serial 1 tx/rx to pro mini rx/tx.
I have managed to complie and load the sketch, but when I go to read the card I see a Log directory with nothing in it.

Liriel
Posts: 10
Joined: Fri Dec 26, 2014 8:46 pm

Post by Liriel »

Yes it must rx to TX and TX to TX.
Are the led in the pro mini blinking?
I usw a arduino nano to protocoll

szakacs
Posts: 18
Joined: Wed Dec 11, 2013 11:53 pm
Location: Sydney, NSW Australia

Re: SD Card Logger

Post by szakacs »

Thanks Liriel ..
I must have an issue as the only serial speed that works for me is 9600.
If I try and speed it up to 57600 or 115200 by adjusting the serial speeds and recompiling, I get nothing but garbage on the card and have to reformat in a pc.
Sometimes I get a log directory with nothing in it.
If I use the SD examples like reading, writing and cardinfo I can run at any speed no probs. I use Arduino 1.0.5 as 1.5.4 won't compile.
Any suggestions ?

szakacs
Posts: 18
Joined: Wed Dec 11, 2013 11:53 pm
Location: Sydney, NSW Australia

Re: SD Card Logger

Post by szakacs »

OK ..Found one of the issues.. I am using a LC Studio card reader and they seem to be very touchy with input voltage. If a feed it 5 volts things are OK, but a 6 volts it is trouble.
It looks like that the onboard 3.3 regulator is not very tolerant.
Still can't go any faster than 9600, and from what I am reading that could be due to poor level shifting of the 3.3v signals.

szakacs
Posts: 18
Joined: Wed Dec 11, 2013 11:53 pm
Location: Sydney, NSW Australia

Re: SD Card Logger

Post by szakacs »

That was it ... supply the board with a 3.3 volt source and all is now working. No more corruption and I have been able to speed the serial up to 57600 with no issues..
Now off to break something else ..

helikalle
Posts: 4
Joined: Wed Jan 21, 2015 3:51 pm
Location: Germany

Re: SD Card Logger

Post by helikalle »

Hello, i have succesfully installed the openlog Hardware. Because the logger prevents the flight controller to initialize correctly i have inserted a 3 second delay in the in the "void Setup ()" section of the code. Now all works perfectly.
What i am missing is a time/date Information somewhere in the logs to identify my flights. How could this be realized? Is it possible to read the date/time-stamp from gps and add it to the log or the log-Name?

Second item: How can i log the baro data from the flightcontroller?

Some suggestions?

Greetings

Post Reply