Debug Session

I was back at the manufacturing plant to test the new software on the robot. The testing uncovered a serious problem which caused almost all measurements on the gauge to fail, even ones which were not changed in this project.

The original design had a single A/D board attached to eight sensors. The driver for the A/D board stores the samples for all the channels in a single circular buffer. So with 8 channels enabled, the driver adds 8 samples in each cycle. A user space program reads the buffer and distributes the samples to a separate program for each channel.

The new system replaces one sensor with a supposedly "better" one but also has a requirement that it have a higher sampling rate than the other sensors. This particular A/D board has a limitation that all the channels must have the same sampling rate. The only solution is to add a second A/D board and, for simplicity, an identical board was used. I hacked up the existing analog input program to take arguments for base address, interrupt, sampling rate, etc., so that it could handle multiple boards. I also disabled channels that are not in use to reduce load on the system, and this is where I shot myself in the foot. The program still assumed all eight channels were in use when it distributed the samples, so every measurement would fail because it used data from the wrong sensor.

While debugging this problem I also discovered the system is now very close to the limit. When I dumped out the sample data from one measurement process, the analog input process lost sample data in the circular buffer. Before, I could dump the sample data from three processes safely.

There is still work to be done on the measurement process for the new sensor. Since it is using the original sampling rate, in theory the algorithms should just work, but it is never that easy.