Senior Design Part 7: Compressing data with Magic? (It's just math!)

2023-06-02 | By Will Siffer

License: Attribution Non-commercial

What's the deal with data?

When we send data, at the end of the day we send a long series of 1's and zeros that are interpreted in various ways. If we want to send more data, we must send a longer transmission. Usually, that's not a huge deal since we use high-bandwidth wireless protocols like Wi-Fi and Bluetooth that can handle larger data transmissions, but LoRa is a low-bandwidth transmission system that takes more and more time to send data as the transmission gets longer.

For our project, we have a fair amount of data to send:

RFID Tag ID (20 Bytes)
Time the RFID tag was scanned (4 Bytes)
Battery Voltage (4 Bytes)
Battery Current (4 Bytes)
Solar Panel Voltage (4 Bytes)
Solar Panel Current (4 Bytes)
Humidity (4 Bytes)
Temperature (4 Bytes)

That's 48 bytes of data! The most we can send reliably is 50 Bytes, so that's far too close to max to be reliable so we must figure out how to make the transmission shorter.

The CRC Checksum Hash

We decided to use a CRC16 checksum to shorten a 20-byte value into a 2-byte value. The way a CRC hash works is it uses a specific algorithm to allow computers to compare two values, while this technique does create the chance that we have 2 tags that create the same checksum, in practice that is not something that happens and even if it did we could abandon that tag before it was registered to a user.

Great! That drops the total bytes down from 48 to 30. Let's see what else we can do to make our data even smaller!

Here's the solution that we came up with:

Transmission

Transmission Breakdown

As you can see, we got our transmission down to only 17 bytes. I was able to figure out how to convert all the 4-byte signals down to 2 bytes! I was even able to remove a byte of data from the time! Here's how we were able to do it!

It's about Time!

When it comes to the time variable, we decided that we didn't need to send the full Unix time, when we really only care about the time of day each day. So, we decided to use math to only send the number of seconds since midnight UTC, effectively removing all the time that has happened before the day, which we don't have any reason to send.

High Voltage Math!

The voltage signal is measured by the INA3221 with three decimal places of precision. Since we know that the voltage measurement will never exceed 5 volts, we decided to multiply the value by 1000 to get rid of the decimal places. Since that means the highest value will be 5000, that's 0x1388 in hexidecimal and thus well within 2 bytes. In fact, 2 bytes can support up to 65 volts! We used this method on both the battery voltage and the solar panel voltage since they both are in the same range.

AMP-ed up about Data! (Are 'current'-ly tired of these EE puns?)

The current measurement is similar to the voltage measurement, but with a crucial difference. Since the battery can be charged by the solar panel, the current is at times negative. In case you are unfamiliar with how negative numbers work with hexadecimal, they take the place of the largest number in hex. This is called "signed 2's compliment." I won't get too technical here with it, but since a 4-byte -1 value would be FFFFFFFF, that is not possible to simply shorten so we need a way to eliminate negative values.

The way we came up with this was first by realizing that we would not ever be charging the battery anywhere close to an entire amp. So the smallest negative value our chips will ever see is -1 at the worst case but the charging chip is limited to 500mA so during common operation we would only really see up to -0.5. By adding 1 to the value, we eliminate all the negative numbers, but we still have decimals. From here, we just handle current the same way as voltage and multiply by 1000. Now we have shortened our data from 30 to 21 bytes!

The last two variables... (I wasn't sure 'weather' or not you would appreciate another data pun)

The nice thing about the humidity variable is that there is no offset, and the range of data is only from 0.00 - 100%. So in order to remove the decimals all we had to do is multiply by 100 and then we had a value that could fit inside of 2 bytes.

Temperature is another story. Our board is rated to be able to handle -40 degrees Celsius at a maximum, so we need to handle as low as -40.00 for the temperature variable. Similar to the current measurement variable, we added 50 to each measurement to eliminate the negative and then multiplied by 100 to eliminate the decimals.

With all this magical math, we were able to successfully shorten a 48-byte data stream to 17 bytes. That's almost a 65% reduction in data length!

Of course, all these transformations come at a cost. I can't go without saying that we aren't actually sending the proper data in the transmissions, and we need to transform the data back before it can be interpreted meaningfully. Fortunately for us, the same team is processing the data on the backend as the team that is working with us on the network side, so we are in communication with them, and they are aware of the transformations and have already written scripts to decode our variables. To decode, all they need to do is the same math in reverse using the key at the beginning of this blog post.

This has been a great exercise for me to practice data manipulation since in today's world data drives almost all decisions. By working so hard at the beginning of the project to make the data as small as possible, it will allow us more flexibility in the future to send even more data and even more information to the base station.

That's all I have this week for this project. I have really enjoyed all the new techniques that I've learned throughout the last 6 weeks. If you learned something today, be sure to build on it and practice it in your own project! Until next time, my name is Will, and I am the Digi-Key student ambassador at Purdue University, and I hope you stay curious.

Recommended Reading

"This week was all about data, and how we need to be mindful of how we manage and interpret it efficiently. Since we manage user data, we need to make sure that we make the correct decision for the user as well as the project. This makes me think about the ethical responsibilities of data and what I expect of other projects and companies that manage my personal data. Since I wouldn’t want my data mismanaged, I expect any team I work on to manage the data we have safely. As we pull this out into a more global view, this project is in the works of expanding to be an actual product, so the data processing needs to be safe and effective. Moving forward we are talking more and more about data management so we can continue being user-focused as we move on through the end of the semester. After I graduate and I move into my professional career, I am continuing to be mindful of how much responsibility I may have to keep customer information safe, whether it be the locations of customers or customer information in general. I have to keep these same values in mind, so I never put myself or the company I work for in a bad situation."

Add all DigiKey Parts to Cart

Have questions or comments? Continue the conversation on TechForum, DigiKey's online community and technical resource.

Senior Design Part 7: Compressing data with Magic? (It's just math!)

2023-06-02 | By Will Siffer

信息

帮助

联系我们

关注我们