Weather Station Archives - Matthew Gove Blog https://blog.matthewgove.com/tag/weather-station/ Travel the World through Maps, Data, and Photography Tue, 03 Aug 2021 18:21:12 +0000 en-US hourly 1 https://wordpress.org/?v=6.1.5 https://blog.matthewgove.com/wp-content/uploads/2021/03/cropped-android-chrome-512x512-1-32x32.png Weather Station Archives - Matthew Gove Blog https://blog.matthewgove.com/tag/weather-station/ 32 32 Digging Deeper: Diagnosing My Raspberry Pi Temperature Sensor’s “Hot Flashes” https://blog.matthewgove.com/2020/02/05/digging-deeper-what-is-causing-my-temperature-sensors-hot-flashes/ Wed, 05 Feb 2020 22:39:29 +0000 https://blog.matthewgove.com/?p=917 I recently identified a rather unfortunate bug that left part of my weather station’s QA/QC algorithm redundant. Fixing it treated the symptom, but not the cause of the problem. Today, we’re going to look at what is causing the temperature sensor on one of my Raspberry Pi’s to seemingly randomly […]

The post Digging Deeper: Diagnosing My Raspberry Pi Temperature Sensor’s “Hot Flashes” appeared first on Matthew Gove Blog.

]]>
I recently identified a rather unfortunate bug that left part of my weather station’s QA/QC algorithm redundant. Fixing it treated the symptom, but not the cause of the problem. Today, we’re going to look at what is causing the temperature sensor on one of my Raspberry Pi’s to seemingly randomly show air temperatures close to 500°F. Can we nip the problem in the bud before it even gets to my QA/QC algorithm?

Before I begin troubleshooting an issue, I always brainstorm a list of at least 2-3 things that I think may be causing the problem. In no particular order, some of the potential causes I came up with were:

  • An electrical problem that is causing voltages to go haywire
  • A malfunctioning or defective sensor
  • A bug in the Python code on the Raspberry Pi that reads the raw data from the sensor and converts it into Celsius

Start with the Easiest Issue to Troubleshoot

Before diving into troubleshooting, I always see if there are any causes I can reasonably eliminate. In this case, a malfunctioning or defective sensor is highly unlikely to be causing the problem. The sensor is an MPL3115A2 I2C digital barometric, temperature, and altitude sensor. It has been in service on the Raspberry Pi since last May and has taken nearly 1.2 million measurements. Of those 1.2 million measurements, the QA/QC algorithm has flagged 590 of them. That works out to 0.04% of measurements being bad, or a 99.96% measurement accuracy rate.

Identify Possible Bugs where the Raspberry Pi Interprets the Sensor Readings

Of the remaining two possibilities, I always start with whatever is easiest to troubleshoot. In this case, it would be a possible bug in the Python code. On the Raspberry Pi, Python interprets sensor readings and logs them in the database. Intermittent electrical problems are a nightmare to diagnose in the best of situations. The first thing to do is to look at the data and try to identify any patterns. If you’ve ever watched any detective shows on TV, you know what they always say: evidence doesn’t lie.

One pattern should jump out at you immediately. As air temperatures go to, and drop below freezing (0°C), the sensor readings on the Raspberry Pi go haywire. It is also important to note here that this does not rule out an electrical problem causing the problem. To strengthen our case, let’s query the sensor database to see if the sensor recorded any temperatures below freezing. We want to see no instances of the sensor recording temperatures below freezing.

No sensor readings below freezing in the Raspberry Pi's database
Results from querying sensor readings for temperatures less than 0°C

We can now say with 100% certainty that something happens when the temperature hits freezing. However, we still don’t know what. To fully rule out an electrical issue causing the problem, it’s time to get on Google and research how the sensor works.

The temperature sensor connected to the Raspberry Pi
The temperature sensor connected to the Raspberry Pi

Can We Fully Rule Out an Electrical Issue Between the Raspberry Pi and the Sensor?

According to the manufacturer, there are three pieces of information that will help us here.

  1. The sensor reads and transmits raw data in hexadecimal format.
    • In traditional decimals (such as 0.625), each element of the decimal can store values from 0 to 9. Those values are represented by the corresponding numerals.
    • In hexadecimal format, each element of the decimal can store values from 0 to 15. Those values are represented by the numbers 0 to 9 and the letters A to F.
    • Python uses standard mathematics to convert between hexadecimal and traditional decimal formats and vice versa.
  2. Because of the way hexadecimal format works, the sensors cannot handle signed (i.e. negative) values in their raw format.
  3. The sensors come with a software module that converts the raw hexadecimal data to a “human-readable” decimal temperature, in Celsius.

The Problem Lies with How Digital Sensors Handle Negative Values

To finally be able to rule out an electrical problem, let’s focus on item #2. The sensors not being able to handle signed (negative) values in their raw format. With any kind of digital sensor, if it gets a reading that’s off the scale, it loops around to the opposite end of the scale.

For example, if a sensor had a scale of 0 to 400 and got a reading of -1, it would output a value of 399. A reading of -5 would output 395. It works on the other end of the scale, too. A reading of 410 would output a value of 10. During my meteorology studies at the University of Oklahoma, we routinely observed this phenomenon while measuring wind speeds inside of tornadoes with doppler radar.

Item #2 is such a key piece of the puzzle because the problem occurs as soon as temperatures drop below 0°C. With this information, we can firmly establish the bottom end of the temperature sensor’s scale. Now all we had to do was figure out what the top of that scale was. For that, we have to look at the Python code, in which I found the following equation:

raw_temperature = ((raw_temperature_msb * 256) + (raw_temperature_lsb & 0xF0)) / 16

In this line of code, Python converts the raw temperature reading to standard decimal. Python reads in the sensor’s raw hexadecimal readings bit-by-bit before converting them to Celsius and sending the to the Raspberry Pi. Each reading from the sensor contains a maximum of 2 bits. For example, if this formula was used on a standard Base 10 number, such as 27, it would be read as:

temperature = (10 * 2) + (1 * 7) = 20 + 7 = 27

Calculating the Sensor’s Upper Bound

While that information may not seem significant on its own, it gives us enough information to calculate the upper end of the sensor’s temperature scale. Remember in hexadecimal, each bit can contain a value up to 16, so calculating the upper bound (in degrees Celsius) is simply:

Upper Bound = 16 * 16 = 256

Now that we have established that the sensor has a raw output range of 0 to 256°C, let’s look back at the query results. Keep in mind what I mentioned above about off-the-scale readings looping around to the other end of the scale.

Raw sensor data on the Raspberry Pi from 4 February, 2020

What is the Actual Temperature of the Elevated Sensor Readings

To calculate the actual temperature, simply subtract 256 from the sensor readings in the 250’s. For example, if you look at the last entry in the above table, the actual temperature would be:

Actual Temp = 254.625 – 256 = -1.375°C

Using the above query as an example, the correct actual temperatures using the loop-around equation would be:

Corrected temperature readings from the Rasoberry Pi database

Implementing the fix before the sensor readings even reaches the QA/QC algorithm on the Raspberry Pi is simple. We simply need to add an if statement to the Python code that converts the raw hexadecimal sensor readings to degrees Celsius. Here is the original code:

try:
    raw_temperature = ((raw_temperature_msb * 256) + (raw_temperature_lsb & 0xF0)) / 16
    unrounded_temperature = raw_temperature / 16
except:
    raw_temperature = None
    unrounded_temperature = None

With the if statement added to fix the problem, the above block of code simply becomes:

try:
    raw_temperature = ((raw_temperature_msb * 256) + (raw_temperature_lsb & 0xF0)) / 16
    unrounded_temperature = raw_temperature / 16
    if unrounded_temperature > 70:
        unrounded_temperature -= 256
except:
    raw_temperature = None
    unrounded_temperature = None

Maintain Data Integrity

The final piece of the puzzle is to update the bad data points in the database to ensure data integrity. Amazingly, we can do that with a simple UPDATE command so we don’t lose any data. Full disclosure, this is not the actual database structure in the weather station. It’s just worded that way to make it easier to understand.

UPDATE `sensor_measurement`
SET `measurement_value` = `measurement_value` - 256
WHERE `measurement_type` = 'temperature'
AND `measurement_value` >= 70;

Well, this has certainly been an interesting one. It’s always a relief to confirm that the sensor is functioning exactly as it should and the problem is nothing more than an easy-to-fix bug in the Python code on the Raspberry Pi. Until next time.

The post Digging Deeper: Diagnosing My Raspberry Pi Temperature Sensor’s “Hot Flashes” appeared first on Matthew Gove Blog.

]]>
Houston, I Think There’s a Bug in My Weather Station’s QA/QC Algorithm https://blog.matthewgove.com/2020/02/03/houston-i-think-theres-a-bug-in-my-weather-stations-qa-qc-algorithm/ Mon, 03 Feb 2020 22:59:35 +0000 https://blog.matthewgove.com/?p=902 Have you ever heard the expression “Measure Twice, Cut Once”? It’s commonly used in woodworking and carpentry. It’s a reminder to always double and triple-check your measurements when you cut a piece of wood. You’d be amazed at how many times you screw it up. Well, let me tell you […]

The post Houston, I Think There’s a Bug in My Weather Station’s QA/QC Algorithm appeared first on Matthew Gove Blog.

]]>
Have you ever heard the expression “Measure Twice, Cut Once”? It’s commonly used in woodworking and carpentry. It’s a reminder to always double and triple-check your measurements when you cut a piece of wood. You’d be amazed at how many times you screw it up. Well, let me tell you about how I completely and utterly failed to do something very similar. Stupidity struck when I programmed the algorithm that QA/QC’s the raw sensor data in my Raspberry Pi weather station. I’ll also explain how I fixed the problem.

I recently logged onto my weather station to check some high and low temperatures for the month of January. While I was casually scrolling through the data, I caught something out of the corner of my eye. When I looked a little closer, I had to do a double take.

What was even more impressive were the “heat indices”. They make summers on the Persian Gulf, where heat indices can reach ridiculous levels, look like absolute zero. For comparison, the temperature of the surface of the sun is 9,941°F.

Hunting Down Bad Data Points on the Raspberry Pi

Both my and several friends’ initial reaction was to cue the jokes: “Well, it does get hot in Arizona…”, but unfortunately on a logic and reasoning test, science will beat humor every single time. Time to figure out why this happened. First, we need to run a query for all raw temperature data my sensors recorded that were greater than 60°C (140°F). I chose that cutoff based on the hottest temperature ever recorded in Arizona. On June 29, 1994, Lake Havasu City topped out at a sizzling 53°C (128°F).

Amazingly, the query returned almost 300 hits. Here is a small sample of them.

Sensors getting screwy readings like this is part of the deal when you operate any kind of data logger like this. I am much more concerned that so many bad data points managed to slip through my QA/QC algorithm on the Raspberry Pi. I’ll admit the QA/QC algorithm was the very basic one I wrote to just “get things going”. It was long overdue for an upgrade, but still, it should have caught these.

Once I queried the dates where these bad data points occurred, the culprit was revealed.

Recently Replacing the Sensor Adds a New Wrinkle

You may recall that this past December, I had to replace the analog temperature and humidity sensor that broke. I formally decommissioned the broken sensor on December 14, 2019. Did you happen to notice the first date of bad data? That’s not a coincidence.

So what caused the QA/QC algorithm to nod off and miss these bad data points? The answer goes back to the broken analog sensor. The broken sensor measured both temperature and humidity. When the relative humidity reading hiccuped, often showing values greater than 3,000% when it did, the corresponding temperature reading would get thrown off by about 10-20°C.

The problem is that 99% of those bad temperature readings were between -5 and 15°C (23 to 59°F). During the winter months, we see actual temperatures inside that range every day here in the Phoenix area, so you can’t simply filter them out. I wrote the original QA/QC algorithm to flag relative humidity values that were greater than 100% or less then 0%. I would deal with the temperature parameter when I updated the algorithm.

The New Temperature Sensor

The new digital sensor I installed only measures altitude, barometric pressure, and temperature. As a result, the Raspberry Pi reverted to obtaining its humidity data from the National Weather Service. The NWS data is already QA/QC’d. Because my original QA/QC algorithm only flagged humidity and not temperature, it deemed every data point that passed through it “OK”, thus rendering the algorithm redundant.

To confirm that this is the issue, the database puts the data that the QA/QC algorithm flags into a separate table in the sensor database. I use that data for troubleshooting and making improvements to the algorithm. A simple query will reveal the dates of temperatures I have flagged. If swapping the sensors did in fact make the QA/QC algorithm redundant, the query will only return dates on or after the sensor replacement. I replaced the sensor on December 14, 2019.

Thankfully, fixing the issue requires nothing more than a few lines of code to add an if statement to the algorithm so it flags temperatures that are outside of an acceptable range of -20 to 60°C. (-4 to 140°F). I chose the upper limit based on the hottest temperature ever recorded in Arizona (53°C/128°F). At the other end of the spectrum, I based the lower bound off of the coldest temperature ever recorded in Phoenix (-9°C/16°F). I will tweak that range as needed.

Looking Ahead

My goal is to continually add small upgrades and fixes to the QA/QC algorithm over the next year. By the time I have the complete network of sensors up and running, it will be up to a level of complexity that is acceptable for a hobby weather station. At the same time, I want it to be held as close to professional standards as I can. Stay tuned for future posts where we will look closer at what happens in the data logger’s electrical system to cause such wacky temperature readings.

The post Houston, I Think There’s a Bug in My Weather Station’s QA/QC Algorithm appeared first on Matthew Gove Blog.

]]>
Troubleshooting a Raspberry Pi Sensor Gone Awry https://blog.matthewgove.com/2019/12/17/troubleshooting-a-weather-station-sensor-gone-awry/ Tue, 17 Dec 2019 23:55:00 +0000 https://blog.matthewgove.com/?p=877 A couple years ago, I built a weather station using a network of Raspberry Pi’s. It obtained data from Weather Underground and the National Weather Service. Last spring, I finally added temperature, humidity, and barometric pressure sensors to the weather station. I have plans to add rain and wind sensors […]

The post Troubleshooting a Raspberry Pi Sensor Gone Awry appeared first on Matthew Gove Blog.

]]>
A couple years ago, I built a weather station using a network of Raspberry Pi’s. It obtained data from Weather Underground and the National Weather Service. Last spring, I finally added temperature, humidity, and barometric pressure sensors to the weather station. I have plans to add rain and wind sensors to the Raspberry Pi in the future.

Housings for the sensor and the Raspberry Pi
Solar radiation shield housing the temperature and humidity sensor for my weather station – May, 2019

Over the summer, I noticed something odd after a monsoon thunderstorm had gone through. The outdoor relative humidity on my weather station read about 20% higher than the outdoor humidity on my thermostat. I wrote it off as an isolated incident related to the monsoon storm and didn’t think much of it.

As the summer progressed, I noticed the same thing would happen whenever a monsoon thunderstorm would pass over us. The weather station would report the relative humidity about 20% higher than it actually was. By the end of the summer, it was reading 20% high all the time, even in sunny weather.

A Possible Hack that Would End in Disaster

On several occasions, the idea popped into my head to just go into the weather station’s Python code and subtract 20% off of the relative humidity values the sensor was reading. Trust me, it was very tempting. I had to keep telling myself that doing so would open up a whole new set of problems.

What happens if the sensor starts working properly and gives a reading below 20%. I wrote the Python code. I know that negative relative humidity values will crash the system. When you calculate the dewpoint, you would have to take the natural log of a negative number, which is undefined.

It turned out that my instinct to not tinker with the source code was correct. Over the course of the fall, that 20% gap between the sensor’s readings and the actual humidity grew to 30%, then 40% and 50%. By early November, the weather station would be reporting 80-90% relative humidity when actual humidity values were in the 10-15% range. That would have made a big, ugly mess of the source code, and would not get me any closer to fixing the problem.

The Sensor Finally Kicks the Bucket

By the 1st of December, the sensor would only give humidity readings of 100%, regardless of what the actual humidity was. Pulling the raw sensor data from the main database on the Raspberry Pi confirmed this. While I pondered my next move, I changed the weather station’s Python code. It would now get relative humidity data from the National Weather Service instead of the sensor.

Raw data from the weather station's database
Raw data dump from the weather station’s primary database: November 15 – December 1, 2019. Notice all of the 100’s in the Relative Humidity columns?

A Deeper Analysis of the Raspberry Pi Humidity Sensor Data

Looking at the above table piqued my interest enough to dump all of the relative humidity data out of the database that contains all of the sensor data and take a look at it. The sensor went into service in early May and takes readings once per minute, so the query returned about 300,000 data points. After crashing Microsoft Excel’s plotting tool several times (I have an old, cranky PC), I adjusted the query to only return the relative humidity values at the top of every hour.

Raw data from the humidity sensor
Raw relative humidity data from the faulty sensor, through December 5, 2019

Now, before I show you the marked up plot explaining everything, consider the following. Think of how it can help show when the humidity sensor went off the rails.

  • The summer monsoon this year was almost non-existent, with only a small handful of storms the entire monsoon season
  • Relative humidities on non-rainy summer days generally range from a minimum of 10-20% to a maximum of 35-45%.
  • The fall was very dry as well, until a series of winter storms started impacting the desert southwest on Thanksgiving Day (November 28th).
  • I also queried the weather station’s main database for days where measurable precipitation (1 mm or more) fell. This is the result:
    Rain data from the weather station

When you put everything together, it should become clear where the sensor goes awry.

Plot of raw data from the Raspberry Pi humidity sensor

How Do We Fix the Sensor?

Unfortunately, the only remedy at this point is to replace the sensor, preferably one of much higher quality. The old sensor that broke was a cheap analog one I paid about $5 for. You get what you pay for, right? Thankfully, the pressure sensor, which is a digital I2C sensor, also measures temperature. With some minor rewiring, I can simply swap them out.

The new I2C temperature sensor connected to the Raspberry Pi
Replacement digital I2C pressure/temperature sensor being installed in the solar radiation shield

The nice thing about I2C sensors is that you can connect multiple sensors to the system using only one data cable, so adding additional sensors is easy at either end of the wire. I can add them either in the solar radiation shield or in the waterproof housing that houses the Raspberry Pi and the power supply. Additional sensors will get connected where the four gray wire nuts are in the above picture. I will definitely be adding an I2C humidity sensor to the Raspberry Pi and likely more as well. Stay tuned.

The post Troubleshooting a Raspberry Pi Sensor Gone Awry appeared first on Matthew Gove Blog.

]]>
DIY Solar Radiation Shield: Results of the Bowls vs Plates Experiment https://blog.matthewgove.com/2019/07/31/results-of-the-bowls-vs-plates-experiment/ Thu, 01 Aug 2019 02:34:00 +0000 https://blog.matthewgove.com/?p=733 I recently built a couple of DIY solar radiation shields to house outdoor temperature sensors I have at my house. I built one out of plastic plates and one out of plastic bowls because I was curious to see if one worked better than the other as a solar radiation […]

The post DIY Solar Radiation Shield: Results of the Bowls vs Plates Experiment appeared first on Matthew Gove Blog.

]]>
I recently built a couple of DIY solar radiation shields to house outdoor temperature sensors I have at my house. I built one out of plastic plates and one out of plastic bowls because I was curious to see if one worked better than the other as a solar radiation shield.

A Word on Solar Radiation Shields

The purpose of a solar radiation shield is simple. Block solar radiation from reaching temperature sensors so you can get accurate readings even if you do not have a location in the shade to mount your sensor.

The goal of solar radiation shields is to ensure temperature readings that are within the accuracy for your sensor’s calibration. They block the solar radiation and allow for ample air flow across the sensor.

Experiment Design

To truly test which solar radiation shield worked best, I wanted to expose them to the most extreme environment possible. Thankfully summers here in Phoenix are more than happy to oblige. As an added bonus, they get exposed to direct sunlight in the early mornings and late afternoons for about 6 to 8 weeks on each side of the summer solstice.

Running the experiment in early June would satisfy both requirements: extreme heat/dry and exposure to direct sunlight. June is typically the hottest and driest month in Phoenix. Afternoon highs pretty routinely hit 110°F (about 45°C). Relative humidity typically bottoms out between 5% and 15%.

I took hourly observations from the temperature sensors in both the bowls and the plates and compared them to the observations at Phoenix Sky Harbor International Airport. Sky Harbor is the official National Weather Service observation station for the Phoenix area. I also compare them to the observations at Luke Air Force Base in Glendale, which is closer to my house. The observations were taken Monday through Thursday for three weeks.

Hypothesis

I went back and forth for a long time on which one I thought would perform better. The key to obtaining accurate temperature readings from inside a solar radiation shield is proper airflow across the sensor. Because of that, my gut was telling me that the shape of the plates would allow for slightly better airflow across the sensor. I opted to stick with my gut feeling for my official hypothesis and go with the plates.

Results

Before diving into the results, there are a few things we need to take into consideration:

  • The greater Phoenix area is one of the largest urban heat islands in the world. You must account for the heat island when comparing the National Weather Service observations. The Weather Service takes observations inside the city, while my house is outside the city.
  • The sensor inside the plates was not connected to a data logger. It is missing data points for where I was sleeping, cooking, not paying attention, etc.

Week 1: May 20 – 23, 2019

Despite my best efforts to run the experiment in as extreme an environment as I could get, the week of May 20th was unseasonably cool across the desert southwest. Normal highs in Phoenix this time of year are right around 40°C/100°F. As you can see from the plots, we didn’t get anywhere close to that benchmark. However, it did provide a nice contrast to the observations I took once the weather heated back up.

DIY Solar Radiation Shield Experiment Results: Week 1

This dataset gave me pretty much what I was expecting:

  • Both day and night observed temperatures were slightly warmer in the city than at my house.
  • Instances of the homemade solar radiation shields having insufficient airflow or being in direct sunlight did not result in any major temperature anomalies.
  • The plates performed slightly better during the heating of the day. They both stayed slightly below the city observations and were closer to the National Weather Service observations.
  • There was no noticeable difference in the performance of the bowls and the plates at night.

Week 2: May 27 – 30, 2019

The Phoenix weather finally started warming up this week, giving some interesting results as the extreme environment returned. The beginning of the week started cool, but became hot by the time Wednesday and Thursday rolled around.

DIY Solar Radiation Shield Experiment Results: Week 2

Right away, a few things really jumped out at me from this plot

  • The effect of the solar radiation shields being in the sun, especially in the late afternoon (about 3 to 6 PM), is very noticeable, especially in the hotter temperatures.
    • Remember the plates and bowls (yellow and blue lines, respectively) should be just below the Sky Harbor (green) and Luke AFB (red) observations due to effects of the urban heat island.
  • When the bowls/plates are in direct sunlight in the morning, they go into the shade at about 9:00 AM. Notice how the observed temperature on the blue and yellow lines temporarily drops right around 9 AM each day. Both solar radiation shields definitely feel the effect of being in direct sunlight.
  • Interestingly, the difference between the bowls/plates observations and the National Weather Service observations remained pretty constant as high temperatures increased.
  • Once again, the plates performed slightly better, as they were closer to the official National Weather Service observations.
    • Bowls are mounted west of the plates and are in direct sunlight for slightly longer in the afternoon. However, that difference is only just a few minutes, so its effect should be minimal.
  • The sensors performed as expected at night, with temperatures at my house reading slightly below the city observations.

I expect that the final week of the experiment to be the hottest of the three. It will be interesting to see if those results corroborate what we’ve seen so far.

Week 3: June 3 – 6, 2019

The final week of the experiment was by far the hottest, with afternoon temperatures finally exceeding 40°C or about 105°F. Keep Week 2’s results in mind as we look at the results of Week 3.

Bowls vs Plates Experiment Results: Week 3

The patterns from Week 2 are also evident in Week 3.

  • The effects of both solar radiation shields being in direct sunlight are clearly evident on all four days.
  • The observed temperatures were as expected at night, with temperatures at my house being slightly lower than the city observations.
  • The temporary drop in observed temperatures in the bowls and plates occurred at about 9 AM each morning when the solar radiation shields went into the shade.
  • The difference between the bowls/plates and the National Weather Service observations do vary a bit more than in Week 2. This is most likely due to higher wind velocities.
  • Again, the plates seemed to perform slightly better than the bowls during the day.

The Verdict

This was a fun and interesting experiment with a few curveballs. I was able to draw the following conclusions and propose solutions to problems that I discovered during the experiment.

  • The plates performed slightly better than the bowls in both cooler and hotter weather.
  • Both types of dinnerware were affected by being in direct sunlight in the early morning and late afternoon.
    • Thankfully the solar radiation shields are completely out of direct sunlight for close to 10 months out of the year.
  • Adding fans to the solar radiation shields to ensure an ample and consistent airflow across the sensor will noticeably reduce the effects of being in direct sunlight. It likely will not eliminate them completely.
  • The sensors all performed as expected at night.
  • Wind velocities should be recorded as well.
  • All sensors should be connected to dataloggers so I can collect data 24 hours a day.

In the future, I would like to run the experiment again in both winter and summer settings. The results of those experiments combined with the results of this experiment would then give us a definitive answer as to whether bowls or plates make the best DIY solar radiation shields.

The post DIY Solar Radiation Shield: Results of the Bowls vs Plates Experiment appeared first on Matthew Gove Blog.

]]>
DIY Weather Station, Part 4: Database Configuration and Programming the Sensor Readings https://blog.matthewgove.com/2019/07/21/diy-weather-station-part-4-database-configuration-and-programming-the-sensor-readings/ Mon, 22 Jul 2019 01:44:00 +0000 https://blog.matthewgove.com/?p=725 Wow, it’s hard to believe that the hardware for taking sensor readings for the DIY weather station is finally all in place. Now, it’s time to dive into the programming and the software end of things that makes everything click. The software for the sensor and the data logger consists […]

The post DIY Weather Station, Part 4: Database Configuration and Programming the Sensor Readings appeared first on Matthew Gove Blog.

]]>
Wow, it’s hard to believe that the hardware for taking sensor readings for the DIY weather station is finally all in place. Now, it’s time to dive into the programming and the software end of things that makes everything click. The software for the sensor and the data logger consists of two primary parts:

  • A database to store the data
  • A script that takes sensor readings, QA/QC’s those readings, and logs them in the database.

From Sensor To Database

The flow of data from the sensors to the weather station’s primary database, is as follows. The primary database contains the full set of weather data. It logs data every 5 minutes. The weather station’s web interface displays those data. Data for which I do not have sensors are obtained from the National Weather Service.

  1. A Python script reads data from the sensors and logs them in the data logger every minute
  2. The Python script QA/QC’s the raw data read from the sensor.
  3. The script puts salid or “good” data points into the database, in default metric units. It flags invalid or “bad” data points and puts them into a separate table so the primary database cannot get any “bad” data.
  4. Every 5 minutes, a Python script associated with the primary database queries the database containing the sensor data. It takes the most recent set of “good” data points that fall within that 5 minute period and inserts that data into the primary database, along with the data it obtains from the National Weather Service. If no “good” data points fall within that 5 minute period or if the sensor or the data logger are offline, it obtains the data from the National Weather Service.
  5. Once per day, data measured by the sensors are synced with the primary database to ensure that the primary database has the correct sensor data in the event of a network outage.

Database Design

I could easily write an entire post about database design, so I’m not going to go into too much detail of why I made the decisions to design the database the way I did. Instead, here is my list of requirements for the database:

  • Uses all the standard practices of relational databases
    • Only store raw, non-calculable data from the sensors.
      • Data that can be calculated from the sensor output do not need to be in the database.
      • This database is separate from the primary database for the DIY weather station, which I described above
    • Properly normalized to third normal form
    • Easily scalable up and down
  • Store information about the sensors (manufacturer, model, serial number, etc)
  • Have a mechanism to handle bad data points flagged by a QA/QC system
  • Be easily queryable to pull data into the primary database for the DIY weather station (more on this later in this post)

From that list of requirements, this is the EER diagram I came up with for the sensor database.

Sensor database design for the DIY Weather Station

Note here that the “good” data are put into the “measurement” table, while the “bad” data flagged by the QA/QC mechanism are put into the “measurement_to_qaqc” table.

The Python Script

The Python script is where all the magic happens. Its main purpose is to QA/QC the data it reads from the sensors, essentially acting as a guard so bad data points do not get into the weather station’s primary database.

The first iteration of the QA/QC algorithm is quite simple, and is to a degree, a bit crude. Essentially all it does is to just ensure that the data are within acceptable ranges. For example, the script flags and removes relative humidity readings of 3,285%, as well as the temperature readings of -145°C.

I haven’t begun coding a version 2 of the QA/QC algorithm, but it will look at trends in the data and eliminate unrealistic spikes. For example, if the sensors show that the temperature rose 20°C over the course of a couple minutes, that would be flagged and removed by the QA/QC algorithm. In the current first version, you could in theory have temperature readings of 10°C and 30°C within a minute of each other and the algorithm would not flag it, because both temperature values are within the acceptable or realistic temperature range.

Python Classes to Parse Sensor Readings

Within the Python code, there are classes for each type (make/model) of sensor. To instantiate an object of one of these classes, they are passed the GPIO pin in the data logger to which they are connected, as well as the Sensor ID, which is the primary key in the “sensor” table in the database described above.

The sensor classes also perform the following:

  • Read the raw data from the sensors and QA/QC those readings.
  • Round all data readings to a maximum of 5 decimal points, which the database requires
  • Inserts the data read from the sensors into the database.
  • Convert raw pressure to mean sea level pressure
  • Run a test measurement. The user manually calls the test method, which takes a sensor reading and dumps the data to a Terminal window without interacting with the database.

The primary script in the data logger Python module performs the following steps to log the sensor data:

  1. Instantiates an instance of the appropriate sensor class for each sensor in the weather station network
  2. Loops through each sensor
  3. Reads the data from the sensor
  4. QA/QC’s the data from the sensor
  5. Inserts the “good” data into the database

Well that just about wraps up the sensor and data logger portion of the DIY weather station project. I certainly had a blast designing and building, and I hope you enjoyed reading about it. Until next time.

The post DIY Weather Station, Part 4: Database Configuration and Programming the Sensor Readings appeared first on Matthew Gove Blog.

]]>
DIY Weather Station, Part 3: Installing the Data Logger and Connecting the Sensors https://blog.matthewgove.com/2019/07/10/diy-weather-station-installing-the-data-logger-and-connecting-the-sensors/ Thu, 11 Jul 2019 05:57:00 +0000 https://blog.matthewgove.com/?p=705 For our DIY Weather Station, we’ve built our sensor housings and run power and internet out to the data logger site. It’s time to install the data logger and hook up our sensors. Thankfully, this is much easier than getting power and internet to the site. First, let’s take a […]

The post DIY Weather Station, Part 3: Installing the Data Logger and Connecting the Sensors appeared first on Matthew Gove Blog.

]]>
For our DIY Weather Station, we’ve built our sensor housings and run power and internet out to the data logger site. It’s time to install the data logger and hook up our sensors. Thankfully, this is much easier than getting power and internet to the site. First, let’s take a look at what we need to ensure we have all the materials on hand. I am planning to add more sensors in the future. For now we are only adding sensors for temperature, humidity, and barometric pressure.

Supplies Needed

  • Raspberry Pi, with case and power adapter
  • GPIO Ribbon Cable for the Raspberry Pi, which makes it easier to connect the sensors.
  • MPL3115A2 I2C Barometric Pressure, Altitude, and Temperature Sensor
  • AM2302 Wired Temperature/Humidity Sensor
  • 22 AWG Wire Jumpers, to connect the sensors to the Raspberry Pi
  • Thermostat Wire
  • Breadboard for Testing (optional)
DIY weather station project supplies laid out on my desk
DIY weather station supplies laid out on my desk

Do a Test Run Inside First

Before installing everything at the data logger site, I always do a test run in a controlled environment indoors first. I set everything up in my home office, which allowed me to spread everything out. I could confirm that I hooked everything up correctly, and easily troubleshoot any issues. As a nice bonus, it let me work in the air conditioning instead of outside in this 100-plus degree heat.

First, get the Raspberry Pi up and running. The Pi will come with instructions for how to install the Raspbian operating system, which is a distribution of Debian Linux. After installing the operating system, simply plug in an external monitor and an ethernet cable. Then plug the Raspberry Pi in to turn it on. Once it boots up, log in and make sure you can get online and open a Terminal window.

Before connecting any sensors, shut down the Pi and unplug it. Never attempt to connect any sensors to the Pi while it is powered on! If you connect them to the wrong ports, you can damage both the sensor and the Pi by either running incorrect voltage through it or by shorting it out. After shutting down and unplugging the Raspberry Pi, connect the GPIO ribbon cable. Then get your sensors and jumper wires ready.

Each sensor will have instructions for which GPIO pin it needs to be connected to. Most sensors will connect to either 3 or 4 GPIO pins. One pin feeds the hot and neutral wires for power, and the remaining pins transfer data. For testing the sensor, connect the sensor to the GPIO pin with the jumper wires. If the sensor has wires built into it, which our temperature/humidity sensor does, you can use those.

Sensor wires attached to the Raspberry Pi's GPIO ribbon cable
Sensor wires attached to the GPIO ribbon cable

A Few Tips for Connecting the Sensors

  • Most sensors support using both 3.3 Volts and 5 Volts for power. For this test (and any other application where you are using short lengths of wire), always use 3.3 Volts. You may damage the sensor if you use 5 volts. Only use 5 volts is you have long lengths of wire. Unfortunately the definition of “long” varies depending on what type of sensor you’re using.
  • Each sensor will come with instructions telling you which GPIO pin to connect it to. Please be aware that if you’re using a GPIO ribbon, those pins will be a mirror image of the GPIO pin diagram that comes with the Raspberry Pi. If you’re connecting the sensor directly to the pins on the Pi’s board (i.e. not using a GPIO ribbon), that diagram will be correct.

Once you have the sensor(s) connected, it’s time to test them, so power the Pi back on.

DIY weather station sensors connected to the GPIO ribbon with jumper wires
Sensors connected to the GPIO ribbon with jumper wires

Test the Sensor After Connecting It

Most sensors come with scripts so you can verify that they’re working. Both sensors I’m connecting to the data logger come with Python scripts for testing, which I downloaded from GitHub. After installing any dependencies those scripts require, run the test script in a Terminal window. If the sensors are connected correctly, the sensor readings will appear in the Terminal window’s output. Make sure the sensor readings are accurate, too! If you’re in the United States, be aware that nearly all test scripts will output sensor readings in metric units by default.

Test script output showing pressure, altitude, and temperature
Test script output showing pressure, altitude, and temperature

Tip #1: Once you have verified the sensors are working correctly, I highly recommend taking a picture of the GPIO ribbon that shows each sensor wire connections. That way you can easily re-create the setup when you move it to the permanent data logger site.

Tip #2: Before you move the Pi to the permanent data logger site, enable SSH logins and turn off the GUI so it will be command line only. I don’t have an easy way to get a monitor to the data logger site so it is much easier to just SSH into the Pi from my laptop over my local network.

Installing the System at the Data Logger Site

Now that we’ve confirmed the sensors are working, it’s time to permanently install them at the data logger site. Make sure you shut down and power off the Pi before unplugging any sensors to move them.

A Quick Note About Thermostat Wire

I chose to use thermostat wire to connect the sensors. It is made out of a high-quality, highly conductive metal (solid copper). The conductors in thermostat wire are the perfect size for connecting to the sensors and the Raspberry Pi. It comes in many different configurations (number of conductors). It’s also cost-effective and won’t break the bank.

For the DIY weather station project, I used 18/5 (18 gauge, 5 conductors inside) wire because the pressure sensor has four connections. 18/3 wire would work fine for the temperature/humidity sensor, which only has three connections. For more complex sensor configurations, I have seen thermostat wire with up to 9 conductors available at most hardware stores.

Installing the Sensors

We are connecting two sensors to the data logger here: the temperature/humidity sensor and the barometric pressure sensor. We will mount the barometric pressure sensor in the data logger housing with the Raspberry Pi. It can be connected in the same manner it was for our inside test using a short piece of thermostat wire.

The temperature/humidity sensor is a little trickier, but if you’ve made it this far, it’s nothing you can’t handle. Before diving in, remove about 8 cm (3 in) of the jacket on the thermostat wire to reveal the conductor wires inside.

Put the Sensor in its Housing Before Connecting any Wires

The first step is to mount the sensor in the solar radiation shields we created in Part 1 of the DIY weather station project. I hung a small piece of wood in the radiation shield’s cavity and screwed the sensor to that. Make sure there is a hole in the top of the radiation shield to run the thermostat wire through. The hole should be just barely big enough to fit the wire through. After attaching the sensor to the wood, feed the end of the thermostat wire through the hole in the top of the radiation shield. Connect the sensor wires to the conductors of the thermostat wire using gray wire nuts. You can cut off any unused conductors in the thermostat wire or fold them back so they’re out of the way.

Sensors mounted in a solar radiation shield for the DIY weather station
Sensors mounted in one of my solar radiation shields

Next, securely run the thermostat wire back to where the data logger will be mounted. In my setup, I ran the wire through a conduit attached to the shepherd’s pole from which the radiation shield hung, then underground and up to the data logger through the same conduit through which the power and internet cables run.

Wired sensor connected to data logger housing for the DIY Weather Station
Solar radiation shield with sensor wired to data logger housing

An Important Note About Voltage and Wire Length

The DIY weather station’s analog temperature sensor can only support a maximum wire length of about 3.5 meters (11.5 ft). If you cannot get a sensor reading using 3.3 volts, use 5 volts instead. You’ll likely need to use 5 volts for wire lengths greater than about 1 meter. If you still cannot get a sensor reading using 5 volts, you’ll have to shorten the wire. By contrast, the digital pressure sensor can support a wire length up to approximately 30 meters (100 ft).

Installing the Data Logger

The data logger can be mounted in the sprinkler box housing we installed on the side of the house in Part 2 of this project. Run pieces of string through holes in the bottom of the Raspberry Pi’s case to attach it to the mount points inside the sprinkler box.

After mounting the Raspberry Pi, connect the sensor wires to their corresponding GPIO pins. If you have issues connecting the thermostat wire directly to the GPIO pins, connect one end of a jumper wire to the GPIO pin and attach the other end of the jumper wire to the sensor wire with a gray wire nut. After confirming the sensor wires are in place, connect the ethernet cable and plug the Raspberry Pi into the outlet in the sprinkler box to power it on. SSH into the Pi from your computer and run the test scripts again to confirm that the sensors are working properly.

Raspberry Pi mounted in data logger housing for the DIY Weather Station
Raspberry Pi and Pressure Sensor mounted inside the data logger housing

Now that the sensors and data logger are in place, in the next phase we will set up the database, program the data logger to pull data from the sensor at specific intervals, and connect the data logger to the rest of the DIY weather station.

The post DIY Weather Station, Part 3: Installing the Data Logger and Connecting the Sensors appeared first on Matthew Gove Blog.

]]>
DIY Weather Station, Part 2: Wiring Power and Internet to the Sensors https://blog.matthewgove.com/2019/06/25/diy-weather-station-wiring-power-and-internet-to-the-sensors/ Wed, 26 Jun 2019 02:19:37 +0000 https://blog.matthewgove.com/?p=686 Disclaimer: Working with electricity can be extremely dangerous. If you’re not comfortable working with electricity or circuitry, please consult an electrician. This blog and its owners are not responsible for anything that happens to you while you are working with electricity. The sensors and database for my DIY weather station […]

The post DIY Weather Station, Part 2: Wiring Power and Internet to the Sensors appeared first on Matthew Gove Blog.

]]>
Disclaimer: Working with electricity can be extremely dangerous. If you’re not comfortable working with electricity or circuitry, please consult an electrician. This blog and its owners are not responsible for anything that happens to you while you are working with electricity.

The sensors and database for my DIY weather station are up and running in my office. It’s finally time to install them in their permanent home. The first step is to mount the housing for the datalogger on the side of the house. I chose the spot because it’s on the north side of the house, where it’s out of direct sun (except for early evenings in the summer), and it’s right next to the junction box that brings the ethernet into the house. For the housing, I used a sprinkler timer box, which are readily available at any garden center or hardware store.

A sprinkler box will house the DIY weather station
Sprinkler Box Mounted on the Side of the House

The sprinkler box comes with a GFCI outlet in it, providing a weatherproof location to plug in the Raspberry Pi that will be used as a datalogger. To connect the power, you’ll need some 14/3 wire to connect the new outlet to an existing circuit. Full disclosure: this setup likely does not comply with electrical codes. Please reference your local electrical codes if you want to bring your setup up to code.

Connect the Power Source

Important: Before doing anything with electricity, always turn the power off to the circuits with which you are working at the breaker. After turning the power off, test an outlet on the circuit to confirm that the power is actually turned off. You’d be amazed at how often you think you’ve turned the power off when you actually turned off a different circuit. Failure to shut off the power can result in serious injury or death.

First, peel back the jacket on the wire to reveal the three wires inside. Then strip the ends of the wires.

Positive, neutral, and ground wires ready to be connected.

Next, insert the three wires into the corresponding terminals on the outlet. For a GFCI outlet, they must go in the end marked “Line”. Remember my “B-to-B” rule: black-to-brass. The black (hot) wire, which the electricity comes in on, goes to the brass screw. The white neutral wire, which returns the electricity to the source to complete the circuit, goes to the silver screw, and the green or bare wire goes to the ground terminal. Tighten the screws and give each wire a tug to ensure they’re secure in the terminals.

Wires connected to the plug that goes inside the DIY weather station housing.

Next, run the wire through the hole in the bottom of the sprinkler box and screw the outlet into place. Connect the other end of the wire to the power source. I recommend you connect the power source in the same manner as you would add an outlet to any other circuit in the house – by connecting the wire to the “Load” terminals on any existing GFCI outlet.

Run the Wires for Electrical Power

If you’re running the wire through conduits or any other channels, run them through there now. Then connect the other end of the wires to the power source. I recommend temporarily turning the power back on and testing the outlet with a GFCI tester to make sure you wired everything correctly.

GFCI Outlet for the DIY weather station mounted in the sprinkler box with power cord connected
GFCI Outlet in Sprinkler Box with Power Cord Connected

Run the Ethernet Cable

We’ll run the internet cable in a very similar manner. Run it through the other hole in the bottom of the sprinkler box, and through the same path as the electrical cables to your live network or internet connection.

Running the ethernet cable for the DIY weather station through the same conduit the power runs through
Feeding the ethernet cable through the same conduit the power is run through

Once the wires are all run, bundle any access wire, secure them with zip ties, and tidy everything else up.

Underground wires ready to be buried
Underground wires for the DIY weather station cleaned up and ready to be buried

Finally, fill in the trench to bury the underground wires. Turn the power back on, and you’ll be ready for the next stage: installing the datalogger, connecting the sensors, and connecting it to the network. We will cover all that, and more, in the next installment of this series.

DIY weather station with power and internet connected
Datalogger for the DIY weather station housing mounted and wired with power and internet, and ready to install the datalogger and sensors.

The post DIY Weather Station, Part 2: Wiring Power and Internet to the Sensors appeared first on Matthew Gove Blog.

]]>
DIY Weather Station, Part 1: Building a Solar Radiation Shield from Scratch https://blog.matthewgove.com/2019/05/29/diy-weather-station-building-a-solar-radiation-shield-from-scratch/ Thu, 30 May 2019 05:32:22 +0000 https://blog.matthewgove.com/?p=673 Over the past two years, I have been (very slowly) building a homemade DIY weather station and data logger using a network of sensors and Raspberry Pi’s. My plan all along has been to collect as much data as I can from sensors in my back yard, and get the […]

The post DIY Weather Station, Part 1: Building a Solar Radiation Shield from Scratch appeared first on Matthew Gove Blog.

]]>
Over the past two years, I have been (very slowly) building a homemade DIY weather station and data logger using a network of sensors and Raspberry Pi’s. My plan all along has been to collect as much data as I can from sensors in my back yard, and get the rest of the data from the National Weather Service. I recently got to the stage where it was time to add sensors. The first sensors I added were temperature, humidity, and barometric pressure.

The biggest challenge of thermometry, especially in a climate like Arizona, is obtaining accurate temperature readings when you routinely have to deal with extreme heat and intense sun. The solution is to use a solar radiation shield, which is an enclosure in which you can mount temperature sensors, which blocks the sun, but still allows ample air flow over the sensor so you can obtain accurate temperature measurements without access to shade.

Unfortunately, quality solar radiation shields that actually work can be rather expensive. Solar radiation shields for just the cheap outdoor thermometers you buy at the hardware store generally run $50 to $100. When you have multiple sensors you need to house (I have 3), it can quickly become prohibitively expensive. For avid DIY enthusiasts like me, coupled with being a cheap skate at times, building one from scratch was a no brainer.

Materials List for a DIY Solar Radiation Shield

  • 5 or 6 white or light gray plastic bowls or plates. Get the cheap ones from your local dollar store. I used four 79-cent bowls from Target for a total of $3.16.
  • Three 1/4-20 x 12 in threaded rods. I picked them up at Home Depot for $0.98/each.
  • One 1-in U-bolt. This is used for hanging the solar radiation shield from a pole or overhang. $0.95 at Home Depot.
  • At least 1 meter (3 feet) of 1/2 in PVC pipe. The pipe will be cut up and used as spacers for the bowls/plates. You can get a 10 foot piece from Lowe’s or Home Depot for $2.20.
  • Three 1/4-20 lock nuts. $0.15/each at Ace Hardware.
  • Three 1/4-20 hex nuts. $0.12/each at Ace Hardware.
  • Six 1/4 in washers. $0.17/each at Ace Hardware

Total Cost: $10.57

Procedure

Please Note that my original design used 5 cm spacers and 5 plastic bowls, which you see in the pictures, but the system was not getting enough air flow, resulting in abnormally high temperature readings. I modified the design to use 7 cm spacers and 4 plastic bowls.

Drill Holes in the Bottom of Each Bowl

1. Using a hole saw, cut holes in the center of all but one of the bowls. Those holes should be just big enough to hold your sensor(s). The sensor for my DIY weather station is small, so I used a 2 inch (50 mm) hole.

2. Drill three 1/4 inch holes in an equilateral triangle pattern around the edges. You will eventually put the threaded rods through these holes, which hold the shield together. The easiest way to ensure they line up is to stack a drilled bowl on top of an undrilled bowl and mark the hole locations with a pencil.

3. Drill 2 holes in the remaining bowl (the one you didn’t cut with the hole saw) for the U-Bolt. Center the U-Bolt so the solar radiation shield hangs evenly. Mount the U-Bolt in these holes.

Materials for the DIY Solar Radiation Shield

Calculate the Spacing Between Each Bowl

4. Using a saw, cut the PVC pipe into equal length sections in the length of your desired spacer (how far you want the bowls/plates apart). I cut mine into 7 cm lengths.

5. Thread a washer and one of the lock nuts about 1 inch onto the ends of each threaded rod. Make sure you thread them all on the same distance, or the solar radiation shield will not hang level.

6. Insert the three threaded rods through the holes in one of the bowls. If you’re worried about the threaded rods sticking out the top, put spacers between the lock nut and the bottom bowl. You may need a fender washer to hold the spacers in place.

Calculating the spacing of bowls in the DIY solar radiation shield

Assemble Your Solar Radiation Shield

7. Put a spacer on each threaded rod, and then add the next bowl. Repeat until you’ve used all of the bowls. Make sure the bowl at the top of the solar radiation shield is the one with the U-Bolt. I recommend that you put your sensor in the shield during this assembly, as it can be difficult to mount sensors inside the shield once it’s fully assembled.

Fully assembled DIY solar radiation shield
Before I realized I had to put spacers on the bottom. It wouldn’t fit on the hanger with the threaded rods sticking out of the top that much.

8. Once all of the bowls are in place, put a washer and a hex nut on the top of the three threaded rods and tighten the hex nut until it’s snug. Don’t tighten it too tight, as you’ll risk cracking or breaking something.

The final product should look something like this. Hang temperature sensors at a height of 1.5 meters.

DIY Solar radiation shield connected to my weather station

And here is the one I built out of plates if you’re curious.

The cavity inside the plates (to mount the sensors) is 10 cm in diameter and uses spacers that are 3 cm in length.

Stay tuned for future posts where we will be digging into how to mount and wire the sensors and data logger, as well as the results of my experiment to determine whether the plates or the bowls makes a better solar radiation shield. If you have any questions, feel free to ask in the comments section below.

The post DIY Weather Station, Part 1: Building a Solar Radiation Shield from Scratch appeared first on Matthew Gove Blog.

]]>