weewx weather system

Copyright

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses.

About weewx

I wrote weewx over the winter of 2008-2009 for two reasons: it was a wet and miserable winter here in Oregon so, aside from skiing, there was no good reason not to, and because I wanted a simple, easy-to-understand server to run my Davis VantagePro2 weather station on a Linux box. I had been using wview, which is a very high-performance, stable, and feature rich system authored by Michael Teele with lots of users. Written in C, it's an amazingly efficient system that can run on very underpowered boxes. In exchange, it's huge (50,000+ lines of code), tightly integrated in with its companion library, radlib (another 18,000 lines), and so brittle that only Michael can reliably make modifications to it. But, if you're looking to be in good company and you want to run on inexpensive, featherweight machines such as the Linksys NSLU2, you can't beat it.

Having made a career in C++ and Java, I was also interested in some more modern languages, so I thought I'd try either Python or Ruby (although, truth be told, the roots of Python are nearly as old as C++!). I picked Python because its libraries are more mature and there are many more choices for third party libraries.

Weewx weighs in at about 5,000 lines of code (although, to be fair, it is missing many features such as support for other weather stations, support for metric units, support for alarms, etc.). It is also architecturally very simple.

Prerequisites

Python

Python V2.5 or V2.6 is required. I don't know whether the newer, V3.0 distribution would work.

Required packages

The following external packages are required to use weewx.

sqlite3 (Version 3.6 or greater) A SQL database written in C. Comes with Debian and many other Linux distributions.
pysqlite (Version 2.5 or greater) The Python interface to sqlite3
configobj (Version 4.5 or greater) Manages the configuration file weewx.conf.
pyserial (Version 2.3 or greater) Manages the serial connection to the weather station
Cheetah (Version 2.0 or greater) The HTML templating engine
Python Imaging Library (Version 1.1.6 or greater) Also known as PIL, this is usually included in most Python distributions.

Details on how to install these packages are below.

Python eggs are available for all of them, making them very easy to install using the Python setup tool "easy_install", part of the python-setuptools package.

To install python-setuptools on Debian and derivatives:

apt-get install python-setuptools

Once this has been installed, installing the rest of the packages is very easy

sqlite3

This is included in many derivatives of Debian. My Ubuntu 8.10 system came with V3.5.9, which works just fine.

pysqlite

While Version 2.3.X of pysqlite is included with many versions of Python, the more recent 2.5.X is required in order to take advantage of transaction contexts. Hence, you may have to install. Because pysqlite builds a C library, you may have to install the Python development environment first, if you haven't already done so (example shown for Python V2.5, the version I used. Adjust as necessary):

apt-get install python2.5-dev

Having done that, you can build and install pysqlite:

easy_install pysqlite

configobj

Easily installed:

easy_install configobj

pyserial

easy_install pyserial

Cheetah

easy_install Cheetah

PIL

My version of Python came with V1.1.6, which works great.

System requirements

Processor

I run weewx on a 500MHz system with an AMD Geode processor and 512 MB of memory. Configured this way, it consumes about 4% of the CPU resources, so I think it could run on something considerably smaller and slower.

Weather Station

At this point, only the Davis VantagePro2 is supported, and even then, only the "Revision B" version (firmware dated on or after 22 April 2002). It would be very easy to port to a "Revision A" station or even the original VantagePro, but I don't have access to the hardware to test it.

Installing weewx

Directory structure

If WEEWX_ROOT symbolize the root location of the weewx directory hierarchy, then

$WEEWX_ROOT/bin is where the Python packages and scripts are installed;
$WEEWX_ROOT/weewx.conf is the configuration file;
$WEEWX_ROOT/templates is where the html templates live;
$WEEWX_ROOT/archive is the directory where the sqlite3 database lives;
$WEEWX_ROOT/public_html is where generated html and .png images are put.

By default, the location for WEEWX_ROOT is /home/weewx (instructions on how to change is below).

Start by unpack the tar ball (substitute your version for X.Y.Z) in any convenient directory where you have permission

tar xvf weewx-X.Y.Z

Then change directory into it:

cd weewx-X.Y.Z

Build the distribution

./setup.py build

Finally, install it.

sudo ./setup.py install

This will install with root directory WEEWX_ROOT=/home/weewx. If you wish to install someplace else, then edit the file weewx-X.Y.Z/setup.cfg to whatever you prefer. If you do this, you will also have to edit the weewx configuration file $WEEWX_ROOT/weewx.conf to reflect your choice.

If you are upgrading from a previous version of weewx, the install process will save your old template directories and will also merge any changes you've made to the configuration file into the new configuration file.

Permissions

Strictly speaking it is not necessary at all to install with root privileges. Indeed, if you edit setup.cfg to install into a directory where you have write permissions, you can install without any root privileges.

Configuring weewx

In the following, $WEEWX_ROOT refers to the weewx root directory, generally /home/weewx.

Virtually every conceivable configuration option is in the configuration file $WEEWX_ROOT/weewx.conf. Most of the important ones are up near the front of the file.

Configuring the databases

Two main databases are created and populated by weewx

$WEEWX_ROOT/$archive_file (nominally /home/weewx/archive/weewx.sdb)
$WEEWX_ROOT/$stats_file (nominally /home/weewx/archive/stats.sdb)

These are created automatically by the main program if they do not exist. If you do not like their nominal locations, you can change them in the configuration file.

Alternatively, if you have an existing wview database, it can be used instead, and then its data used to backfill the statistical database, using the configuration script configure.py. For example:

mkdir $WEEWX_ROOT/archive
cp /usr/local/var/wview/archive/wview-archive.sdb $WEEWX_ROOT/archive/weewx.sdb
$WEEWX_ROOT/bin/configure.py --create-stats $WEEWX_ROOT/weewx.conf
$WEEWX_ROOT/bin/configure.py --backfill-stats $WEEWX_ROOT/weewx.conf

If your existing database is large, backfilling could take some time. On my modest 500 MHz fit-PC with 512 MB of memory it took a little over 4 minutes for a year and a half (25 MB) of data (while wview was running in the background).

Configuring your weather station

The only two variables weewx tries to manage on the VantagePro is the time and the archive interval.

Time

The time on the VP is automatically synchronized with the server every four hours. However, you should run a NTP daemon on your server to insure that it is synchronized with the correct time. Doing so will greatly reduce errors, especially if you send data to services such as the Weather Underground.

Archive interval

The archive interval is set in the main configuration file $WEEWX_ROOT/weewx.conf. Look for the entry archive_interval in the VantagePro section. Set it to the number of seconds. Valid entries are 60, 300, 600, 900, 1800, 3600, and 7200. However, if you are ftp'ing lots of files to a server, setting it to 60 seconds may not give enough time to have them all uploaded before the next archive record is due. If this is the case, you should pick an archive interval of at least 300 seconds, or trim the number of files you are using.

After setting to the desired interval, run the configure.py script to set it on the VantagePro. If it differs from the old archive interval, the main memory log of the VantagePro will be cleared.

$WEEWX_ROOT/bin/configure.py --configure-VantagePro $WEEWX_ROOT/weewx.conf

Running weewx

From the command line

Weewx can easily be run from the command line. Start by making sure you have appropriate permissions to the serial port your weather station uses. For example, if you are using a plain old serial port:

sudo chmod 666 /dev/ttyS0

Then run the main loop program, giving the configuration file as its only parameter:

$WEEWX_ROOT/bin/weewxd.py $WEEWX_ROOT/weewx.conf

It should start by downloading any archive data from your weather station into the database $WEEWX_ROOT/archive/weewx.sdb. As the Davis VantagePro can store a couple thousand archive records, this could take a minute or two.

It will then start monitoring LOOP data, printing a short version of the received data on standard output, about once every two seconds.

Running as a daemon

First, select the appropriate run script. They can be found under $WEEWX_ROOT/start_script. Copy it to the proper location for your system:

SuSE:	cp $WEEWX_ROOT/start_script/SuSE/weewx /etc/init.d
Debian/Ubuntu:	cp $WEEWX_ROOT/start_script/Debian/weewx /etc/init.d

Make sure it is executable

SuSE:	chmod +x /etc/init.d/weewx
Debian/Ubuntu:	chmod +x /etc/init.d/weewx

Create symbolic links in the run level directories:

SuSE:	???
Debian/Ubuntu:	update-rc.d weewx defaults 98

Incidentally, a nice tool for setting run levels is sysv-rc-conf. It uses a curses interface to allow you to change easily which run level any of your daemons runs at.

Weewx will now start automatically whenever your system is booted. You can also manually start, stop, and restart the weewx daemon:

/etc/init.d/weewx start
/etc/init.d/weewx stop
/etc/init.d/weewx restart

Compatibility with view

The sqlite3 database used by weewx (weewx.sdb) is completely compatible with the database used by wview (usually called wview-archive.sdb), at least as of Version 5.2.X. The schema and its semantics is identical. However, the statistical file stats.sdb is different, and must be rebuilt

HTML Generation

Types

The following types are available to be used in your template:

Type	min	mintime	max	maxtime	avg	sum	rms	vecavg	vecdir
barometer	X	X	X	X	X
inTemp	X	X	X	X	X
outTemp	X	X	X	X	X
inHumidity	X	X	X	X	X
outHumidity	X	X	X	X	X
wind	X	X	X	X	X		X	X	X
rain	X	X	X	X	X	X
dewpoint	X	X	X	X	X
windchill	X	X	X	X	X
heatIndex	X	X	X	X	X
ET	X	X	X	X	X
radiation	X	X	X	X	X
UV	X	X	X	X	X
extraTemp1 extraTemp2 extraTemp3	X	X	X	X	X
soilTemp1 soilTemp2 soilTemp3	X	X	X	X	X
leafTemp1 leafTemp2	X	X	X	X	X
extraHumid1 extraHumid2	X	X	X	X	X
soilMoist1 soilMoist2 soilMoist3 soilMoist4	X	X	X	X	X
leafWet1 leafWet2	X	X	X	X	X
rxCheckPercent	X	X	X	X	X

Architectural notes

Goals

The primary goals of weewx are:

Architectural simplicity. No semaphores, no named pipes, no complex multi-threading to manage.
One code base. The goal is not to have to build and deploy for specific platforms or configurations. The same code base should be used in all situations. Ample configuration options should be provided so the user doesn't feel tempted to start hacking code. At worse, the user may have to subclass, which is much easier to port to newer versions of the code base.
Minimal reliance on external packages. So the user doesn't have to go chase them down all over the Web.
"Fast enough." Architectural simplicity trumps brute speed.
Support only the Davis VantagePro2 initially (that's what I have), but make no architectural decisions that lock out other stations.
As "pythonic" as I know how to make it. I'm a beginner Python programmer with two decades of experience in C++. I tried hard to not make it look like the latter language!

Strategies

To meet these goals, the following strategies were used:

A powerful configuration parser, ConfigObj, by Michael Foord and Nicola Larosa, was chosen to read the configuration file. This allows many options that might otherwise have to go in the code to go instead in a configuration file.
A powerful templating engine, Cheetah, was used. This allows many variables that I may not have thought of to be accessed from within the HTML templates, without starting to modify the code.
No static variables (except read only variables) to make it easy to simultaneously support multiple weather stations and to allow some level of multithreading.
Pure Python. The code base is 100% Python, so no builds of underlying C libraries need be done.

While weewx is nowhere near as fast at generating images and HTML as its successor, wview (this is partially because it uses fancier fonts and a way more powerful templating engine), it is 'fast enough' for all platforms but the slowest. I run it regularly on a 500 MHz machine where generating the 9 images used in the "Current Conditions" page takes just under 2 seconds. Compare this with wview's 0.4 seconds.

Terminology

This is a glossary of terminology used throughout the code.

packet	Something obtained off the weather station. Frequently uses a complex internal encoding, so it requires some processing to be useful.
record	Something obtained off the SQL database.
archive packet	A packet obtained off the store on the weather station. For example, with a Davis VantagePro, it's obtained using their DMPAFT command.
loop packet	A packet with the current observations. For example, with a Davis VantagePro, its obtained using their LOOP command.
archive record	A record obtained off the SQL database
tuple-times	9-wise tuples that represent a time. It could be in either local time or UTC, though usually the former. See module time for more information. They are useful because they are a little closer in format to what the Davis VantagePro uses, although they still require a bit of processing.
epoch time	Sometimes referred to as "unix time," or "unix epoch time." The number of seconds since the epoch, which is 1 Jan 1970 00:00:00 UTC. Hence, it always represents UTC (well.... after adding a few leap seconds. But, close enough). This is the time used on the sqlite archive and appears as type 'dateTime' in the schema. Very easy to manipulate, but it's an opaque big number.
time stamp	A variable in unix epoch time. Always in UTC.

Units

This package is imperial (U.S.) units only. However, it has been set up to make it easy to extend to metric.

In general, there are three different areas where the unit system makes a difference.:

On the weather station. As far as I know, the Davis VantagePro series supports only imperial units internally. So, any translation to metric must be done in one of the other two areas:
In the database. The unit system of any individual record is indicated by the "usUnits" field. The numerical value 1 indicates imperial. Other numbers haven't been assigned (yet). Right now, only imperial units is supported.
In the presentation (i.e., html and image files). Right now, whatever unit system is in use in the database is carried through.

The transition from 1 to 2, i.e., from data in the VantagePro to the database, is handled by a translation function. Right now, only one is supplied, weewx.VantagePro.translateToImperial. Others could easily be introduced.

None

The Python special value 'None' is used throughout to signal a missing data point. All functions expect it.

However, the time value must never be 'None'. This is because it is used as the primary key in the SQL database.