(c) 2009 by Tom Keffer <tkeffer@gmail.com>
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with
this program. If not, see
I wrote weewx over the winter of 2008-2009 for two reasons: it was a wet and miserable winter here in Oregon with not much to do, so there was no good reason not to, and because I wanted a simple, easy-to-understand server to run my Davis VantagePro2 weather station on a Linux box. I had been using wview, which is a very high-performance, stable, and feature rich system authored by Michael Teele with lots of users. Written in C, it's an amazingly efficient system that can run on very underpowered boxes. In exchange, it's huge (50,000+ lines of code), tightly integrated in with its companion library, radlib (another 18,000 lines), and so brittle that only Michael can reliably make modifications to it. But, if you're looking to be in good company and you want to run on inexpensive, featherweight machines such as the Linksys NSLU2, you can't beat it.
Having made a career in C++ and Java, I was also interested in some more modern languages, so I thought I'd try either Python or Ruby (although, truth be told, the roots of Python are nearly as old as C++!). I picked Python because its libraries are more mature and there are many more choices for third party libraries.
Weewx weighs in at about 5,000 lines of code (although, to be fair, it is missing many features such as support for other weather stations, support for metric units, support for alarms, etc.). It is also architecturally very simple.
Python V2.5 or V2.6 is required. I don't know whether the newer, V3.0 distribution would work.
The following external packages are required to use weewx.
Details on how to install these packages are below.
Python eggs are available for all of them, making them very easy to install using the Python setup tool "easy_install", part of the python-setuptools package.
To install python-setuptools on Debian and derivatives:
apt-get install python-setuptools
Once this has been installed, installing the rest of the packages is very easy
Weewx stores data pulled from the weather station in sqlite3 databases. A version of sqlite is included in many derivatives of Debian. My Ubuntu 8.10 system came with V3.5.9, which works just fine.
While Version 2.3.X of pysqlite is included with many versions of Python, the more recent 2.5.X is required in order to take advantage of transaction contexts. Hence, you may have to install. Because pysqlite builds a C library, you may have to install the Python development environment first, if you haven't already done so (example shown for Python V2.5, the version I used. Adjust as necessary):
apt-get install python2.5-dev
Having done that, you can build and install pysqlite:
easy_install pysqlite
Easily installed:
easy_install configobj
easy_install pyserial
easy_install Cheetah
My version of Python came with V1.1.6, which works great.
I run weewx on a 500MHz system with an AMD Geode processor and 512 MB of memory. Configured this way, it consumes about 4% of the CPU resources, so I think it could run on something considerably smaller and slower.
At this point, only the Davis VantagePro2 is supported, and even then, only the "Revision B" version (firmware dated on or after 22 April 2002). It would be very easy to port to a "Revision A" station or even the original VantagePro, but I don't have access to the hardware to test it.
If WEEWX_ROOT symbolizes the root location of the weewx directory hierarchy, then
By default, the location for WEEWX_ROOT is /home/weewx (instructions on how to change is below).
Start by unpack the tar ball (substitute your version for X.Y.Z) in any convenient directory where you have permission
tar xvf weewx-X.Y.Z
Then change directory into it:
cd weewx-X.Y.Z
Build the distribution
./setup.py build
Finally, install it.
sudo ./setup.py install
This will install with root directory WEEWX_ROOT=/home/weewx. If you wish to install someplace else, then edit the file weewx-X.Y.Z/setup.cfg to whatever you prefer. If you do this, you will also have to edit the weewx configuration file $WEEWX_ROOT/weewx.conf to reflect your choice.
If you are upgrading from a previous version of weewx, the install process will save your old template directories and will also merge any changes you've made to the configuration file into the new configuration file.
Strictly speaking it is not necessary at all to install with root privileges. Indeed, if you edit setup.cfg to install into a directory where you have write permissions, you can install without any root privileges.
In the following, $WEEWX_ROOT refers to the weewx root directory, generally /home/weewx.
Virtually every conceivable configuration option is in the configuration file $WEEWX_ROOT/weewx.conf. Most of the important ones are up near the front of the file.
Two main databases are created and populated by weewx
These are created automatically by the main program if they do not exist. If you do not like their nominal locations, you can change them in the configuration file.
Alternatively, if you have an existing wview database, it can be used instead, and then its data used to backfill the statistical database, using the configuration script configure.py. For example:
mkdir $WEEWX_ROOT/archive
cp /usr/local/var/wview/archive/wview-archive.sdb
$WEEWX_ROOT/archive/weewx.sdb
$WEEWX_ROOT/bin/configure.py --create-stats $WEEWX_ROOT/weewx.conf
$WEEWX_ROOT/bin/configure.py --backfill-stats $WEEWX_ROOT/weewx.conf
If your existing database is large, backfilling could take some time. On my modest 500 MHz fit-PC with 512 MB of memory it took a little over 4 minutes for a year and a half (25 MB) of data (while wview was running in the background).
The only two variables weewx tries to manage on the VantagePro is the time and the archive interval.
The time on the VP is automatically synchronized with the server every four hours. However, you should run a NTP daemon on your server to insure that it is synchronized with the correct time. Doing so will greatly reduce errors, especially if you send data to services such as the Weather Underground.
The archive interval is set in the main configuration file $WEEWX_ROOT/weewx.conf. Look for the entry archive_interval in the VantagePro section. Set it to the number of seconds. Valid entries are 60, 300, 600, 900, 1800, 3600, and 7200. However, if you are ftp'ing lots of files to a server, setting it to 60 seconds may not give enough time to have them all uploaded before the next archive record is due. If this is the case, you should pick an archive interval of at least 300 seconds, or trim the number of files you are using.
After setting to the desired interval, run the configure.py script to set it on the VantagePro. If it differs from the old archive interval, the main memory log of the VantagePro will be cleared.
$WEEWX_ROOT/bin/configure.py --configure-VantagePro $WEEWX_ROOT/weewx.conf
There are many, many options that can be set in the configuration file, weewx.conf. They are all documented in this section, although you can safely ignore most of them. The truly important ones, the ones you are likely to have to customize for your station, are shown in bold face and in blue.
While default values are provided for most options, the default configuration file explicitly sets them all, so you can see what they look like.
What follows is organized by the different sub-sections of the configuration file.
This section covers options relating to the entire weather station setup.
The station location should be a string that describes the geography of where you weather station is located, such as 'Hood River, Oregon'. Required. No default.
The lat/lon should be set in decimal degrees, negative for southern and eastern hemispheres, respectively. Required. No default.
Should be set to the altitude of the station. In this version, the only unit accepted is feet. Required. No default.
If your area uses a rain year that starts on something other than the first of January, you may want to set this variable. For example, set to 10 if your rain year starts in October (as mine does). Default is 1.
Unless you are lucky enough to start using weewx on the very first day of your rain year, some rain is likely to have fallen already, throwing off the totals for the first rain year. If you know it, set this variable to the amount of rain that has already fallen in your rain year. Be sure to also set variable rain_ytd_year. Default is 0.00
Set this variable to the rain year in which you started using weewx.Example: if your rain year starts 1 October and you installed weewx on 8 February 2007, then this variable should be set to 2006. The amount of rain set in variable rain_ytd will be added to the rain year you specify. No default.
This variable is available in the HTML templates. Set it to an appropriate URL to display a radar image for your area. No default.
Set to the type of hardware you are using. For this version, only 'VantagePro' is accepted. Required.
Set to how long to wait before declaring a socket time out. This is used when FTP'ing data to a web server or sending data to the Weather Underground. Twenty (20) seconds is reasonable. Default is 20.
This section is for options relating to the VantagePro hardware.
Set to the port name used by your station. Example, /dev/ttyUSB0 is a common location for USB ports under Debian, /dev/ttyS0 for serial ports. Required. No default.
Set to the baudrate of your station. The default is 19200.
Set to the desired archive interval of your station, in seconds. This variable is only used when setting up your station. Otherwise, this value is read directly from the station. Required if you configure your station. No default.
Set to the ID number of your Integrated Sensor Suite (ISS). This is used in the formula to calculate reception quality for wireless stations. The default is 1.
How long to wait in seconds after the top of an archiving interval before fetching new data off the station. For example, if your archive interval is 5 minutes and archive_delay is set to 15, then the data will be fetched at 00:00:15, 00:05:15, 00:10:15, etc. This delay is to give the station a few seconds to archive the data internally, and in case your server has any other tasks to do at the top of the minute. Default is 15 seconds.
How many seconds to wait for a response from the station before giving up. Default is 5 seconds.
How many seconds to wait before retrying again. Unless you have a good reason to, this value should not be changed from the default, as it is long enough for the station to offer new data, but not so long as to go into a new loop packet (which arrive every 2 seconds). Default is 1.2 seconds.
How many times to try again before giving up. Default is 4.
How often to check the VantagePro's onboard clock for drift, in seconds. Default is 14400 (every 4 hours)
The maximum amount of drift to tolerate, in seconds, in the VantagePro's onbouard clock, before resetting the clock. Default is 5.
What unit system is in use on your weather station hardware. Possible values are '1' (Imperial) or '2' (Metric). As far as I know, all Davis instruments support only Imperial. In any case, Imperial is the only system supported by weewx. Default is 1.
If you FTP your images and HTML files to an external web server, weewx can FTP them for you. It does an incremental update, that is, it only FTPs any files that have changed, saving outgoing bandwidth with your Internet connection.
If you do not use such a server, comment out the entire section, including the '[FTP]' sub-section heading.
user
Set to the username you use for your FTP connection to your web server. Required. No default.
password
Set to the password you use for your FTP connection to your web server. Required. No default.
server
Set to the name of your web server (e.g., www.threefools.org, in my case). Required. No default
path
Set to the path where the weather data will be stored on your webserver (e.g., 'weather'). Required. No default.
Weewx can easily be run from the command line. Start by making sure you have appropriate permissions to the serial port your weather station uses. For example, if you are using a plain old serial port:
sudo chmod 666 /dev/ttyS0
Then run the main loop program, giving the configuration file as its only parameter:
$WEEWX_ROOT/bin/weewxd.py $WEEWX_ROOT/weewx.conf
It should start by downloading any archive data from your weather station into the database $WEEWX_ROOT/archive/weewx.sdb. As the Davis VantagePro can store a couple thousand archive records internally, this could take a minute or two.
It will then start monitoring LOOP data, printing a short version of the received data on standard output, about once every two seconds.
First, select the appropriate run script. They can be found under $WEEWX_ROOT/start_script. Copy it to the proper location for your system:
| SuSE: | cp $WEEWX_ROOT/start_script/SuSE/weewx /etc/init.d |
| Debian/Ubuntu: | cp $WEEWX_ROOT/start_script/Debian/weewx /etc/init.d |
Make sure it is executable
| SuSE: | chmod +x /etc/init.d/weewx |
| Debian/Ubuntu: | chmod +x /etc/init.d/weewx |
Create symbolic links in the run level directories:
| SuSE: | ??? |
| Debian/Ubuntu: | update-rc.d weewx defaults 98 |
Incidentally, a nice tool for setting run levels is sysv-rc-conf. It uses a curses interface to allow you to change easily which run level any of your daemons runs at.
Weewx will now start automatically whenever your system is booted. You can also manually start, stop, and restart the weewx daemon:
/etc/init.d/weewx start
/etc/init.d/weewx stop
/etc/init.d/weewx restart
The sqlite3 database used by weewx (weewx.sdb) is completely compatible with the database used by wview (usually called wview-archive.sdb), at least as of Version 5.2.X. The schema and its semantics is identical. However, the statistical file stats.sdb is different, and must be rebuilt
[This section is very incomplete and probably wrong]
The following types are available to be used in your template:
| Type | min | mintime | max | maxtime | avg | sum | rms | vecavg | vecdir |
| barometer | X | X | X | X | X | ||||
| inTemp | X | X | X | X | X | ||||
| outTemp | X | X | X | X | X | ||||
| inHumidity | X | X | X | X | X | ||||
| outHumidity | X | X | X | X | X | ||||
| wind | X | X | X | X | X | X | X | X | |
| rain | X | X | X | X | X | X | |||
| dewpoint | X | X | X | X | X | ||||
| windchill | X | X | X | X | X | ||||
| heatIndex | X | X | X | X | X | ||||
| ET | X | X | X | X | X | ||||
| radiation | X | X | X | X | X | ||||
| UV | X | X | X | X | X | ||||
| extraTemp1 extraTemp2 extraTemp3 |
X | X | X | X | X | ||||
| soilTemp1 soilTemp2 soilTemp3 |
X | X | X | X | X | ||||
| leafTemp1 leafTemp2 |
X | X | X | X | X | ||||
| extraHumid1 extraHumid2 |
X | X | X | X | X | ||||
| soilMoist1 soilMoist2 soilMoist3 soilMoist4 |
X | X | X | X | X | ||||
| leafWet1 leafWet2 |
X | X | X | X | X | ||||
| rxCheckPercent | X | X | X | X | X |
Weewx logs many events to the system log. On Debian systems, this is /var/log/syslog. When troubleshooting the system, be sure to check it!
The primary goals of weewx are:
To meet these goals, the following strategies were used:
While weewx is nowhere near as fast at generating images and HTML as its successor, wview (this is partially because it uses fancier fonts and a way more powerful templating engine), it is 'fast enough' for all platforms but the slowest. I run it regularly on a 500 MHz machine where generating the 9 images used in the "Current Conditions" page takes just under 2 seconds. Compare this with wview's 0.4 seconds.
The main thread of the program is used to monitor the VantagePro. All interaction with the hardware goes through it. The program starts by downloading any data from the VP that has been stored on board, but not yet put in the weewx databases. If the statistical database doesn't exist (nominally located at $WEEWX_ROOT/archive/stats.sdb), or is out of date, then it will repopulate the missing data from the main database. Finally, it sets up a thread to publish data to the Weather Underground and other sites that interact through HTTP.
The software then puts the VantagePro in LOOP mode. In this mode, the VP offers up data every 2 seconds, sleeping in between. Weewx monitors the port and when new data is available, it adds it to the running statistical tally, in particular highs, lows, and wind rms data, kept in the stats database .
When an archive interval is due, typically every 5 minutes or so, then the software cancels the loop, and downloads the new archive data, putting it in the main database. It also uses the archive data to update averages in the stats database. It then starts a thread to process the data (specifically, to generate images and HTML pages, then FTP them to a web server, if desired).
Every four hours the code checks the on-board clock on the VantagePro and adjusts it if it differs by more than 5 seconds from the host's clock.
The code makes ample use of exceptions, to insure graceful recovery of things such as network outages.
This is a glossary of terminology used throughout the code.
| packet | Something obtained off the weather station. Frequently uses a complex internal encoding, so it requires some processing to be useful. |
| record | Something obtained off the SQL database. |
| archive packet | A packet obtained off the store on the weather station. For example, with a Davis VantagePro, it's obtained using their DMPAFT command. |
| loop packet | A packet with the current observations. For example, with a Davis VantagePro, it's obtained using their LOOP command. |
| archive record | A record obtained off the SQL database |
| tuple-time | An instance of the Python object time.struct_time. This is a 9-wise tuple that represent a time. It could be in either local time or UTC, though usually the former. See module time for more information. They are useful because they are a little closer in format to what the Davis VantagePro uses, although they still require a bit of processing. Variables carrying tuple time usually have a suffix '_tt'. |
| epoch time | Sometimes referred to as "unix time," or "unix epoch time." The number of seconds since the epoch, which is 1 Jan 1970 00:00:00 UTC. Hence, it always represents UTC (well.... after adding a few leap seconds. But, close enough). This is the time used on the sqlite archive and appears as type 'dateTime' in the SQL schema, perhaps an unfortunate name because of the similarity to the Python type 'datetime'. Very easy to manipulate, but it's an opaque big number. |
| time stamp | A variable in unix epoch time. Always in UTC. Variables carrying a time stamp usually have a suffix '_ts'. |
| datetime | An instance of the Python object datetime.datetime. Variables of type datetime usually have a suffix '_dt'. |
This package is imperial (U.S.) units only. However, it has been set up to make it easy to extend to metric.
In general, there are three different areas where the unit system makes a difference.:
The transition from 1 to 2, i.e., from data in the VantagePro to the database, is handled by a translation function. Right now, only one is supplied, weewx.VantagePro.translateToImperial. Others could easily be introduced.
The Python special value 'None' is used throughout to signal a missing data point. All functions expect it.
However, the time value must never be 'None'. This is because it is used as the primary key in the SQL database.