Downloading historical weather station data using Advanced PowerShell functions

by Klaus Graefensteiner 17. June 2009 04:07

Introduction

The company I am working with produces EMI (Enterprise Manufacturing Intelligence) software. It basically does dashboards and reports for manufacturing data. It is always a challenge getting some real world data for demos, presentations, development and test. I decided to come up with my own canonical demo app called: Swimming Pool. I decided to create a simple PLC (Programmable Logic Controller) program that controls the water level of a swimming pool. The PLC basically opens the fresh water valve, when the water level gets lower than a certain threshold and, in case of heavy rain, switches the backwash valve to the drain setting and starts the circulation pump to pump pool water out of the pool and into the drain system. In my example the pool is outside and the water level is influenced directly and indirectly by the weather conditions. Simply put, lots of hot weather evaporates the water and rain water makes the level raise. To simulate the water level changes I thought about using historical weather data, feed it into the PLC and have it calculate the water level based on a simple weather model. My controller program can then work with this simulated input and open/close valves and start and stop pumps.

Swimming Pool Water Level

Figure 1: Floating on a swimming pool surface

Milestones

For this little project I came up with the following milestones.

  • Part 1: Download weather station history
  • Part 2: Transform data into a more usable format
  • Part 3: Make data readable from a PLC
  • Part 4: Write water level simulation and pool level control in PLC
  • Part 5: Record control events, weather data and pool level in a factory historian
  • Part 6: Model entities in VantagePoint based on PLC UDTs.
  • Part 7: Create reports in Excel, Trend, XY Plotter, XCelsius 2008 Dashboards

This blog post covers Part 1 and subsequent articles will continue with the remaining parts.

Weather Underground

I searched the internet to find a free source of weather history data. It wasn’t as easy as I thought. Some places provide web services to get current weather data, but not data from past days. Other websites would only sell weather history data. I came finally across Weather Underground aka Wunderground.com. This website is a hub for collecting and sharing weather data provided by a variety of weather station owners. You can find your current weather data is to use a map by using a map at http://www.wunderground.com/wundermap. You could just navigate to your location of interest and click on a pin on the map.

Wunderground Map  

Figure 2: Wunderground map

If you click on the station id link in the popup, then you get forwarded to a details page for this particular weather station. Here is an example URL http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCABEAUM3.

The details page has graphs of your weather station sample data and the records in tabular form.

Weather Graphs

Figure 3: Weather data graphs

Tabular Data

Figure 4: Weather tabular data

On the bottom of the table there is a button that lets you retrieve the data of the table as comma delimited file (CSV). The button basically uses the same URL as the page, but passes a format parameter: http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCABEAUM3&format=1. On top of the table are three combo boxes that let you pick a day from the past. In this case the URL looks something like this http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCABEAUM3&month=6&day=17&year=2009 or http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCABEAUM3&month=6&day=17&year=2009&format=1 if the CSV download format is requested. Now, this is where PowerShell enters the stage. I didn’t want to sit there for hours or days, change the dates in the combo boxes, hit the CSV download button and do the “SaveAs” dance. Instead I created a simple PowerShell script that lets me pick a start and an end date, download the weather data between these dates automatically and save the CSV files with station id and date in its name. KCABEAUM3_20090613.html is an example of a file name.

Design choices

I wanted to take advantage of some of the PowerShell 2.0 features for this weather data download script. In particular I was looking forward to play around with advanced functions, parameter sets and parameter validation attributes. I also tried out the new Try Catch Finally construct. I successfully applied the ValidateSet and the ValidateScript attributes and stumbled over the limitations of the ValidateRange attribute. I will report about this experience in a follow up blog post.

PowerShell script

Here is the PowerShell script. The key here is the use of the [DateTime] object for generating the URL strings and the SaveAs file names. The AddDays() function and the –f format operator make the string generation a piece of cake.

Download in action

Figure 5: Download in action

Work horse

   1: #Weather Stations http://www.wunderground.com/wundermap/?
   2:  
   3:  
   4: function Download-File ([string] $URL, [string] $SaveAsFile, [Switch] $WhatIf)
   5: {
   6:     if($WhatIf)
   7:     {
   8:         $URL
   9:         $SaveAsFile
  10:     }
  11:     else
  12:     {
  13:         $WebDownloader = new-object System.Net.Webclient
  14:         $StartTime = Get-Date
  15:         Write ("Download started at {0}" -f $StartTime)
  16:         Write ("Downloading {0} bytes from {1} and saving file as {2}" -f $Length, $URL, $SaveAsFile)
  17:         
  18:         try
  19:         {
  20:             $WebDownloader.DownloadFile( $URL, $SaveAsFile);
  21:         }
  22:         catch
  23:         {
  24:             throw $_
  25:         }
  26:         finally
  27:         {
  28:             $WebDownloader.Dispose();
  29:         }
  30:         
  31:         
  32:         $EndTime = Get-Date
  33:         $Duration = $EndTime - $StartTime
  34:         Write ("Download ended at {0}" -f $EndTime)
  35:         Write ("Download took {0:F} seconds, which is {1:F} minutes" -f $Duration.TotalSeconds, $Duration.TotalMinutes )
  36:         
  37:     }
  38: }
  39:  
  40:  
  41: function Get-WeatherStationHistory
  42: {
  43:     
  44:     [CmdletBinding(DefaultParameterSetName="PreviousDays")]
  45:     PARAM(
  46:         
  47:         [Parameter(Position=0, Mandatory=$false, ValueFromPipeline=$false, ParameterSetName="PreviousDays")]
  48:         [ValidateScript({([DateTime]::Now - $_) -ge "1/1/2006"})
  49:         [TimeSpan] $PreviousDays,
  50:         
  51:         [Parameter(Position=0, Mandatory=$false, ValueFromPipeline=$false)]
  52:         [ValidateScript({$_ -le [DateTime]::Now -and $_ -ge "1/1/2006"})
  53:         #[ValidateScript({$_ -le $LastDay })] Problem $LastDay is not defined yet
  54:         [DateTime] $FirstDay=[DateTime]::Now,        
  55:         
  56:         [Parameter(Position=1, Mandatory=$false, ValueFromPipeline=$false, ParameterSetName="NextDays")]
  57:         [ValidateScript({($FirstDay + $_) -le [DateTime]::Now})
  58:         [TimeSpan] $NextDays,
  59:         
  60:         [Parameter(Position=1, Mandatory=$false, ValueFromPipeline=$false, ParameterSetName="LastDay")]
  61:         [ValidateScript({$_ -le [DateTime]::Now -and $_ -ge "1/1/2006"})
  62:         [ValidateScript({$_ -ge $FirstDay })]
  63:         [DateTime] $LastDay=[DateTime]::Now,
  64:         
  65:         [Parameter(Position=2, Mandatory=$true, ValueFromPipeline=$false)]
  66:         [ValidateSet("KCARIVER16", "KCAMOREN6", "KCABEAUM3", IgnoreCase = $true)]
  67:         [String] $WeatherStationCode = "KCAMISSI5"
  68:  
  69:     
  70:     )
  71:     Process{
  72:     
  73:         switch ($PsCmdlet.ParameterSetName) 
  74:         { 
  75:             "NextDays"    
  76:             { 
  77:                 "Parameter Set NextDays"
  78:                 Process-Downloads -FirstDay $FirstDay -NumberOfDays $NextDays -Station $WeatherStationCode
  79:             } 
  80:             
  81:             "LastDay"       
  82:             { 
  83:                 "Parameter Set LastDay"
  84:                 $Days = $LastDay - $FirstDay
  85:                 Process-Downloads -FirstDay $FirstDay -NumberOfDays $Days -Station $WeatherStationCode
  86:             }
  87:             
  88:             "PreviousDays"       
  89:             { 
  90:                 "Parameter Set PreviousDays"
  91:                 $StartDay = [DateTime]::Now - $PreviousDays
  92:                 Process-Downloads -FirstDay $StartDay -NumberOfDays $PreviousDays -Station $WeatherStationCode
  93:             }
  94:         } 
  95:     
  96:     }# End Process
  97: }
  98:  
  99:  
 100: function Process-Downloads([DateTime] $FirstDay, [TimeSpan] $NumberOfDays, [String]$Station)
 101: {
 102:     $i = 0;
 103:     while( $FirstDay.AddDays($i) -le $FirstDay.AddDays($NumberOfDays.TotalDays))
 104:     {
 105:         $CurrentDay = $FirstDay.AddDays($i)
 106:         $i++
 107:  
 108:         $CurrentUrl = "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID={0}&month={1}&day={2}&year={3}&format=1" -f $Station, $CurrentDay.Month, $CurrentDay.Day, $CurrentDay.Year
 109:         $FileName = "{0}_{1}{2:D2}{3:D2}.html" -f $Station, $CurrentDay.Year, $CurrentDay.Month, $CurrentDay.Day
 110:         
 111:         try
 112:         {
 113:             Download-File -URL $CurrentUrl -SaveAsFile $FileName
 114:         }
 115:         catch
 116:         {
 117:             write-warning "$_`n`n"
 118:         }
 119:     }
 120: }
 121:  
 122:  

Usage example

   1: $Pwd = "C:\SCHREIBTISCH\BlogPosts\InWork\DownloadingHistorialWeatherStationData\"
   2: cd $Pwd
   3:  
   4: . .\Download-WeatherData.ps1
   5:  
   6: $Start = ([DateTime]::Now).AddDays(-8)
   7: $End = ([DateTime]::Now).AddDays(-1)
   8:  
   9: $Stations = "KCARIVER16", "KCAMOREN6", "KCABEAUM3"
  10:  
  11: $Start
  12: $End
  13: $Stations
  14:  
  15: $Stations | Foreach-Object {Get-WeatherStationHistory -FirstDay $Start -LastDay $End -WeatherStationCode $_}
  16:  

Download

Scripts and sample weather data can be downloaded here: DownloadingHistorialWeatherStationData.zip

Ausblick

The first step is done. Now I need to refactor the original data and convert it into a more generic format.

Tags: , , , , ,

Swimming Pool | Incuity | PowerShell | Weather Station

Comments

8/16/2009 11:36:43 AM #

pingback

Pingback from powerscripting.wordpress.com

Episode 80 – Klaus Graefensteiner «  PowerScripting Podcast

powerscripting.wordpress.com | Reply

Add comment




  Country flag

biuquote
  • Comment
  • Preview
Loading



Administration

About

Powered by:
BlogEngine.Net
Version: 1.5.0.7

License:
Creative Commons License

Copyright:
© Copyright 2009, Klaus Graefensteiner.

Disclaimer:
The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

Theme design:
This blog theme was designed and is copyrighted 2009 by Klaus Graefensteiner

Rendertime:
Page rendered at 3/11/2010 7:12:01 PM (PST Pacific Standard Time UTC DST -7)