Introduction

At Ardan Studios we have spent the last 6 months, in our spare time and on weekends, building a consumer based mobile application called OutCast. The mobile application is tailored towards those who like spending time outdoors, whether that be fishing, hunting or any other type of activity.

The Main Weather and Buoy Screens

This first release of OutCast shows the conditions for the buoy stations and marine forecasts areas within the United States. All this information is updated every 10 minutes and there are map views with traditional grids and search.

The backend processing for buoy and marine data is built using Go and running at Iron.IO as a scheduled worker task. The buoy processing downloads a text file from the NOAA website and rips through it, updating MongoDB with any changes. The marine processing is a bit more complicated. This requires pulling down multiple web pages from the NOAA website and parsing out all the text. Go made building and running these tasks a breeze.

Another important aspect of OutCast is real time weather radar for the last 50 minutes. This has been very challenging on multiple levels. Mainly because we needed a real good image library that would run on Linux and could be integrated with Go. We were fortunate to find ImageMagick’s MagickWand C API and their Go package that provides the CGO bindings.

Processing images is an intense piece of work. Sometimes it takes 3 seconds to clean a single image. With 155 radar stations that need to be processed every 5 minutes, it took us several refactors to get things working well. The MagickWand library can only handle processing one image at a time. This restriction places a lot of stress on getting things done accurately within an acceptable amount of time.

Here is a sample of a radar image before and after processing:

There is another interesting constraint. NOAA updates the images every 120 seconds on different time boundaries. If the program can’t download all the images very quickly, the application could have images out of sync when they are animated across the map. Many radar images cross each other like in Orlando, FL. On this area of the map we have 3 radar images overlapping each other.

Sometimes the images are not available. The program goes out to get the image and it doesn’t exist. This creates problems with gaps in the timeline. When this happens, there is an alternate image location the program attempts to use. If the image is still not available, then the image from the previous run is used. Unless you are looking for it, you usually can’t tell.

Then you have the issue of updating S3 storage and MongoDB for each image. This again needs to happen quickly to prevent image overlays from being out of sync. At the end of the day, you want to do your best to make sure that the images for all the radar stations are in sync. This will provide the best user experience.

Radar image processing happens in three stages and runs on a VM at Digital Ocean with 2 Gig of memory and 2 Cores.

So how does Go help make this all happen every 5 minutes all day and all night without fail?

Stage 1: Download Images

There are 155 images that have to be downloaded. The images range from 1k to 20k in size depending on the activity of the weather at that moment. In this stage, the program spawns a Go routine for each image that needs to be downloaded. The Go program consistently does all 155 downloads in less than one second:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

 15:15:02 : main : downloadImages : Started

 -- Spawn Go Routines
 15:15:02 : main : downloadImages : Info : Image [1] of [155]
 15:15:02 : main : downloadImages : Info : Image [155] of [155]

 -- Sample Download Per Image
 downloadImage : Info : HEADER : Url , N0R/JAX_N0R_0.gif
 downloadImage : Info : HEADER : Last-Modified , [Fri, 06 Dec 2013 20:12]
 downloadImage : Info : HEADER : Content-Type , [image/gif]
 downloadImage : Info : HEADER : Vary , [Accept-Encoding]
 downloadImage : Info : HEADER : Cache-Control , [max-age=180]
 downloadImage : Info : HEADER : Expires , [Fri, 06 Dec 2013 20:18:02 GMT]
 downloadImage : Info : HEADER : Date , [Fri, 06 Dec 2013 20:15:02 GMT]
 downloadImage : Info : HEADER : Connection , [keep-alive]
 downloadImage : Info : HEADER : Server , [Apache/2.2.15 (Red Hat)]
 downloadImage : Info : HEADER : Accept-Ranges , [bytes]
 downloadImage : Info : HEADER : Content-Length , -1
 downloadImage : Info : HEADER : Image-Length , 6873

 -- All Images Complete
 15:15:02 : main : downloadImages : Completed

Stage 2: Image Cleanup

Now that all 155 images have been downloaded, they need to be cleaned using the ImageMagick API. Unfortunately, this can only be done with a single Go routine. Trying to clean more than one image at a time causes the program to pull a lot of memory. It really slows things down and can cause the program to be terminated by the OS. I have also seen other very odd behavior. The program will consistently complete this work in 90 seconds or less:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

 15:15:02 radar.go:453: main : cleanImages : Started
 15:15:02 radar.go:457: main : cleanImages : Info : Image [1] of [155]

 -- Sample Processing Per Image
 Worker-RIW : cleanImage : Started
 Worker-RIW : cleanImage : Info : ReadImageBlob
 Worker-RIW : cleanImage : Info : TransparentPaintImage
 Worker-RIW : cleanImage : Info : WaveImage
 Worker-RIW : cleanImage : Info : Crop
 Worker-RIW : cleanImage : Info : Resize
 Worker-RIW : cleanImage : Info : EqualizeImage
 Worker-RIW : cleanImage : Info : GaussianBlurImage
 Worker-RIW : cleanImage : Info : BrightnessContrastImage
 Worker-RIW : cleanImage : Info : ResetIterator
 Worker-RIW : cleanImage : Info : GetImageBlob
 Worker-RIW : cleanImage : Completed
 Worker-RIW : cleanImage : Info : Defer : PixelWand Destroy
 Worker-RIW : cleanImage : Info : Defer : MagicWand Destroy

 -- All Images Complete
 15:16:20 radar.go:477: main : cleanImages : Completed

Stage 3: Upload To S3 and Update MongoDB

The last stage requires uploading all the cleaned images to S3 storage, removing the old images from S3 and then updating MongoDB with the new list of available image files. I am using the goamz package from Ubuntu for accessing S3 and the mgo package from Gustavo Neimeyer to access MongoDB.

Just like when we download the images, this stage spawns a Go routine for each image that needs to be uploaded and recorded. The Go program consistently performs this work in one second:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

 15:16:20 radar.go:485: main : updateImages : Started
 15:16:20 radar.go:491: main : updateImages : Info : Image [1] of [155]
 15:16:20 radar.go:491: main : updateImages : Info : Image [2] of [155]
 15:16:20 radar.go:491: main : updateImages : Info : Image [3] of [155]

 -- Sample Processing Per Image
 collate : Started : StationId[RIW] FileName[US/WY/RIW/20131206-2015.gif]
 collate : Info : Remove : Minutes[50.01] FileName[US/WY/RIW/20131206-1935]
 collate : Info : Keep : Min[45.00] FileName[US/WY/RIW/20131206-1940.gif]
 collate : Info : Keep : Min[40.00] FileName[US/WY/RIW/20131206-1945.gif]
 collate : Info : Keep : Min[35.00] FileName[US/WY/RIW/20131206-1940.gif]
 collate : Info : Keep : Min[30.01] FileName[US/WY/RIW/20131206-1945.gif]
 collate : Info : Keep : Min[25.02] FileName[US/WY/RIW/20131206-1950.gif]
 collate : Info : Keep : Min[20.01] FileName[US/WY/RIW/20131206-1955.gif]
 collate : Info : Keep : Min[15.01] FileName[US/WY/RIW/20131206-2000.gif]
 collate : Info : Keep : Min[10.00] FileName[US/WY/RIW/20131206-2005.gif]
 collate : Info : Keep : Min[5.01] FileName[US/WY/RIW/20131206-2010.gif]
 collate : Info : Keep - New : FileName[US/WY/RIW/20131206-2015.gif]
 collate : Completed
 storeImageMongoDB : Info : Updating Mongo
 storeImageMongoDB : Completed
 storeImageS3 : Started : B[SRR-Dev] FileName[US/AK/APD/20131206-2015.gif]
 storeImageS3 : Info : Put File S3 : FileName[US/AK/APD/20131206-2015.gif]
 storeImageMongoDB : Completed

 -- All Images Complete
 15:16:21 radar.go:505: main : updateImages : Completed

Conclusion

This project has taught me a lot about Go. It is exciting to see how fast the Go routines can download the images and perform all the S3 and MongoDB work. Thanks to CGO, I was able to leverage a powerful image processing library and make calls directly from my Go code.

Currently we are porting the web service that powers the mobile application to Go. It is currently written in Ruby. We are using the beego package for our web framework, the goconvey package for our tests and the envconfig package to handle our configuration needs.

Our goal for OutCast is to provide people the ability to know in advance that the weekend is going to be great. We plan on using Go and MongoDB to analyze outdoor condition data with user preferences and experiences to deliver relevant information and forecasting. In the future, users will interact with OutCast by providing an experience review after their outdoor activities have ended.

Currently OutCast is only available in the Apple App Store. We will have our Android version complete in January 2014.