How I built my earthin24 twitter bot
At work my team have recently taken over a golang app. To learn a new language I like to build something that keeps my interest and by the nature of exploring something of interest you tend to cover way more surface area of a language this way.
🌏 12pm 25th August - 12pm 26th August 2017 pic.twitter.com/KUL3iz4goQ
— Earth (@earthin24) August 26, 2017
Check out the code on github.
Space is awesome
If you follow me on twitter you may notice I retweet a lot of space stuff so building something space related was inevitable. Way back something came up in my twitter feed, glittering.blue, which is an awesome site that has a video of earth over 24hrs as seen from the incredible japanese himawari8 satellite. This satellite is in geostationary orbit and gives us a nice fixed view of Australia – where I’m from.
Live background image of earth
Initially I wrote a go cli tool to download the latest image of earth available from himawari8 and set it as my background, I was heavily inspired by the code from Charlie Loyd’s python script to download the images found on glittering.blue.
Why not a video every day
To extend the idea I thought why not generate a video everyday and post it to twitter, I did a search and found that a decent twitter name was still available, @earthin24. If you follow you’ll get a gorgeous video once a day.
Things I learned
My background is in front-end development so Go was new territory for me here’s some things I learned along the way while creating this bot.
goroutines and WaitGroup
Initially I had the for loop requesting each frame for the video synchronously which took quite a while as it would fetch 144 images, luckily go is really good at concurrency so I explored the sync package to help track goroutines and only exit once all images have downloaded. Using concurrency would allow me to create 144 goroutines very quickly and concurrently download all of them over a much shorter time period.
Starting a goroutine is as simple as prefixing a function with the go
keyword.
for i, _ := range frames {
go getImage(framePath(first), i)
}
However the process won’t wait for all goroutines to finish before exiting so we need to use sync.WaitGroup
func main() {
var wg sync.WaitGroup
for i, _ := range frames {
wg.Add(1)
go getImage(framePath(first), i, &wg)
}
wg.Wait()
}
For each goroutine we increment the WaitGroup
counter using Add
and at the end of the block we use wg.Wait()
which blocks from exiting until the counter has reached zero.
You’ll also notice I pass a pointer reference to WaitGroup
&wg
to the getImage
method so I can call wg.Done()
outside of the current function scope. For more on pointers see the asterisk and the ampersand – a golang tale.
func getImage(url string, name int, wg *sync.WaitGroup, date time.Time) {
defer wg.Done()
//...
}
wg.Done()
decrements the counter once it’s done. The defer
keyword means to delay the function call until the end of the function. It would be the same as just calling wg.Done()
at the end of the function but it is idiomatic Go to do this at the beginning using defer
.
Automated hands free artisanal space tweets
So far I’ve been running this for a few months and it has posted daily without me needing to think about it. It’s pretty straight forward but there’s a few more learning’s when I set up the cronjob.
bash date
On MacOS you can type date
to get the current date in your terminal, when my cronjob runs my bash script it needs to get today-1 in the following format YYYYMMDD:
date -v-1d +%Y%m%d
-v
making it very easy by being able to subtract a day from the current date using -1d
and then using the +%Y%m%d
for the date output_format so a date could come out like so 20170819
.
The only problem was when I tried to run this on my Linux based server the date command failed! Seems MacOS uses its own date version different from Linux, to subtract a day we need to use the --date="1 day ago"
argument.
date --date="1 day ago" +%Y%m%d
ffmpeg
The go app downloads all the frames ready to be passed into ffmpeg to actually generate the mp4 video that’s posted to twitter & instagram. Since some of the frames are dropped due to the satellite not processing the image at the time the sequential file naming of the files can sometimes skip some numbers throwing off ffmpeg. If you try and pass a sequentially ordered file name pattern to it, ffmpeg will error out when looking for a file name that may be missing.
To get around this we use -pattern_type glob
argument for ffmpeg and then just use a simple *.png
wildcard.
Instead of ffmpeg -i %4d.png
we need to do ffmpeg -pattern_type glob -i '*.png'
to cater for the potential non-sequential frames.
Amazing
It continues to blow my mind that I can get images from a satellite and actually do this, the Japanese Space Ageny (JAXA) is absolutely incredible for releasing this to the public.
Bonus
I also extended this to instagram @earthintwentyfour so if you don’t like twitter follow me there.