Remembering that a user has read a Twitter direct message is the hardest problem in computer science.

— Robin Ward (@evil_trout) March 29, 2013

I get annoyed when I read a Twitter Direct Message in my iPhone app, and the next time I open the website, it says I have a new Direct Message. And then when I open a Twitter app on another device, it's still unread. I have to read the message over and over, even if I replied to it. How can it be 2013 and this is allowed to be the case? How?

The pre-emptive TL;DR for this is because the way Twitter gives apps (and the website) data through its API only care about what's happening right now, and incidentally, what happened just before right now. There's just no method to say "this is the last thing I read" in Twitter API-ese, and the way that Direct Messages are handled make it even more complex for apps to deal with.

I like to save as many emails, pictures, chat logs, and other digital ephemera of my life as I can. For me, being able to see the emails and facebook messages leading up to a skydiving trip, or whatever it may be, adds that much more to just having pictures or video. As part of a general hoarding strategy, I decided I wanted some details about all of Kickstarter projects I backed, and went looking for an API I could use to pull the data down. Googling and digging through the site, all the data I could find were a simple year-on-year stats page and some blog posts about milestones the service has hit. As it happens, Kickstarter does have an API - they just haven't opened or documented it up yet.

Finding the API

Because it seemed like a safe place to start, I opened https://api.kickstarter.com, assuming that there would be a registration page, or maybe some docs. Alas, all I got was

    {"error_messages":["You are not authorized to access this resource."],"http_code":401,"ksr_code":"unauthorized"}

This is quite a bit better than nothing, as it tells me that there are resources I could potentially be authorized to access.

To say that I'm happy with the feedback I got about Storyboard would be a wild understatement. While I can't turn the nice words people said about it on metafilter or elsewhere into anything, I did work hard to add or improve things people had trouble with. Setup is still pretty complicated, but I do have ideas on how to make it better - those'll be coming in another few weeks.

In the meantime, I've added to and tried to improve the setup instructions.

If that doesn't work, please follow the normal setup instructions. If you have any troubles with them, please send me an email, get in touch with me on twitter, or create an issue on github. If you already have Storyboard, you should be able to update it:

storyboard --update

The main issue I wanted to fix this time around was that of improper subtitles. Relative to other metadata about video files, the sites that support subtitles are pretty spotty. To work around this, you're presented with the list of all available subtitles that are returned for your video file. While the example below says that these are all subtitles that should work with this exact video file, the presence of the alternate pilot underscores the bad data that can come through.

$ storyboard "Seinfeld 6x04.avi"
There are multiple subtitles that could work with this file. Please choose one!
All of these are subtitles made for this exact video file, so any should work.
1: 'Seinfeld - 6x04 - The Chinese Woman_eng.srt', added 2005-11-15 00:00:00
2: 'Seinfeld [1x00] - Alternative Pilot - Good News, Bad News - FoV -.srt', added 2005-09-04 00:00:00
choice (default 1):

Gifboard

More than a few seemed to assume that Storyboard was used to create the GIFs from TV shows and movies that litter tumblr, or if it could support doing that. Well, it turns out that adding GIF support was pretty simple because Storyboard already knew when a line of text was on screen, how to extract those frames, and add subtitles to them. All that was left was saving the GIF itself, which also turned out to be pretty simple, using the same Imagemagick tools that Storyboard already required.

I only put a few hours of work into this, and it so it's lacking a lof of options - most notably ones that could control the size and quality of the GIF. Along with not being able to specify exact time ranges, font colors, or other options I can't think of right now, this makes the results pretty take-it-or-leave-it. Nevertheless, I can't stop laughing at the GIF on the right, and so it passes muster with me.

Instead of cramming more into Storyboard, I broke it out into a seperate applicaton: gifboard. Using it is pretty simple:

$ gifboard -t "wearing a cape" "Seinfeld 6x04.avi"

Storyboard was born of my insane desire to consume videos without actually having to watch them. Normally that would involve putting the TV on in the background and ignoring the video while listening to the audio, but what about the reverse? All visual without the audio. On my kindle.

Storyboard is very much a work in progress, though it does work for most files that it has been tested against. Right now it will only create PDFs, but ePUB and Mobi are just a few commits away. The source is available on Github where pull requests are gladly accepted, especially ones that deal with open issues. Even just hearing back that Storyboard did or didn't work for a certain file can be of great help.

Installing

Storyboard requires Ruby 1.9.3, FFmpeg 1.1, and ImageMagick. For OSX, the INSTALL file has a lot more details, but if you already have a development environment setup, you shouldn't have to do much.

brew update && brew install ffmpeg imagemagick ghostscript libtool
gem install storyboard

For Ubuntu, installing imagemagick and ghostscript from apt will work, but you need to download and compile ffmpeg on your own for now.

Using Storyboard

From a terminal, just call storyboard

storyboard "Your Video File.mkv"

Storyboard will then output a file named Your Video File.pdf in the directory you ran the command from. With non-HD content, this process can take just a minute or two for a 22 minute file, and the output file will be around 12MB.

SSH known_hosts and Chef

A problem that the documentation for Chef's Deploy_resource talks about is a Chef run pausing while a program it runs waits for user input. One way this presents itself is with SSH's host fingerprint checking, which ensures that the host you're connecting to now is the same host you connected to earlier. When first connecting to a host with anything that runs over SSH, you'll see something like this:

root@precise64:/tmp# git clone git@github.com:rails/rails.git
Cloning into 'rails'...
The authenticity of host 'github.com (207.97.227.239)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)?

Chef can't type 'yes' for this, and so it waits. And waits. The second issue that can come up is having a keyphrase on the SSH key. Normally, this is a very good idea, but it just doesn't work well with automated deployments because, again, it prompts for input.

To get around this, The Chef docs recommend creating a wrapper around SSH called wrap-ssh4git.sh which disables Strict Host Checking and tells SSH to use a key that has had it's keyphrase removed. While there's no good way to use a keyphrase protected key, we can use SSH's built in tools to add hosts to the known_hosts file that SSH consults when connecting.

I've been working with a lot of virtual machines lately, and using RVM to install the version of ruby I need took longer than installing the base system. If you go through the motions of rvm install {VERSION}, you might just gloss over this message:

No binary rubies available for: ubuntu/12.04/x86_64/{VERSION}.<br />
Continuing with compilation. Please read 'rvm mount' to get more information on binary rubies.

Not reading the documentation for rvm mount might end up costing you a lot of time.