What I learned from fnstraj - Backend Edition

After spending as long as I have with fnstraj, I wanted to share what I’ve learned throughout this crazy experience. Beware - This is a technical post!

  • Perseverance and Rewrites: fnstraj is currently in it’s third rewrite! I started out in Python, finding it the easiest way to work with GrADS (NOAA’s weather service), and eventually had a proof of concept coded showing that the algorithm worked. I really hoped to bring it to the masses, web service style, and python quite simply wouldn’t cut it for that. My next major rewrite was in Javascript for the browser, which I was excited for and made really good progress on, until finding that it’s not possible to work with GrADS in the browser due to CORS policy. I gave up for a short while, and started on my third rewrite: one for Node.js. Node.js doesn’t seem to be the ideal environment for this sort of thing, involving many calculations and working with tons of HTTP requests, but in the end it became the perfect solution. Yes, the codebase is overtly complex, but it works wonderfully.

  • Queue Systems: Working with queue systems was very new for me. I really wanted to create something that would just sit there and run for months at a time, executing prediction items as they came in. I choose Heroku and Cloudant for this task. I wrote my own queue worker for fnstraj and it works well! I am still learning lots about how to queue more effectively through a new project, and plan on merging those changes into fnstraj in the future. It’s hard! Queue systems make tons of compromises, and I have a much better understanding of what the minimum requirements for a queue for heavy-processing items are now.

  • Data and Interface Standardization: I decided some time ago to work with CouchDB for all of fnstraj’s schema work. The datasets were expected to become much more complex as time went on, and that’s exactly what happened! Since CouchDB stores objects like their native Javascript counterparts, it works really well for this development model. When working with something that allows this much freedom, data standardization is needed very early on, however. I had to map out what was being stored and what would be stored from the beginning. I think that this one step has improved fnstraj development significantly because instead of making compromises with database layout, they’ve already been made and I don’t have to backtrack to fix things. I also standardized various interfaces to the database, how to handle errors, and so forth.

  • Project Organization: fnstraj has about 8 libraries to bring everything together right now. It’s been really fun learning how to break things down for a single project (without pulling all this stuff in from NPM, too). I’ve learned lots about how to separate things and when to keep things together. Being Javascript, with an annoying require-type workflow, I do have to pass program objects through these libraries for now. I hope to change that in the future.