Project MathQuiz: Scraping the Bits Together

Project MathQuiz: Scraping the Bits Together

My cousin Jason has been plugging away at putting the pieces of a web app math quiz together. We’re staying focused on just solving the basic nuts and bolts problems first: how do you make an interactive web page that talks to a server in real time? How do you structure your code and how do you manage it? We pick up from Day 1.

Day 2

Building on Day 1’s progress, Jason wrote some Javascript that read an array of questions from a JSON-formatted string. These questions were stuffed into a Javascript array of “problem objects”, each containing the question text and the answer. He also wrote a function that was bound to a button for each question that compared the entered answer with the problem’s correct answer. This snippet of code would print “Correct!” or “Incorrect! depending on whether the entered answer matched the stored answer. All in all, it was a good workout of our Javascript/jQuery knowledge.

After this was working, we then stored a text file with the JSON-formatted questions on a web server, and learned how to use the jQuery .get() function to retrieve it. This lead to a discussion about asynchronous events and how to handle them. A good day!

Day 3

After getting the basic question loop and server-based text file to work, we had a single file that had all our code in it. Knowing we would soon have to expand our library of functions, this was a good time to refactor our code and clean it up. This meant tackling a few things:

  1. Splitting up the code into loadable javascript modules on the client side for ease of development, with separate functions in separate files.

  2. Finding a way to ensure that the javascript files loaded in the right order, which meant cleanly-defining the various stages of program startup.

  3. Figuring out how the hell Javascript object oriented programming worked using prototypes and other arcana.

  4. Figuring out how to take advantage of Javascript closures when writing event handlers that needed to access objects that ordinarily would be out-of-scope when they were called.

  5. Committing the refactored files to our centralized source control repo on Bitbucket, figuring out how SourceTree worked at the same time.


p>This took most of the day, and involved a bit of theorizing about the “right way” to start up a program. I have a crude understanding of it; it goes like this:

  • Allocate any global data storage and definitions
  • Construct all object instances, but don’t initialize them. They should just allocate storage.
  • Initialize all object instances, now that they can refer to each other. Load data, initialize major data structures, etc.
  • Initialize object managers and major system event bindings
  • Enter Run Mode – Render Page / Bind Controls

Constructing the startup chain above meant figuring out when and where we should call certain functions contained in our “init.js” file. We used the head.js library to handle our javascript app source loading asynchronously, and then upon completion fired off each event one after the other.

Rewriting our kludgy first-pass prototype code into Javascript objects meant we had to spend some time talking about object oriented design, like how to model the components of a Quiz and how to think about the entire project as defining an organization with objects managing other objects. We also were somewhat confounded by object prototypes and how they worked for a while, until we realized that it was a simple child-to-parent lookup chain that was handled automatically at runtime.

At the end of the day, we had a cleaner source code arrangement and had the new directory structure committed to source control, and we were both able to push and pull changeset to the central repo. A good day!

Day 4

Most of the coding of the day was spent thinking about how to handle the server component. We needed a database to contain the questions. Neither of us are very familiar with database programming or SQL, so we talked out what we needed: a way of requesting a group of questions. We momentarily were daunted by the prospect of designing and normalizing a bunch of database tables, until I remembered the relational databases are actually much simpler than the convoluted explanations about “joins” and so forth would have you belief. We could easily get away with just having a table of Problems with two fields, and another table of Students with other data. That would be just fine for our nuts and bolts approach of the moment.

Next, we had to figure out how to actually talk to the database. On the Javascript client side (the browser), we knew we could use jQuery to handle our httpRequests to a web server. Jason was interested in using Web Sockets, which allow a persistent connection suitable for realtime applications. We’d both been curious about using “node.js” to implement it, because we’d heard it was cool and ran on the server. We just had no idea exactly what it did, or how it worked.

We spent a good part of the day at a coffee shop just reading about it. We had some knowledge of how web servers actually work: a request is sent by the browser using the http protocol, which is simply a text command with the name of the resource (the URL) and any additional values. The web server receives the request and chews on the URL, deciding how to handle it. Once it decides how to handle the request, the resulting response is sent back to the requester. We wanted our server to be able to respond to our particular request with the question data.

I asked my other cousin, Ben, whether he’d done anything with Node.js, but he hadn’t had the direct experience of setting it up. Rather than install Node.js on our own servers and risking breaking them, we thought of looking into using cloud computing to cut down on costs. As it happened, it wasn’t that much cheaper than $30/month virtual hosting, but Ben did tell us about Bitnami, an “app store for server software” that provides pre-configured server downloads, virtual machine images, and web-based automated deployment to Amazon’s Elastic Compute Cloud. You only pay for using the automated deployment feature, so I downloaded a Node.js server virtual machine and set it up on my main windows workstation. Jason actually installed node.js on another virtual machine running the Mint distro and started going through the tutorials.

It took us a while to figure out how to think about Node.js. It essentially is a web server construction kit that is built on-top of Javascript running on the server. While this might sound super slow, it’s actually pretty fast because it uses Google’s V8 Javascript engine. It’s also designed from the ground up to handle asynchronous requests without blocking, so you can write very efficient real-time responders.

Because it’s a web server construction kit, you don’t install it as an Apache module. Instead, you have it listen to a number of ports for connections, waiting for httpd requests. Instead of making a configuration file, you write a little program that creates a server connection object and bind it to a port. The listener redirects everything it gets to the function handler that you provide, which can do just about anything you’d want. Because your writing programs that listen to ports, you can put your node.js programs anywhere on the server so long as you can start it up. This last bit confused me for a while, since I wasn’t sure where to put node.js files or whether I needed an Apache webserver to handle other files. I think I’ve come to a conclusion about how to set this up, though, for my development server.

Anyway, our biggest challenge of the night was just getting a test node.js program to respond to a request with data from a mysql database. We were following a tutorial online that ended up being incorrect; it was a good opportunity to go through debugging methodology to test our assumptions one after the other until we found the point of failure, then applied our own understanding of how the system should work to uncover the bug in the tutorial (an event that never fired), followed by a bug in the SQL statement being used to pull the data out of the database (unnecessary quotes around the table name). After that, everything worked!

Tomorrow, Jason is going to work solo on his stuff while I take care of other business. It’s pretty exciting to see our basic understanding improve day-by-day!