Schemamigration

As progress is made through any project, there comes a time when the creators realize that the models that they had in mind for the final product aren't sufficient to complete the task. At this point, the models must be revised in order to accomplish the goals of the project.

Unfortunately, if the project that is being built involves a database, this adapting of the model can be a difficult process. In our case, we decided to add in a schedule to every light, room, and house. The Django framework is under strict instructions to create tables, but not to update them. Then, when you choose to add something, the Django website will throw an error because it doesn't understand why there is an extra thing added into its design. If the story ends there, that means that every time we adapt our model we will have to re-create all of the test data that we have generated, adding in the schedule information as we build it new.

Fortunately, the story doesn't end there. The good folks at Aeracode have developed South, a data migration tool for Django projects. South takes care of the difficult task of adapting the database for you. All that one needs to do is set up an initial state of the database with the South tool and then when a change has been made, create and apply a data migration. For our purposes, it worked very smoothly and the tutorial was easy to follow. 

After installing South, there were just four simple commands to run:

./manage.py syncdb

./manage.py convert_to_south myapp

./manage.py schemamigration myapp --auto

./manage.py migrate myapp

The only minor hiccup in the migration was that the database did not know what value to fill in for the schedule for each of the lights. Since the lights are allowed to not be on a schedule, a quick null=true fixed this problem and our app was fully functional again.

Let Me Just Write That Into My Schedule ... Not

I knew when I was assigned the task to develop a schedule for our project it wouldn't be easy. I just didn't have any idea how not easy it would be. I'll review a few of the decisions that we made regarding how schedules will be implemented in our project.

1. Lights, Rooms, and Houses will have schedules. This is opposed to a system where a schedule would be in charge of managing all of the parts of a house. If a schedule were to be in charge of managing the house, it would simplify things because all of the schedule information would be located in one place. On the other hand, Lights, Rooms, and Houses each having their own schedule feels like a better reflection of how the world works. I like to think that I have my own personal schedule that I choose to follow in going to classes and appointments, rather than my schedule ruling over me and dictating what I must do.

2. Triggered events are represented by a start time AND an end time. This decision was primarily made based upon how we plan to implement checking whether or not the light should be on a schedule. The plan is to examine the current time and the light's schedule, and if the current time indicates that the light is scheduled to be in a certain state, then the light will be switched to that state. Then if a user changes the state of the light from its scheduled state, a variable will be set indicating that the light is intended to be in the state that it is, rather than its scheduled state. Once the end time has been reached, this "override" switch can be reset to off so that the schedule will work as planned in the future.

In an ideal world, we would like the database to push information regarding whether the lights should be on or off to the client rather than having the client constantly fetch what state the lights are in on the database. For this reason, our decisions on how to implement the schedule may change in the future. 

I Speak the Language of URLs

While I was trying to accomplish my goal of allowing users to turn on a light, I ran into a problem. I wanted to direct users to a new page when I clicked on a link. I put in the URL that I wanted them to be directed to and created a page to correspond to that URL. Still being confused with the "reverse lookup" patterns that Django offers, I wasn't surprised that I received an error when I tried to click on the link. I probably worked on this problem for an hour straight before I was able to discover the problem.

Django offers regular expression matching for URLs so that a new URL doesn't need to be written for every page on a site. One of these special characters in a regular expression is a '$'. I had seen this dollar sign before, and knew that I could put it on the end of URLs. In fact, most of the URLs I wrote in the Django tutorial had this character hanging on the end. I naturally included this in my URLs that were being used to navigate to the new page.

The key that I was missing out on was that the '$' character represents the end of the line. This means that the '$' character should only exist in a URL regular expression if there is no possibility for more URL to come after. I had foolishly put one of these dollar signs at the end of one of my URLs which could still have more content come after it. When Django tried to search for the URL I was expecting, it would reach the '$' and fail to find the page I was trying to access. After removing the dollar sign, everything worked great.

So the lesson to be learned is: "Pay careful attention to all of the components of an error statement and make sure that you know what all of them mean." The dollar sign was staring me in the face the entire time that I was working on the problem, and I assumed because I didn't know what it was, it wasn't important. Bad decision. On a related note, the '^' symbol represents the start of a string, so watch out for that guy too.

The Registration Page

Wilbur's success story was implementing a view that would allow a user to create an account on our website. Like a large majority of the things that we do with our website, the way to implement this came from a very helpful post on Stack Overflow.

Unfortunately, while this provided all the details for how to log the user in behind the scenes, it doesn't provide the method of allowing the user to input their data. Thankfully, this webpage provided a template that can be used for logging a user in. Using this template with the code from the previous post should work right?

Wrong! It was at this point that I was introduced to one of Django's security features, csrf tokens. Cross Site Request Forgery Tokens are used to prevent malicious websites from accessing our websites data. Sounds like a good idea to me, as long as I don't have to spend valuable hours of sleep figuring out how to make it work with our software engineering project. The fix was simple enough, I just had to add one line of code:

{% extends "base.html" %}
{% block title %}Create an account{% endblock %} 
{% block content %} 
<h1>Create an account</h1> 
<form action="" method="post"> 
    {% csrf_token %} <--------------This line needs to be added.
    {{ form.as_p }} 
    <input type="submit" value="Create the account"> 
</form> 
{% endblock %}

Now I don't know if my site will be more protected because I added this one line of code, but I do know that it lets me experience success logging in, and that's what's most important to me right now!

Logging in Unit Tests

This past week I've been working on implementing Amber's success story into 'unit test format.'  We already had a test making sure that a user is added to the database, but we thought that actually logging in was something different.  I had little to no experience with unit tests, and the environment so my first few attempts were spent figuring out how things worked.

Eventually I came across a hidden section in the django tutorial using Clients.  This is perfect for what I was looking for, as I can use the client to 'post' their username and password to the login page.  I created two tests, one for a successful login (correct username and correct password), and one for an unsuccessful login (correct username, incorrect password).  The way I was checking to make sure it was working (for the successful login) is listed below:

1. Create test user (Username = Amber, Password = password)
2. Create client
3. Client posts <Username = Amber, Password = password> at '/login/'
4. Save the response code from the post.
5. Check to make sure that the HTTP response code was equal to 200 (the OK response code).

I did the same process with the incorrect login information, and checked to see if the response code WASN'T equal to 200.  The successful login worked, and the unsuccessful login test failed.

After I spent more time messing around with the tests, I discovered the problem.  After a login attempt, successful or otherwise, the system redirects the user towards the correct page (profile if logged in, and login page if not).  Well the system does a GET request to grab that correct page, and the HTTP request always sends back a 200.  That means no matter what, both tests were always returning a 200.  The HTTP request I was saving was bypassing the POST request and going straight to the GET request.  I am not sure why this is, but it is obviously a problem.  

My next possible solution is to have the client login, then check what view the user is currently on.  For the successful login I will assert that [location] (which is saved along with the HTTP code) is equal to /accounts/profile.  While with the unsuccessful login I will assert that [location] is equal to /login/.

I haven't gotten time to attempt to implement this new strategy, as we were unable to pull our project from our SVN repository on Dijkstra.  I will edit this post once I have implemented it.

Standing Meeting 4: Breaking Out

Let's start the review of our standing meeting how we always do, what is our progress from last week?

• Create a subversion repository for our project.
• Implement tests to simulate Amber's success story.
• Develop a mock-up of what we expect the user interface might look like.
• Implement Wilbur's success story.
• Make a preliminary structure of the models that we plan to use in our project.

The past week was very successful, as evident from the number of items crossed off our list. I think that we were even more successful than the list indicates however, because we figured out how we can work more effectively as a team. 

Creating a subversion repository for our project turned out to be an easy goal for this week because we were required to do it in class just after we set the goal for ourselves in our standing meeting. After a little confusion about what the project would be named and who would upload it, we managed to have a version stored that everyone was able to download and simulate the success stories that we had already put into place.

Amber's success story made some initial steps toward completion. We have some tests that create a user and check that they are in the database, but we need to do a little bit more research into how to simulate the user navigating to the website and signing in. Zach will be in charge of completing this goal for next week. 

We developed a mock-up of what the user interface might look like, as well as making a rough UML-esque diagram of the classes and method we expect to use in our project. I say UML-esque because I am fairly certain that we didn't follow all of the conventions of the UML. Despite this, I think it was a very useful exercise. In fact, I think that these two activities were probably the most significant achievements of this week. By outlining how we expect the program might be organized, we have a more unified idea of what the project will look like. 

We also implemented Wilbur's success story. You can expect a separate blog post describing that in the near future.

Before our standing meeting this week, we had another group meeting that was very significant. As we have been working on the project, we have just had one major success story that we tried to implement and we had everyone in the group try to work towards that goal independently. We realized that this doesn't work very well because there isn't a lot of motivation to work on the project. Each group member can independently hope that the other group members will accomplish something. From this point forward, we've decided to give each group member a specific task to accomplish so that everyone has an individual goal to work towards. Even though we have these individual goals, we also determined that it will be easier to achieve them if we gather as a project team to work on them. We plan on having regular meeting times scheduled from this point forward.

So with these points in mind, here are our goals for next week:

• Arik:
• Make the website look pretty. All that we have so far is black text displayed on a white background. Arik plans on looking into how Twitter Bootstrap can be integrated into Django.
• Zach:
• Complete tests to simulate Amber's success story. This will be accomplished by checking to make sure the views that are returned are the correct views for a logged in user.
• Karl: 
• Re-organize code so that authentication files are contained in one location. Implement the Light model to the project.
• Entire Group:
• Write tests for the basic model structure that will be implemented for this week.
• Further solidify the UML diagram for the project.

Here's a sneak peak at the UML we started to develop.

Unit Test Exceptions in Python

One of the goals for the next meeting was to develop tests for logging a user in to the system. This is something that is to me personally as a software developer, so I was grateful that the Django tutorial covered how to run tests in Python/Django. I did run into some difficulty though, when it came to trying to retrieve a user that was not in the database. Trying to do this would raise an exception during execution of the program.The Django framework has a method for expected exceptions called assertRaises(), but when I tried to use this, I kept getting an exception during run-time.

I thought to myself, "What's the point of this silly method? It's supposed to detect when an exception is raised, but it doesn't catch the exception before it is thrown." Here is how I was calling the assertRaises() method.

self.assertRaises(User.DoesNotExist, User.objects.get_by_natural_key('fyodor'))

Can you guess why I was still getting an exception? I'll give you a hint, it has something to do with how function argument are passed in the stack frame. Still stumped? I'll show you the solution, and maybe that will give you an idea.

with self.assertRaises(User.DoesNotExist):
    User.objects.get_by_natural_key('fyodor') 

The exception was still being raised because when function had arguments, they are placed on the stack before the function is called. This means that User.objects.get_by_natural_key() was being called before the program called the assertRaises() function was called. The program had no idea that the exception was expected at that point, so the test kept on failing.

At least, that's how I made sense of the scenario.

Amber's Success

After a long period of struggling, Amber has finally experienced success with her user story! Amber was able to access the website and log in using her name and a password. Amber is slightly confused by the website though, all that shows up is some small 12pt font in the upper left corner that lets her know that she has accessed the website. She then has to add "/login" to the address bar to bring up the log in screen so that she can enter her credentials. After logging in, a familiar mostly blank screen appears with some different text this time that says "You are now logged in!" Even more confusing, in order to log out, she has to navigate to a "/logout" page. Not the most useful website in the world, but she feels secure knowing that she has an account with this website.

Finding out how to get a standard user to log in with Django's framework proved to be a fairly complex process. After wading through seemingly endless looping links in the Django documentation, I finally stumbled on this page that seemed to be just what I needed. I found that looking at the "login_required" section was particularly helpful. Adding this redirected me to a predefined URL which should host the log in page. Unfortunately, there was an error with displaying this log in page. I was probably missing some obvious piece of code, but it seemed to me that the django directory was missing some template files. By searching in the directories for this log in template, I was able to come across another log in template for the admin half of the site. I pasted this template into the directory that the log in page was looking for the template and I had an instant (though incorrectly labelled) log in page. Using my knowledge from the Django tutorial, I put together the simple text pages that greet the user to the website and notify them that they have logged in.

I hope to come up with a more elegant solution to this log in problem in the future, but for the moment, it feels so good to have completed this first step.

Django Woes: "You mean I don't have to write code?"

While I was going through the tutorial for Django, I have had some fairly emotional experiences (considering that all I am doing is writing computer code). The first one was when I was accessing the admin portion of the site that I had just effortlessly created. I then recreated the view of the polls so that instead of just listing the text of each of the entries in the database, it also displayed the date that they were published and whether or not they were published recently. I then edited the code so that Django was aware that the was_published_recently() method returned a boolean value. After doing this, little green check marks appeared where True was earlier displayed. This simple aesthetic change was so unexpected and pleasant, I may or may not have raised my fist in the air and screamed a little bit.

The second emotional experience I had was in tutorial 4. We wrote some pretty generic views in tutorial 3, and then we replaced a good chunk of the code in tutorial 4 with generic code that took up a lot less space. I should have been happy that there was a much easier way to do the code that I had just written, but instead I was angry that I had to erase the code that I had so carefully crafted earlier. This is something that I should learn to overcome as an extreme programmer though, because there might often be times when it will be better for me to get rid of code and write new code than stick with older code.

A final note on the Django tutorial. Since it went through the process of logging in with an admin account, I thought it would also go through the process of how you could implement user accounts. Unfortunately it didn't, leaving our project in a precarious place. There seems to already be a useful built in user account system, but I worry that we won't be able to implement it correctly and cause problems later in the development process.