Make No Assumptions: Software Engineering is Not Software Engineering Unless You've Been Through IDPM

The full impact of software engineering has not truly been gauged until now. It's multiple belts up from the white-belt WattDepot katas we previously tackled. It's the combination of our basic skills with WattDepot, our fleeting glimpse of Continuous Integration (CI), and our whirlwind introduction to Issue Driven Project Management (IDPM). And the most lethal ingredient of them all is teamwork, as implied by CI and IDPM. Lethal, but unavoidable. When working with others on a project, CI makes things a bit easier by assuring project team members that the system they are developing is in working condition at all times. The status of verification of the build is automated under a CI server such as Jenkins, which triggers an automatic verify test on the system in the event that a team member has recently committed changes to a Subversion (SVN) repository such as one hosted by Google Project Hosting. CI also requires team members to commit regularly, such as at least once every two days. Meanwhile, IDPM is designed to keep all project team members on task at all times by associating each task with a specific issue at hand, and ideally, issues should not take longer than a few days to complete. Successful IDPM requires frequent group interaction, always having another issue at hand after completing one, and general knowledge of where the project is headed at all times. In particular, Google Project Hosting (along with SVN and Jenkins) was used to drive IDPM; there, it is easy to see what tasks are (mainly, among other statuses) Accepted, Started, and Done by opening all issues under Grid view.

The project, which can be found at this website, was to create a command line interface program to understand various aspects of energy use in the Hale Aloha Towers (freshman dorms) of the University of Hawaii at Manoa. My team in particular consists of two other members. As in the real world, we did not get to choose who to work with. We had to choose a name for our team and, after mulling over a brainstorm-generated list, settled for "Pichu" (not necessarily after the Pokemon of the same name). We exchanged phone numbers and Skype usernames and had the project broken down into the important issues at hand, all in the first day the project was assigned to us.

Mentioned above, the dynamic trio of Google Project Hosting, SVN, and Jenkins was used to drive IDPM. Before each commit I made (except for one or a few times in which case I was positive the program would pass verify - but alas, lesson learned: never assume), I made sure to run verify on the system. Then, after a commit, I would check the associated Jenkins job to make sure that it built the system successfully. To address IDPM, after the disastrous surprise in-progress technical review, I made sure to associate each and every commit (known as a revision in Google Project Hosting) with the issues I addressed. To take it a step further, when I updated the issues themselves, I added links to the revisions that handled these issues.

Our team originally had the issues all planned out from the beginning, but as we finished our main issues, if we thought up more issues, those were simply added to the existing pool of issues. Because of IDPM and overall constant communication, we were able to gauge what everyone else was doing and hence finish the project in time. The basic program functionality has six commands; a description of each can be found on Team Pichu's home page. There were four main components to the program, namely: the Main class, the Processor class, classes that implement the Command interface, and JUnit test classes. Main contains the driver for the program, establishes connection to the WattDepot server, and creates instances of the Processor class for each command or invalid input entered by the user. Processor takes the input, parses it, and determines which Command-type class to call based upon the input; if an error arises with the input, then Processor returns the appropriate error message to Main. Each Command-type class must implement the Command interface's run, getOutput, and getHelp methods; in addition, each Command-type class should have a default constructor (in addition to constructors with parameters, if any). If the Command-type class takes one or more parameters to the constructor, then that constructor should throw an IllegalArgumentsException, and a checkArgs method should also be implemented. Lastly, each Command-type class should have an associated JUnit test class (Main and Processor have test classes as well).

Unfortunately, we were not able to implement Java Reflection, so most of the code is hardcoded. For example, if a Command-type class is added to the Command package, then changes to Processor and Help need to be made, and a corresponding JUnit test class needs to be created in the Test package. Hence, decreasing the number of places that code needs to be updated any time a new command or feature is added to the program is a future improvement that could be made. Overall, this experience has been completely meaningful and worthwhile because of the real-world experience of working with others in real-time to produce a single smooth and functioning program. Communication, which could certainly have been better, was definitely the key element to success, as well as consistently working on the project over a long period of time. I felt that both communication and consistency were quite challenging when one is out of sync with the other. Without constant feedback through communication, it becomes more difficult to gauge what each team member should be working on. Another critical learning experience is what to do in the unexpected event of some greater force negatively impacting the project at hand. The WattDepot server had crashed and there was a hard disk failure for the first time in about four years. Luckily, our group was able to step up to the plate and align our project with the newfound situation at hand. In conclusion, I feel that IDPM is much more difficult than it had first seemed; moreover, the overall quality of our software could have been better and less hardcoded.

Energy Drained is Energy Gained

WattDepot: A one-stop shop for all your energy and power needs

Complementing the previous blog entry on energy in Hawaii, I had recently gained firsthand experience in manipulating the output of energy data, all from the comfort of my own home, using the WattDepot API. WattDepot is a web service that allows a host computer to connect as a client to an existing web server. This web server collects energy and power usage data stored in meters and saves the information in a database, which the WattDepot client can access. The meters I was concerned with are located in the Hale Aloha Towers (freshman dorms) on the University of Hawaii at Manoa campus. The sheer proximity of the meters only made this task of working with live energy and power data more relevant and meaningful.

Not to say that it wasn't stressful.

Enter the WattDepot katas:

Kata 1: SourceListing

The task was to print out the names of each source of data (where the meters were installed) across from their respective descriptions. This kata was absolutely deceiving, only because it was so straightforward, given the wattdepot-simpleapp to start us off with. A simple copy-and-paste from the simple application, with minor tweaks to the format of the output, made this kata breeze by in about 5 minutes.

Kata 2: SourceLatency

This seemed fairly straightforward: print out the sources and their latency values (in seconds). Another simple copy-and-paste from wattdepot-simpleapp got me off to a quick start. Wonderful, it printed out the latency values and the sources just fine. But not-so-wonderful, it didn't print them out by ascending order of latency value. I ended up taking the longest time, around 4 or 5 hours (not including the breaks in between), on this kata because of time spent searching for a suitable Java Collections class to use that would allow duplicate keys to map to different values. I finally figured out that Java doesn't have any built-in libraries that hold for this simple concept. While Googling, I discovered the Guava libraries, created and maintained by Google employees, who use it as their core libraries for Java-based projects. It took less than a minute to download the jar file and import it into my WattDepot project in Eclipse. Yes, now I could work with the Multimap interface, which allows duplicate keys to map to more than one value.

Unfortunately, printing out the keys and values using a Multimap in the format required by this kata was not a task I wished to partake in, as it seemed to be too much trouble than it was worth; I felt the same way about making my own Collections class. In the end, it was the first proper Collections package that I wrote, CompactDiscCollection, that gave me inspiration to use a TreeSet data structure of String objects. The String value was a concatenation of latency and the source name. Yes, I would say this was hardcoding, as it relied on the fact that my latency values were 2 digits long, but it was quite the shortcut, alleviating much stress.

Kata 3: SourceHierarchy

The task was to output a text-based visual representation of the hierarchy of sources and their subsources. I probably spent the second-to-least amount of time on this kata, for about 1 to 2 hours. Browsing through the relevant topic on the class Google Group definitely helped. I, like others before me, wasn't sure whether or not to include certain sources as top-level sources, since some of them were already listed as subsources for other sources. Ultimately, I didn't list them to avoid redundancies in the output. The time-consuming (but not difficult) part of this kata was parsing all the information returned by certain WattDepot methods. But learning a little more about java.util.regex.Pattern was quite rewarding.

Kata 4: EnergyYesterday

Here, the task was to retrieve all of yesterday's energy consumption data for each source. I believe this kata took me longer to complete than the previous one. Since it took such a while, I decided to leave in the code I originally wrote to acquire yesterday's year, month, and date (commented out, of course). Apparently, to my surprise, I didn't read the Calendar API closely enough because there already exists two methods that can acquire yesterday's date. Setting the start and end timestamps to 00:00:00.000 AM and 11:59:59.999 PM was a simple manipulation of Calendar objects, but getting to know how these objects mix with XMLGregorianCalendar objects and how timestamps actually work was quite a bit to tackle all at once. What frustrated me most was trying to figure out which of the 5 existing Tstamp.makeTimestamp() methods from the WattDepot API to use.

Kata 5: HighestRecordedPowerYesterday

This kata was quite a change from the previous one, as we were tasked to calculate the highest power (not energy this time) for each source recorded the day before. The class Google Group and kata specification helped me determine that 15-minute time intervals for a total of 96 queries from the database would provide more accurate data. But there were 64 sources, so for each source to query the database 96 times, it took quite a while to produce the output, which was not at all fun and friendly. I had to move to a place with faster Internet speed and wait for a few minutes before acquiring decent output (yet not without red error messages from a catch clause due to miscellaneous errors). I believe I took about 2 hours on this. This kata was definitely more trouble than fun.

Kata 6: MondayAverageEnergy

The last task was to compute and print the average energy consumption during the two previous Mondays for each source. It was simple enough to copy over the contents from Kata 4 and begin computing the offset value required for one Monday ago, relative to the current date. After that minor calculation, adding another 7 days to the offset did the trick for two Mondays ago. This probably took about an hour to complete. Like Kata 5, I had more trouble with acquiring the data for this, perhaps due to the fact that I was querying data from over a week ago.

Conclusion

To be honest, the title of this blog post is the first thing that popped to mind. Yay, something catchy. Yay, it rhymes. But after getting this far in the post, I realized that the title is not without purpose, for indeed, energy drained (into these katas) is energy gained (I feel much, much more knowledgeable about Java APIs and programming lessons learned in general). I learned that the folks at Google are more awesome than they already are because they've recognized the deficiencies in the Java Collections API and have shared with the world their extensions to it. Despite this fact, I have also learned that no matter how many different types of collections are created by others in the world of open source, you will usually find that a certain promising-looking collection just doesn't meet your needs exactly as you wish. I have also practiced the art of lazy programming by taking advantage of String victims and Java TreeSet instead of going through the trouble of making a collection class to suit my needs. Hardcoding was my friend during this time of stress, as I was pressed beyond reasonable allowable time as it was. Regardless of it all, I have never been better off. Energy data manipulation is just plain cool. And something quite useful for the future of tomorrow.

Small Steps for Clean Energy in Hawaii

Growing up in Hawaii, my parents had always warn me to stop playing video games or turn off electric devices because the electricity cost was so high. But it wasn't until today, thanks to these screencasts, that I discovered just how high that cost was: roughly three times as much as the mainland's, at $0.30 per kilowatt-hour (kWh). Leaving these devices on for most of the day is certainly not such a smart idea, but it's quite difficult to break out of the habit when one is a computer science major. But I realized if I want to play a part in the Hawaii Clean Energy Initiative, now is a better time to start than never.

The Hawaii Clean Energy Initiative was signed off by former Governor Linda Lingle in January of 2008 as a partnership between the state of Hawaii and the U.S. Department of Energy. It requires full participation and support from all Hawaii state residents if, together, we are to achieve 70% clean energy by the year 2030. 70% clean energy means that 40% of the projected energy in 2030 is generated from renewable resources and 30% of the projected energy is reduced by means of conservation and other efficiency measures.

Yet 40% renewable energy is just the start. What makes Hawaii an even better place to live apart from its unique geography, climate, and mix of people is that, unlike any other state, it has the potential to generate all of its power from renewable energy resources and has almost every source of renewable energy available. This includes wind, wave, solar, geothermal, and more. Renewable resources are generally more cost-effective for Hawaii, and because of its small size (in comparison to other states), Hawaii's energy needs are modest enough that all-renewable energy resources is possible.

Negative consequences loom ahead if we do not pick up our act now. Hawaii still generates most of its energy from imported oil, which is extremely expensive, as we know it, and the price is only likely to get even higher as the years go by. Furthermore, Hawaii as it is now uses inefficient means to generate its energy: each island has its own source or sources of energy so that not one is connected to another. These unconnected energy grids aren't what we'd call teamwork. If we don't do something about this, not just the price of gas and electricity would go up, but the prices of water, food, and clothing, etc., would all go up as well.

In conclusion, if Hawaii is to remain the dear place we call home, we must take it upon ourselves to conserve energy, as much as we can, on a daily basis. Nothing big was ever achieved in a giant leap. It's the small steps that we take each day that truly count in the end.