Rails Rumble was an amazing experience. I think I slept about 6 hours over a 56 hour period and while I became grouchy I don’t think I ever stopped having fun. The fun stopped after the first day of voting. Starting during the second day of voting I <3 Games plummeted in the rankings. I am not really sure what happened to cause this fall. I am not the most objective person but in testing some of these other apps I’ve run into some real problems that I don’t understand how something that doesn’t quite work could be ranked higher than our app which I think is about 90% done.
Problems with voting
- With any voting system there are always ways to game the system. The OpenID requirement for Rails Rumble voting would make gaming rather easy I would think. Without a captcha or email validation it would be fairly easy for a machine to sign up for voting as many times as it wanted.
- Voting with most people for Rails Rumble is only based off of a glance, there is very little actual signing up to use the app going on.
- No way to explain what your feelings about the app were.
I think if some of these issues were addressed I would feel a little better about the voting system. I think the 8 random apps a day idea was a good one that helps prevent the event from becoming an popularity contest.
Gaming the system is always going to be a problem. Bots can be created that get around almost every system to stop them, but when the systems are few and far between to stop them they can have a field day and can dramatically skewer results. I do know a few people who aren’t bots that just said “Oh, well I’ll just give every other app 1 star across the board” and it shows. Rather than 3 stars being the “average” value for an app it is the high value. I know there won’t ever be a 5.0 over all app, but we’re talking about 2.9 being the high value and 1.6 being the low value. That is a lot of 1 and 2 stars over the 4 or 5 stars. On a scale of 1-5 then 3 should be the average, that should be “this app is unremarkable, there is nothing wrong, but nothing right either” instead if we keep going along the way we are currently going then 2.9 is going to be the winning score, which using the normal averages would mean that it is slightly below average, and while I’ll be the first to admit none of these apps are above average against an application where you have an unlimited time frame and tons of QA testers. If you keep in mind that every single one of the apps was designed in 48 hours by 1-4 people I would have to say a great many of them are above average.
One of the problems I have with the voting system is that the results tend to be skewed towards pretty on the front page apps. Powered By Geek didn’t exactly cater to this, we hid a lot of functionality after sign up, but we did put a little video up explaining about the app. The real problem I have is the number of people who aren’t even signing up who are voting on the app. I know for a fact that our login/signup process works because we had a guy dedicated to testing that procedure during the actual competition and I’ve created about 10 accounts as well with my numerous Open IDs and as regular users. Looking at our numbers we’ve had 79 users sign up, and when you take into account all my test accounts that is about 70 users total who have signed up and yet we have 240 votes on our site. That is a 1:4 ratio of people who actually looked at our site vs people who just voted either after looking at our home page or without looking at the site at all.
I’m wary to point my finger at people and point out flaws as to why they shouldn’t be at their position, however some of the apps I tested had main functionality broken and yet they are sitting in the top 30 spots. Testing is almost nonexistent, a friend was testing other sites and uploaded a picture that he figured would disappear after a while but instead it was on there for over 24 hours. The site only had slots to show 5 pictures that means that it took over a day to upload 5 pictures on the site.
My final real problem is that there is no way to tell why people voted one way or another. I think a short description on why they are voting one way or another would help everyone involved to understand what people liked and what people disliked about the app, not to mention it would make it easier to tell when someone was gaming the system.
Overall I think the Rails Rumble brings an awesome service to the community. 48 hours with the right motivation (competition) is really useful as it can really get an interesting idea or product built. Not to mention it helps bring you together with friends towards a common goal. I really just wish everyone could come away from it feeling like the voting wasn’t just a game.