Data distribution ideas

[thelusiv]

Our full data package tarball has grown to a whopping 220 MB. The problem with this is that no matter how we split up our data package from our game package, we will always either have a huge amount of data for people to get, or people who don't have all the data.

We need to find a solution for a lot of reasons. First off, people don't try out the minimal package first to make sure it works. Everybody wants the "best one" and doesn't want to have to come back and re-download it after they try it out the dinky minimal version. Then if it doesn't work...that means we're wasting their time and bandwidth, and our hosts' bandwidth too. Bandwidth costs money so that's bad and generally people don't like to download a huge big thing for absolutely nothing.

Here's how I propose we fix this problem. Let's add to the game a simple data management system that can download a list of available cars and tracks, and then let the user choose which ones to download. Now of course it's easy to say that...but there are a thousand different ways that can be done.

So here's how I propose we do it. Let's keep all our cars and tracks in the project's Subversion repository provided by SourceForge. We're not using it for anything else, and that would prevent it from putting the burden of every user's download on our current SVN hosting (which is typically much faster than SF.net).

Now to get them in the game all we need to do is use the Subversion C library (for more info on using Subversion this way see this introduction to the Subversion API). Now to satisfy Joe who is usually against adding more libraries I have to justify it...so let's compare using SVN for this to doing it manually. If we do it manually we have to package each car and track, at least as often as every release. Now we could automate this with SCons or something probably, but that means more support code to write. Then we will still have to use some kind of library anyway to unpack the data packages once the user gets them.

On the other hand, if we use the SVN library, we can set up one repository where users download individual directories. Getting a list of cars and tracks is just a matter of getting the directory list of the repository. A small preview graphic and description of each car/track could be also downloaded automatically to give the user an idea of what they're getting. Since 'svn up' is all that's needed to update the user's checkout, the cars can be easily updated as we check in fixes.

Now, the next good thing about this, is that if we move the game data to a new repository, there is really no reason to keep it in the main code repository. In fact Subversion has a handy function that allows it to automatically fill a subdirectory from another repository. I can even think of a way to make it so that checking out the code doesn't mean checking out all the data. This would significantly reduce the size of the code repository, the time it takes to check out the game, and so many other problems we've encountered due to massive amounts of game data.

The real goal though is to get VDrift back down to one download for release, not a confusing system of data packages and all that. This will help out the users as well as the developers.

This post is getting really long and I'm quite tired, I just wanted to throw the idea out there. I'm interested to hear any other good ideas for distributing data, or any modifications to my idea that might make it better, criticisms, or whatever.