Data distribution ideas

[thelusiv]

Our full data package tarball has grown to a whopping 220 MB. The problem with this is that no matter how we split up our data package from our game package, we will always either have a huge amount of data for people to get, or people who don't have all the data.

We need to find a solution for a lot of reasons. First off, people don't try out the minimal package first to make sure it works. Everybody wants the "best one" and doesn't want to have to come back and re-download it after they try it out the dinky minimal version. Then if it doesn't work...that means we're wasting their time and bandwidth, and our hosts' bandwidth too. Bandwidth costs money so that's bad and generally people don't like to download a huge big thing for absolutely nothing.

Here's how I propose we fix this problem. Let's add to the game a simple data management system that can download a list of available cars and tracks, and then let the user choose which ones to download. Now of course it's easy to say that...but there are a thousand different ways that can be done.

So here's how I propose we do it. Let's keep all our cars and tracks in the project's Subversion repository provided by SourceForge. We're not using it for anything else, and that would prevent it from putting the burden of every user's download on our current SVN hosting (which is typically much faster than SF.net).

Now to get them in the game all we need to do is use the Subversion C library (for more info on using Subversion this way see this introduction to the Subversion API). Now to satisfy Joe who is usually against adding more libraries I have to justify it...so let's compare using SVN for this to doing it manually. If we do it manually we have to package each car and track, at least as often as every release. Now we could automate this with SCons or something probably, but that means more support code to write. Then we will still have to use some kind of library anyway to unpack the data packages once the user gets them.

On the other hand, if we use the SVN library, we can set up one repository where users download individual directories. Getting a list of cars and tracks is just a matter of getting the directory list of the repository. A small preview graphic and description of each car/track could be also downloaded automatically to give the user an idea of what they're getting. Since 'svn up' is all that's needed to update the user's checkout, the cars can be easily updated as we check in fixes.

Now, the next good thing about this, is that if we move the game data to a new repository, there is really no reason to keep it in the main code repository. In fact Subversion has a handy function that allows it to automatically fill a subdirectory from another repository. I can even think of a way to make it so that checking out the code doesn't mean checking out all the data. This would significantly reduce the size of the code repository, the time it takes to check out the game, and so many other problems we've encountered due to massive amounts of game data.

The real goal though is to get VDrift back down to one download for release, not a confusing system of data packages and all that. This will help out the users as well as the developers.

This post is getting really long and I'm quite tired, I just wanted to throw the idea out there. I'm interested to hear any other good ideas for distributing data, or any modifications to my idea that might make it better, criticisms, or whatever.

[FFuser]

Well this seems a very good idea.

I thought before to change the windows installer so that you can check/uncheck cars and tracks, and only the ones selected should be downloaded, but this has indeed the disadvantage that it can only be done on install (or reinstall) and not in game. And that (as you mentioned) it should be done manually for every package (windows, Linux, mac, freebsd)

Your method seems better but I forsee one problem:
In the svn version in the car/track definitions something is added (like in this release startplaces). The user updates his svn checkout of the tracks => vdrift cant load it because there is an unknown line in the track definition.

[thelusiv]

Good point. I think maybe we could get past this by using Subversion's branching feature, so that we could branch the data for new features, without breaking cars for old versions.

Actually though as long as nothing is removed from the car/track files, old versions of VDrift will probably work fine with newer data. Because of the nature of our .config files, the new stuff will just be ignored.

However it is possible that we'll have to discontinue support for some old versions at some point if we radically change the format of cars or tracks. It's also possible that we could just set up something like this:

version 2007-03-23 - use data up to r400
version 2007-04-xx - use data up to r450
version 2007-05-xx - use data up to r500

Thus instead of getting HEAD (hehehe :lol the older version of the game won't update past a certain point, and the data can keep working for those versions.

[TerraRoot]

seems very complicated, i do want a smaller download, is the svn option easier than just hosting the file's up and letting the user decide what cars he/she wants to play with? (i've never used svn)

[thelusiv]

Well I don't mean that the users would use SVN themselves. The game would manage the checking out, updating, and all that for the user. All you would have to do is open VDrift, go into the data manager, select the new cars and tracks to download, and that's it - everything else is taken care of for you.

[joevenzon]

It's a cool idea, and it seems like it'd work, but it'd be veeery complicated to implement correctly. Does the svn update in the C library give you any sort of downloading status? The command-line svn tool doesn't, and when downloading a large file, it might be frustrating to not have a percentage bar....

Also, it seems to me like there would still be a need for a "full" package, so that people without internet connections or with slow connections could download all of the data on one computer, burn it to a CD, and take it home. Or, at least, download it for the entire day while doing something else instead of having to have VDrift open for the entire time it's downloading.

Thinking more about this, maybe there's not really much advantage to having this feature in-game. What about just having a minimal package plus a really nice web interface that allows people to browse and download select cars/tracks?

[thelusiv]

It's a cool idea, and it seems like it'd work, but it'd be veeery complicated to implement correctly. Does the svn update in the C library give you any sort of downloading status? The command-line svn tool doesn't, and when downloading a large file, it might be frustrating to not have a percentage bar....

I'm not sure if this is provided, but I agree it's needed, some tracks are pretty big. I'd have to check out the API more.

Also, it seems to me like there would still be a need for a "full" package, so that people without internet connections or with slow connections could download all of the data on one computer, burn it to a CD, and take it home. Or, at least, download it for the entire day while doing something else instead of having to have VDrift open for the entire time it's downloading.

Good point, but I think we could also provide instructions on the wiki telling people how to check out the entire data set all at once, and then explain how to move it. Otherwise we're always going to end up with people who just automatically download the 200+ MB file before they test the game.

Thinking more about this, maybe there's not really much advantage to having this feature in-game. What about just having a minimal package plus a really nice web interface that allows people to browse and download select cars/tracks?

Good point, it doesn't necessarily have to be in-game, and that's a lot of time that the user has to wait while SDL hogs 100% CPU and all that. To have a nice web interface, this requires some kind of web site (someone has to write, maintain & manage it), which needs hosting (SF.net would be ideal but it's pretty slow especially for casual browsing), and will probably run into security issues.

So what if, instead of a web app, we provide the user with a simple PyGTK app that uses the Python bindings for the Subversion library (they are way easier to use than the C library after all) to get cars and tracks. That would remove the complexity from the game, make it super easy to write and port, and allow the user to download their cars and tracks in the background...plus, it would prevent them from getting more data than they can use if the data manager comes with the game. Also if they do find it works, then they could choose to download everything without having to have the game running in the background, and burn to a CD or whatever they like.

Oh and one more thing, which I didn't think about before...using GTK would avoid me having to write a bunch of new GUI code to support a data manager in-game, so that works out well too.

[joevenzon]

So what if, instead of a web app, we provide the user with a simple PyGTK app that uses the Python bindings for the Subversion library (they are way easier to use than the C library after all) to get cars and tracks.

I like this idea a lot.

Also, while we're just brainstorming, it'd be cool to later on add some community-based features into it, like ratings or reviews or uploading mods.

[thelusiv]

Yeah I think a standalone app does make more sense than my original idea. I also like your idea of ratings/reviews.

On the topic of mods...we need to work on our car parts system, and get it working again. It would make some sense to manage parts/mods through the data management app we're talking about. That is, it should at least download the parts available for each car (by the way, I think we should nuke the old data/carparts/ way of storing car parts, and make a parts/ directory under each directory in data/cars/ instead).

Other "long term" kind of ideas to think about the bigger picture here...If we're already managing cars & tracks in this app, the next logical step is to make it so that you can edit cars & tracks. Earlier on I had the idea to write a PyGTK track editor as well, why not go ahead and combine these apps into one - or at least two that can work together. Likewise, it would be very helpful to have a car editor, which could also be used to create car parts.

So the best way to tackle this is probably to split it up and set up goals. First we need to start out with the base data management features using the Subversion API, and set up the new data repository. I think this is a reasonable goal for the next release. After that we can work on improving the data manager. First milestone feature set and a few use cases:

* Get a list of directories in the data repository under tracks and cars
* Provide the user with a list of available cars and tracks
* If the user selects a data item, preview it (Just description is OK for now)
* Under the preview pane provide button for Download (which will be greyed out if it's already downloaded)
* Another button here should be the Delete function - users should be able to remove data they don't use or like to free up disk space
* The program should be able to read/write VDrift's .config format - we need to write a .config Python module that's easy to use (this should not be hard)
* I'd like to avoid this program having its own configuration file. It would be nice if it just "discovers" everything whenever it starts up.

Basic data to display about cars:
* Name of car
* Author
* Max HP
* Weight
* Short description

Right now there is an about.txt file with each car (more or less). I'd like to either move this all into the .car file or make a separate .config file to store this and other information. Keeping the file separate is probably the best idea, so that the whole .car file doesn't have to be downloaded just to get preview info about the car.

Tracks will be similar but different kinds of info:
* Name of track
* Author
* Length
* Location
* Short description
* Perhaps number of turns...?

That's all the ideas I can come up with for tonight...

[joevenzon]

Sounds good! Those seem like realistic goals.

[yochenhsieh]

Hi, I saw you have lots of interesting ideas already... Though I cannot get VDrift working right now, I have a simpler proposal about distributing.

1) The minimum release remains the same, unchanged.
2) Host another full data package, which includes all other cars and tracks not included in minimum release.

This is easier for upcoming releases, as (I think) it almost does not involve any code change. In the mean time, the data management application can be considered as a long-term goal.

[joevenzon]

How would 1 and 2 be any different than what we already do?

[yochenhsieh]

The "full data package" will not include the material which already in the minimal release. It will be only an "extra data pack", and cannot be run as a stand-alone game.

[thelusiv]

Well, we could have already packaged the game like that, but the game itself is not that big so it doesn't add that much to the full data set, so we might as well include it. Plus, people will still just download both files, before trying the game. This also doesn't address the problem that some people may not even want the full data set, just more than minimal. For instance they may want to download one or two extra tracks, but they may not want the really big ones, as those won't work on every computer.

I don't think that it will be that difficult to write a simple data manager application. I'll start working on one as soon as I have time. Joe, do you think you could write a python module to read our .config files?

[joevenzon]

It'd be pretty far down on my list. I don't remember enough about python to pump it out very quickly.