Re-homing Maps in Git

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Re-homing Maps in Git

aaalafayette
Administrator
This post was updated on .

 Latest:


If you would like to be an early adopter of maps in github, post a link to your map zip file to this thread. I'll then run a script to zap it into github. Send me your github name as well and you'll get hooked up as an admin of that zip (assuming you actually do own it and all that good stuff).

Github ticket seeking developer/admin consensus this is the right way forward: http://github.com/triplea-game/triplea/issues/242. Once there is agreement we'll update the game engine to use these map zips, i'll be inclined to get every map hosted and then verified. Then it'll be on to figuring out how to update the game engine (short term the XML file hosted in SVN likely could just get some new URLs)


Git org and repositories:  http://github.com/triplea-maps
Project page:  http://github.com/triplea-maps/Project
Chat room: http://gitter.im/triplea-maps/Project

The project page has links to:
- Map Maker setup Tutorial
- New Map Submission and Update Instructions
- Admin Maintenance Docs and Travis map builds link


 Original Post


Moving maps from SVN (sourceforge hosted) to Git (GitHub hosted) to follow the game engine code would have quite a few advantages. Easier collaboration I think being the biggest. In this thread I'd like to discuss a potential move of maps to GitHub hosting, why we would do it, and see what questions/concerns are out there.

The Process Today

There are a number of manual steps. The maps live here as zips: https://sourceforge.net/projects/tripleamaps/files/maps/

First the maps being stored as zip files is not very good for source control. Typically with source control files are stored as text and then a packaging process, or a "build" is used to package up the files. On the other hand, without that automated build process, having a straight zip file in SVN is handy. Thus there are perhaps reason for having it this way.

Let's trace what happens if someone wants to modify a file in a map:
- First you need to contact the SVN repo owner and or map owner so you can know where to send the update, this may not always be obvious
- Download the map zip
- Expand it
- Make modifications
- Re-package the map zip (this could be quite an iffy step if you did not package it in the first place)
- Send email to repo owner with zip file attached
- Owner gets email with zip, opens it, tests things out, uploads it if they like it

Some Downsides to the Existing Process


First there is quite a lot of manual work, and not much feedback during the workflow. It's hard for the modifier and map owner to communicate, and there is nothing to help track or coordinate that. There are more problems if multiple people try to work on the same map, merging that all together cohesively could be difficult. Last, repackaging a zip is a bit troublesome, potentially error prone, a barrier that makes it that much harder to go through all this. At some point it is difficult enough, nobody but the map creator will be willing to modify it - which hurts community involvement (and ultimately less gets down since public collaboration is reduced/taken away)

Git is not a silver bullet, not all problems go away, but it will help automate a lot of things, generally make the process easier, more automated, and even more transparent.


How Would GitHub Help

Git helps automates much of the source code management workflow. It makes things easier, and much of the collaboration problems are automated or go away. With git:
- Repositories have standard commands for working with them, to download and then upload changes. The commands are pretty simple, and diverse too providing many advanced source code functions (though we'll keep it simple and provide a cheat sheet with the couple key commands you'll need).
- The process of requesting modifications, reviewing them, and merging them is all done through a web UI and each step is automated. This will make it easy for anyone to come along and easily help out with a map.
- 1 GB per repo, closely related maps will get their own repo. This means you have plenty of space to store all sorts of files. You could work on a new map and store different prototypes, and you have 1GB of storage for it all. The actual map in the end will need to be smaller since it is included in the game, but you get a lot of working room in the repo.
- Every file associated with the map will have line by line version control. This means you can track changes to the XML configuration over time. We can also more easily do bulk updates, like if an XML tag changes, we have a *hope* of finding and changing it everywhere. Today, unzipping and rezipping every file I think kills the little hope there is of doing something like that.
- Auto-Magic building of maps into zip files will be provided. So essentially you'll be able to push a version of a map into "GitHub Releases" where others can download it. The DL speed from GitHub is pretty fast as well, I get 5Mb/s from github (compared to 200kb/s from sourceforge).
- Other benefits, we'll be able to more easily package maps with the game engine


So, please let me know what questions, thoughts or concerns ya'll have. If there is support for this or alternatively maybe reasons I have not considered for sticking with SVN.

Cheers
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

The Red Baron
I like the idea. One question: Why wouldn't we ask map-makers to host their own maps on Github? Core maps that come with TripleA like Classic and Napoleonic Empires would be github.com/triplea-game/classic and so on, while say TWW would be github.com/lh-studios/total_world_war, or something like that.

Atom.io, Github's hackable text editor that they built in-house has a system similar to this with their packages and themes. They are listed on the Atom website with download links and descriptions as one might expect, but they are each hosted on their developer's Github account. Just throwing it out there

Here is a link:
https://atom.io/packages
"The aggressive spirit, the offensive, is the chief thing everywhere in war, and the air is no exception." - Manfred von Richthofen
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

Veqryn
Administrator
I had a lot of problems in the past with people hosting their own maps, and me linking directly to them.
I would prefer we have them loosely managed centrally.
Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

redrum
Administrator
So I think this is a good idea but there a few things to consider:
- How many repos will be need and what is the best way to group maps?
- Would it be best to first move the maps over as zips then work on unzipping them?
- Changing where maps are hosted is actually a larger change then moving the map source over since then we have to update TripleA to point to the new file hosting location and make sure it works well. Need to think about whether we do a hard cut over or manage both github/SF file hosting for a period of time.

I think probably setting up a new repo and doing some prototypes is the best way to start.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
If we agree to move to git, then I would want the prototype to be accepted after a few review iterations. The threat of work being thrown away, this getting dragged out or design by committee is unappealing enough to make this not happen.

Moving to git can take many forms. A decentralized, many-repo model with "map-owners" instead of "map makers" solves a ton of problems and really encourages and invites community involvement. I would only want this kind of structure for it to be worth moving out of SVN.

Some good news is that this can also be done in an incremental fashion, without updating one line of game code yet. For now simply having each map file in source control and available as a zip file in github releases is very valuable. Worst case, map addition requests will instead of attaching a zip file, will include a link to a github releases URL.

Best case, we update the game code to have a white list of game maps and versions with checksums that have been reviewed and are "blessed" for official release. The game then would be automatically be bundled with a selection of these and  the rest available for in-game download. Those updates could come down the road, but I would personally be inclined to see that they get done. I've scoped out those code changes, they are reasonable and could be tied together with some other game download improvements I would like to make.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
Quick prototype repository for 270BC created.  It won't be completely pain free to get all the maps into git, but it does look like it could be done in a day or so.

Source code:   http://github.com/triplea-maps/270bc
Zip file download: http://github.com/triplea-maps/270bc/releases


For the most part the prototype repo is complete. The only thing missing are two automated build files. Happily we've already set those build files up for the game engine, so setting them up here should not be too much of a problem.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
If the prototype is looking good, some next steps:
- add map packaging script
- add repositories for more/all maps
- add map-admin teams
- add a tutorial repository with instructions on how to clone and update maps
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
This post was updated on .
@RedRum

- "How many repos will be need and what is the best way to group maps?"

Effectively a repo per map. A different org could be of benefit even, I see more reason for it than reason against. The activity in maps is going to be much more than what there will be in game engine, and the admin effort required will be far greater. The team structure for maps will also be quite different, will be decentralized, and will need to support magnitudes more activity than compared to what is happening with the game engine. The current git org admin structure is not going to work for maps.


- "Would it be best to first move the maps over as zips then work on unzipping them?"

Move unzipped.


- "Changing where maps are hosted is actually a larger change then moving the map source over since then we have to update TripleA to point to the new file hosting location and make sure it works well."

I do not think this is quite true yet. As mentioned previously, but I'll answer here for completeness -> We can host the map files in github and push the built zipped versions to source forge. We gain quite a bit of benefit from even just this intermediary step (more map collaboration, process automation, and version tracking at a file level rather than zip level), and it buys time if needed while needed code updates are done.



To proceed, either I'll need temporary super-user privileges on the current github org to set up the repo's and teams, or alternatively we'll roll with the triple-maps org. If we go with the latter option, I can get the repo's set up in the next two weeks, probably the next few days even. If we go with the former, if the permissions happen soon, then a few days, otherwise I'll be more occupied in a couple of weeks and it'll take longer.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
Great news, the map release automation is completed, and now the prototype is relatively complete (http://github.com/triplea-maps/270BC). Only thing left to figure out is how to handle map skins, whether they get their own repo or are bundled with the map repo (though answering that question is not a blocker for starting the migration)

One bit of bad news, there is a one time setup per repository for authorization tokens and the map name as environment variables in the Travis web UI. This adds an hour or two to the whole set up process. On the plus side, this and some hand curation of the map files are the only manual things that need to be done per repository. Given there will 50 to 100 repositories, anything that needs to be done per repository will be quite time consuming.


Some overall good news, the total time to set up all the repos I estimate to be between 5 and 10 hours, not too bad:
 - 2 to 4 hours to deal with the per map oddities, cleaning files up and getting things consistent
 - 2 to 4 hours to set up all the repos and unzip the maps and get everything uploaded into github
 - 1 to 2 hours to do the manual setup for each repo (a minute long operation repeated 50 to a 100 times)

Once that is done, we'll have maps hosted in Git. From there we could:
- form map owner teams around maps and assign people into them
- update the game engine XML to download maps from github instead of source forge (in game download speed should then increase by a factor of 5 to 20 depending on time of day and geography. I have yet to get better than 300kbs/s downloading from sourceforge, while I can easily hit 5 or 6Mb/s from github).
- update game engine code so the XML can be structured to not know about map versions, just map names. This will allow map makers to update maps without needing to involve game engine developers in any way.


Okay, so with the release automation completed, here are what the new work flows would look like:


WORKFLOWS:

A) MAP OWNER RECEIVING AN UPDATE REQUEST
- map owner waits around for an email that notifies a request has been submitted
- they go to github and can go through a colored line by line diff of everything that chnaged
- if things look good, the map owner clicks the "accept" button, and they are DONE.

B) MAP OWNER RELEASING A NEW MAP VERSION
- map owner heads to the github web UI releases page (ex: http://github.com/triplea-maps/270BC/releases)
- they'll see an edit button next to each release, they click that button and then uncheck the "pre-release" checkbox. They are then DONE. Without too much effort the game engine can be updated to automatically pick up on this, we won't need to update XML files whenever a map version changes.

C) MAP-OWNER MAKING A CHANGE
- get set up with git (one time process, 10-30minutes, there will be some instructions on how to do this).
- run  a command to get a local copy of the map repository (git clone)
- make changes
- run a command to confirm the changes (git commit)
- run a final command to push changes (git push), and you are DONE. If you want to update the official released version of the map, now follow the steps in B.

D) NON-MAP-OWNER MAKING A MAP CHANGE
- go to github webUI and click the "fork" button, this will create personal copy of the repository that will be used for staging changes.
- follow the steps from C, but insteaad work against the forked repository
- once done and changes are pushed per the last step in C, now go to the GitHub web UI page and you'll be given a button to click to submit the change request. If there is no feedback from teh map owner, then you are DONE.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

ZjelcoP
So this would make it easy, organized and easy to track changes when working on a map?
I'm working on the Tutorial map with Crazy German and willing to try it out.
Now we keep sending files to and from over the forum.

I could only make a new repository: https://github.com/ZjelcoP/Tutorial

Should we work there and move it when finished to triplea-maps or can you create a repository that I can manage for me?

Any other help and tips is appreciated.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
@ZjelcoP - send me a zip file, or a post a link to one with all of the tutorial map files that we have so far. I'll create a repository, seed it, and will set up permissions.

Otherwise, I also need to create some various documentation:
- most important for the people here, a set of tutorial documentation for working with map files that are hosted in github
- a document on how the map build system is put together
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

ZjelcoP
Some documentation would be essential, because now I can't even create a folder or upload a file to my own repository.
I can do anything with a regular map folder, but Github is still quite a mystery to me.

I'll see if I can put togetether the files for the tutorial map and send it.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

ZjelcoP
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
Tutorial map repo created: https://github.com/triplea-maps/Tutorial

Documentation links added to the top of this post, notably a tutorial that helps a person get set up step by step (windows, all GUI, no command line needed, hopefully will be very easy for almost anyone to follow)
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
This post was updated on .
This thread hasn't been updating in a while.

So the "Project" repo looks to be working out as a home for map 'issues' and wiki pages. It's a unifying place for things that are global to all maps. A place to ask questions: http://github.com/triplea-maps/Project/issues/new

(http://github.com/triplea-maps/Project)

The documentation on how to get started will all be living in the wiki at that repo:
http://github.com/triplea-maps/Project/wiki

Also, there even is a *chat* room: https://gitter.im/triplea-maps/Project  : )
The chat room is open to everyone! The conversation is recorded into a temporary history, so others can get a run down of what has been discussed. The chat is good for quick back and forth things, often the beginnings of conversations that eventually can lead into a feature request, or what not.. Namely, an avenue for less formal communication.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
In reply to this post by aaalafayette
Right now automation has taken over and is uploading all maps from from a recent SVN snapshot. Quite a bit of progress has been made on the automation and some key first pieces are looking complete.

Next Steps:
- let automated map upload finish
- run automated map checker on all repos, fix any problems
- some by-hand verification
- community adoption
- update game engine code to download from github instead of source forge

Migrating maps from SVN presented a number of challenges. My first approach was to try bundling several maps into a single repository. So to start I began with 3 repositories each with one map. Due to size limits, to store all 110+ maps we were expecting to have around at least 12 to 15 total repositories to start. So my starting with three repositories was a limited proof of concept in this direction. I quickly found even with just 3 repositories, triplicating everything was a PITA. Clearly repeating things 12, 15, 20 times is not going to fly.

From here I tried a different strategy, namely to boost hte level of automation for these three repositories. With automation, the cost of many repositories is pretty much the same as one, you just have to let your script run for longer (and there is more verification since there is just more overall stuff). Also part of seeking automated solutions was from some readings about Git Submodules. Namely those require scripting support otherwise management spirals out of control.

Once I started working on the automation, it became clear that dozens of arbitrary maps in a dozen repos was going to present a lot of problems. But, if each repository was uniform, then the automation becomes much easier. To boot, if each repository is uniform, you just have to learn one to understand them all, adding a new one is easier since you have a model to follow. Last, constructing them is actually a lot easier since you just need to worry about one set of uniform build rules. So then at this point it started to become convincing that one map per repository would be the way to go.

The automation for this quickly came into place, and two weeks ago I was able to create github repositories out of all 110+ SVN hosted maps with about 4 hours of manual clicking and copy pasting work (mainly to set up the travis build, and add the map admin teams).

From here, I realized three things. First, the automation needs to be an idempotent script that we can point at a zip file to either create either a new repository, or to update an existing one. This is great since the old SVN work map admins did would be automated with this. Pass me map zips, and any updates, and I could just use this script to update things. 3 months down the road, nothing to be relearned, an admin would just use the script and the script remembers all of the gory setup. Second, the full process needed automation. 4 days the whole setup was automated. But backing up, last, not only did we need an automated set up, but we also needed automated verification too. With 110+ map repositories, nobody is going to believe they are all correct and working, automated set up or not.

The checker script then was subject to intense development, I pointed the checker script at an empty folder and built it up until it identified everything that was missing from a fully working repository. More recently, about a week ago, adding Git LFS into the mix broke bundled map images.

Some final updates were then needed to the checker script, namely to catch this problem and to fully verify the map zip release.  So now, the full checker script verifies nearly 30 different bullet points. Broadly, it checks:
- local git configuration
- local travis (build) configuration  
- local folder layout
- map xml present and has correct mapName property
- repository exists in travis and the latest build succeeded
- downloads the latest map zip file from github and verifies:
  - zip file is between 100KB and 100MB
  - extracted file count of the zip file matches what is in the repository "map" folder
  - file sizes of what is extracted matches the originals in the "map" folder


So again, the automation is looking pretty good now. The very next step in this project will be to verify *all of the things*. Otherwise, I think we're quite ready to really start considering official adoption of maps in Github.
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
This post was updated on .
Slight change of plan, I'm going to prune back to just a half dozen repositories, plenty sufficient for a proof of concept rather than making everything perfect.  If we get agreement to follow up and actually use maps in github, then I'll spend some time to get everything uploaded and set up perfectly.


@Veqryn/@Redrum, @MapMakers:

Next Steps:
- community adoption (agreement to move forward with maps in github / http://github.com/triplea-game/triplea/issues/235)
- make map repo's perfect (run automated checker and fix the couple one-off problems)
- update game engine code to download from github instead of source forge
Reply | Threaded
Open this post in threaded view
|

Re: Re-homing Maps in Git

aaalafayette
Administrator
In reply to this post by aaalafayette
@MapMakers: if you want to be an early adopter and use github to help build maps, please post a link to your map zip file to this thread. (This is while we wait to figure and agree how / if to update the game engine. In the meantime the github repo is pretty useful for keeping track of your files and building them up into a zip).

Any zip link posted I'll zap with the newly built admin script and it'll become a fully functional github repository. Send me your github name as well so I can add you as admin of your map.

Note, your map does not at all have to be ready. Github is a tool, it's encouraged to use it earlier in the map making process if you can and if it helps out.