GitHub Migration!

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

GitHub Migration!

aaalafayette
Administrator
This post was updated on .
Unless Veqryn has objection:

let's move to github!


To do this migration, let's stay focused on what we are replacing! Specifically this migration means we will be modifying the development work flow for tripleA:
1. When getting an initial copy of code, instead doing a SVN checkout from source forge there will be a git clone from git hub instead.
2. Instead of attaching patch files to patch tickets, we'll put a git hub pull request link into a patch ticket. One day soon we'll migrate ticket engines perhaps away from source forge. IMO for a release or two we may be better off using as much of the existing process as possible.


Management of the git hub repository

My understanding is existing convention is that only redrum or Veqryn can approve code to be merged. For now let's keep that convention.

Next steps

I'll try to lead the charge on this as much as possible since I brought it up and am interested. To do this migration, I'll probably need some time from Redrum, Veqryn, and any other volunteers willing to help. Below are the steps I plan to take:

1. I'll set up a git repository
2. Seed it with the current SVN
3. Update this thread
4. Do a practice run of a pull request (and create some documentation)
5. Do a practice pull request with either @Redrum or @Veqryn
6. Transfer git hub ownership to Redrum and Veqryn
7. Send Redrum/Veqryn documentation to be posted
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
We still haven't addressed the issues that come with moving to GitHub:
1. The forum
2. The website
3. The tickets (bugs, feature requests)
4. The hosting of our release files (the installers for the game)

If we move without addressing the above, then we will basically have a bifurcated community, with things split between sourceforge and github.

In any case, I am not going to stand in the way of this.  Everyone wants not just git, but github, so I'm not going be a blocker.

There are some things I would like to see if this does happen though:
1. No personal ownership.  The master/origin repository should be owned and managed by an organization which you guys are a member of.
2. Our SVN history, including who did which commit, is kept intact.


I can help with #1 at least.  I've created a new github org, called "TripleA-Game" (sorry, but 'triplea' was taken), and started a single new repository in it just called 'triplea'.
https://github.com/TripleA-Game/triplea

Please list your github usernames and I'll give you all access/ownership of it.

You can make a second repo called triplea-test or something and experiment with migrating with history, and then once you get it right we can delete that and do it with the main triplea repo (and post instructions on how it was done).  Or whatever.

Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

aaalafayette
Administrator
Okay, some good news and progress with respect to git migration.

The generic SVN to GIT migration process is documented here pretty well: https://www.atlassian.com/git/tutorials/migrating-overview/

Bernat had a start on this already with his git repo: https://github.com/gaborbernat/triplea. It contains history, the git repo redrum started I think does not.

Next steps

* Veqryn to add LaFayette to github organization
* Following SVN to GIT migration guide (link above), LaFayette will fork the repo started by bernat, put it under control of the tripleA github organization, and then update it to include the latest SVN commits
* We'll then continue periodically updating Git with the latest SVN until we have more experience using Git, and also have decided when/how exactly to transition. If things move quickly we may be able to get current in-flight patches to be converted to pull requests maybe.
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

aaalafayette
Administrator
In reply to this post by Veqryn
@Veqryn

I agree with keeping the SVN history and no single-ownership.

I suspect that the other topics not yet addressed are complicated enough discussions that each one might need its own thread. My concern is that we get too bogged down deciding which tool to use for XYZ that we remain unnecessarily blocked on git migration.

I think it's worth perhaps being clear on why we are migrating to git:
- it largely automates the existing SVN patch process. This means we can get more people reviewing more easily, and generally this part of the code update process is not as much of a bottleneck.
- git offers more powerful code management tools than compared to SVN (though SVN is quite functional)
- last, the larger development community has amazingly arrived at general consensus that GIT is the best tool of its kind (for the moment, and unlike other more notorious debates like tabs vs spaces and Java vs C).

Okay, with all that said, I'm going to create a new thread to discuss the unaddressed issues: http://tripleadev.1671093.n2.nabble.com/SourceForge-Deprecation-Discussion-tp7588771.html 

We can then keep this thread focused on issues relating to git hub migration, input on the process, and notifying of progress and next steps.
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

aaalafayette
Administrator
@Veqryn

With respect to bifurcation, github migration hopefully will not be too bad. My thinking is that we'll get git hub up and going in a "mirror and update" mode. Where it is updated/mirrors the SVN version for a while until we are comfortable doing a cold turkey switch.

I don't think we'll be able to get everything switched to its final form. There is risk that we start the migration and don't finish it all the way. I expect for us to be migrating from source forge for a while. Perhaps the goal for now would be in places where there is a clear win, try to do the migration in a surgical type of fashion, migration only what is needed, and identify what else needs to be moved. So meaning, if we don't complete the migration in one go, at least we'll be sure to small pieces of it, but at least they'll be complete pieces
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

redrum
Administrator
@Veqryn - Here is my github username: ron-murhammer

@aaalafayette - The only thing I'll add is I'd really like to avoid having to push code back and forth between the 2 for very long. I think we can even move each piece you've identified one at a time. So if we get comfortable with the code in github then we can make it the source for that as we continue to move the remaining pieces.

I'll add my comments on the other pieces in the other thread.
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
In reply to this post by aaalafayette
ok, so i am giving this a college try, and i've already run into a small problem:

we have data that doesn't really belong in the trunk, but isn't a branch or tag

https://sourceforge.net/p/triplea/code/HEAD/tree/

/additionalImageData
/doc
/tdice
/triplea_map_creator
/triplea_old_map_creator
/upnp
/utilities

and maybe also:
/trunk/triplea/ant


i am pretty sure that /doc is just an outdated copy of /trunk/triplea/doc, but everything else needs a home somewhere, especially the /upnp directory

so the big question: what to do with these?

also, as far as branches and tags go, I really would only want to keep the largest number stable in each major (2nd digit) release, and only for the last few (plus 1.0.0.2).  so this means keeping branches (actually tags):
1_0_0_2
1_2_5_5
1_3_2_3
1_5_2_2
1_6_1_4
1_7_0_5
1_8_0_5
Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

aaalafayette
Administrator
One general rule of thumb is Git is for source code only. Strictly source code, no build artifacts, jars, third party stuff, etc.. Git does not delete anything ever, so anything added to git or modified will create new copies that never go away.

Basically means it's a good time for us to clean house. For each thing we should figure out if it is needed, should be moved to out of source control, or if it should be migrated into git.
 
With respect to releases @Veqryn. Wherever we store binaries/releases is the place to store the last few copies. If tags are not carried over in the SVN -> GIT migration, it should be pretty possible to amend the Git history with tags (and then worst case check out those revisions and build those artifacts in the new process).


Should also address branches. Branches which were used as ways to revisit history, we will have automatically in GIT and they can be discarded. Branches that have unmerged feature work are more troublesome. Hopefully there are not many of these branches out there, I would recommend we try to get them all merged in so we only need to worry about migrating the trunk.
ice
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

ice
can someone explain in english what u guys are talking about? does this have any impact to the game/comunity and or mods, or is it all development related?

will it be forum nr 3 for tripple a? if so is it possible to merge everything into 1 uptodate website with 1 forum? warclub and developpers sites im talking about. or is it something else?

whatever happens veqryns vote would mean allot

ice

Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
In reply to this post by aaalafayette
i think we may not need to keep track of these, since they are already elsewhere:
/additionalImageData
/doc

the following are source code, but source that is not directly used in the triplea.jar.  I would like to keep this stuff somewhere, even if it is just a zip somewhere:
/tdice
/triplea_map_creator
/triplea_old_map_creator

the following is source code that compiles into the upnp.jar, which we should keep track of somewhere, even if just a zip somewhere:
/upnp

the following are some utilities, and I think they can be removed:
/utilities
/trunk/triplea/ant


also, since git is 'just for source code', does that mean we should have our other assets in there, like the images, the sounds, etc?  where-ever those live, i do want them managed by version control in some way.

also, i'm not sure I understood what you were talking about with tags and branches.

the versions i listed above in my post are 'tags' that i do want to have in git (currently they are in svn as branches).

i do not want to have the releases (the binaries and installers) to be part of git, those will just go to wherever we host our releases.


Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

pug6000
It was mentioned that git does not forget. The same is true for svn. Both keep all artefacts and ressources.

If you migrate to git, all should be moved, source code, images, sounds, etc.
Same for the libraries, as you are using a specific version that might not exist in the web later on.

Self compiled stuff should not be stored, as it can be recompiled anytime.

Best regards
Pug
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

redrum
Administrator
I don't think its a big issue if we store images/sounds in github. If we want it somewhat separate then we can create another repo under our org for those.

As for the other items:

/tdice
/triplea_map_creator
/triplea_old_map_creator

Aren't these other standalone applications? If so putting these in a separate repo would probably make sense.

/upnp

Isn't this a third party library? Why do we have the source instead of a jar? Did we have to modify it?
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

aaalafayette
Administrator
In reply to this post by pug6000
This is a time a video conference is called for :)   Trying to answer every bullet point is proving to be a futile task.

For us to have a good git experience, we must try to follow the rule of thumb that binary objects do not get stored in git. This was taught to me by people that are actual git experts (I am not a git expert). Though, to not be taken out of context, I am *not* saying that we ditch all binary objects from version control. Pretty much everything should be in a version control system of one sort of another. It's simply that Git is not the right version control system for everything (to name others to be more specific: Maven Central or Nexus for java libraries, git-annex or even S3 perhaps for images and sounds).

For text files which we are not sure to move or not:
* Default answer is leave it in SVN and move it when you find you have a specific need for it.
* SVN is not going away right away, so we can get stuff out of there if we need it.
* When considering different repo or same, I'd probably lean to keeping each repo very well defined and would go with a different one maybe. Ultimately considering how much shared code there is I think drives a lot of this decision. It's not necessarily a bad thing to have multiple top level folders in your git repo (so both a triplea and mapeditor project could be in the same repo).

For library files (JARs):
* Gradle (using Maven), the build system, will get the library files. Before you would get them with an SVN checkout, now they'll be obtained after a git clone via one of:
./gradlew eclipse  | ./gradlew intellij | ./gradlew build


For binary objects:
I think we'll want to use one of either GIT LFS or Git annex. I'm not sure which is the better tool, or if there is an even better solution.  

Versioning for these objects would be done with file naming (ie: my_image_v1.0.jpg, my_image_v1.1.jpg). S3 in this case could be an option. If we get things going correctly with gradle, the number of times these objects would actually be downloaded would be pretty small.



@Veqryn
Sorry to not have been more clear with tags and branches.

For SVN branches, I think we can leave them alone. If there are any that are feature branches which are not yet merged back into trunk, I suggest we try to get them merged before the migration.

For tags, we will hopefully get them during the migration. If not, we can add them into git by hand.


@Bernat Does this jive with how you envision gradle integrating in with the overall build process?
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
i can agree in principal with most of what you said,
but i prefer git/svn versioning for binary files over name based versioning (which i feel is both error prone, and annoying to code references to)

i think for now, lets definitely keep everything in the svn triplea/trunk/triplea directory

last question for now is about trunk/triplea/ant/findbugs
do we want to keep that, or not?
Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

bernat
In reply to this post by aaalafayette
Hello,

I've extensively worked with git over the last four years (for both my own projects and at work). Would not say that I'm expert with it, but have strong experience with it. At some point I've used most of the features git provide, and have a somehow strong understanding of its inner workings. I'm for one is for the move.

Binary files, they can go into git. Git excels at text data, but supports binary data. Adding binary data will significantly increase the repository checkout procedure and size requirements. No other drawback against using it. Just not strongly advised for it. There is no need to decide now, although could be beneficial. One can remove them later on with https://rtyley.github.io/bfg-repo-cleaner/.

I've heard good things about Git-LFS for asset storing (https://git-lfs.github.com/). So would incline to that.

My approach would be, port the current svn repository as it is (not that much fuzz, already done it once earlier here - https://github.com/gaborbernat/triplea).  Make the transition, add gradle, keep using the system as it is. Remove with bfg binaries no longer required. Move out large binaries to git lfs or something similar. As we move out more and more binary files, cloning the repo will be easier and easier.

Would not want to remove any features without having an alternative. For findbugs may first integrate the gradle findbugs plugin.

For non strictly triplea related stuff - yeah such as upnp, they can get versioned; just not under the triplea project. We should create an organization and have multiple repositories under it. All stuff closely related to triplea should stay in the triplea repository.

Strong supporter of single repository clone should already set up a development environment (that will allow for new people coming in, and easily contributing). Not sure how much does this count, but having to use svn and send patches in mail did slowed down my contribution to this project significantly, and still is a pain do deal with it.

There's no need for version branches. A single main history branch is enough, version releases should be just tags on this main branch. Branching should be for features and bug fixes.

PS. Also prefer git versioning over numbered versioning.

Thanks,
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

ubernaut
In reply to this post by ice
@ice they are simply talking about the way the code is developed for triplea maybe it will be moved to a newer platform, one which is maybe easier for people to use than the current system (SVN/sourceforge).
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
This post was updated on .
In reply to this post by bernat
Ok, I've added everyone to the GitHub project.

I also finished converting triplea svn into git and commit it (sorry for delay, git-svn took 30 hours to complete).

I would greatly appreciate if you guys could review it: https://github.com/TripleA-Game/triplea-test1

But please note, that I will be deleting this and starting over once I've ironed out all the mistakes from the process.

I will post my planned implementation in another post, which I would also like reviewed.

For now, I would appreciate if you guys let me give this a shot, as I need to learn.



Things that I know I need to figure out for the next try:

1. Top level of git repo came in as the 'triplea' directory under /trunk/, instead of coming in as /trunk/triplea (I am still debating if this is good or bad)
2. Tags came in as branches
3. I wasn't able to specify which branches/tags I wanted to bring it, it brought in everything (which is why it took so long)

Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

redrum
Administrator
@Veqryn - Looks pretty good. Comments:

1. You don't want /trunk/ in git. The directory structure looks correct to me. Though forking it and cloning it locally will be the true test.

2/3. I don't see any tags or branches.

One other thing you'll probably want to do is when you convert it is an authors file to map sourceforge users to github users. This will make it so all your commits in sourceforge show up under the proper user in github so you'll have more than 1 commit :)

To that point you may want to reach out to those that have contributed significantly to see if they have github usernames. This shows a little on the authors file mapping: https://www.getdonedone.com/converting-5-year-old-repository-subversion-git/
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

Veqryn
Administrator
i followed the directions in the link that was provided (https://www.atlassian.com/git/tutorials/migrating-overview/)
i did the authors file and the branches, so they should both work
Please contribute to the TripleA 2013 donation drive:
http://tripleadev.1671093.n2.nabble.com/2013-TripleA-Donation-Drive-tp7583455.html
Reply | Threaded
Open this post in threaded view
|

Re: GitHub Migration!

redrum
Administrator
Oh ok. Looks like the authors file either wasn't used or isn't specifying either the sourceforge or github users right. Can you post the authors.txt file?
123