GammaLib

The only git workflow I have used so far is the one used in the open source world as e.g. on github or bitbucket:
http://docs.astropy.org/en/latest/development/workflow/index.html

Basically every developer forks the main repo, creates as many branches there as she likes, and makes pull requests against the main repo when she thinks something is ready.
Then the main repo maintainer reviews the contribution and then typically after a few iterations merges it into the main repo.

Having all branches in the main repo will not work very well even for 10 developers I think, but I never tried it.

What I would like to have as a developer is the ability to have unit tests run on one of my branches or pull requests and to have the issue / pull request discussions connected to the code nicely.

Jürgen, would it be technically possible to have forked repos for developers or is that too hard to set up (at least now as long as there are only few developers)?

What do you think about mirroring the gammalib repo on github?

It could be read-only for now, but I would “watch” it to get notifications of what is going on in the code.
This is what we do with the HESS CVS repo, there’s a read-only bitbucket repo.

Also github is rolling out project-wide search very soon (already live for private repos), which overall gives a nicer code browsing and searching experience than
https://cta-redmine.irap.omp.eu/projects/gammalib/repository

Christoph mentioned that the developer should rebase on devel before sending a pull request, so that the reviewer can merge without conflicts:

http://docs.astropy.org/en/latest/development/workflow/development_workflow.html#rebasing-on-trunk

Christoph Deil wrote:

What do you think about mirroring the gammalib repo on github?

It could be read-only for now, but I would “watch” it to get notifications of what is going on in the code.
This is what we do with the HESS CVS repo, there’s a read-only bitbucket repo.

Also github is rolling out project-wide search very soon (already live for private repos), which overall gives a nicer code browsing and searching experience than
https://cta-redmine.irap.omp.eu/projects/gammalib/repository

How would the repositories by synchronized? If it’s just a “window” for our code, I’m in favor. I guess we may want to do the same for sourceforge and other possible distribution sites. This also keeps our name reserved

Actually sourceforge is slow, ugly and full of spam. Many open source projects have switched to github (or bitbucket) in the last years, gammalib could do the same.
There’s so little content in the sourceforge bug tracker and mailing list that it’s not worth migrating that.
The mailing list could become a google group and we could use google talks and hangouts for weekly short status meetings and coding sessions (coding is more fun that way and it helps to get started if you can ask questions and have a reply quickly).

Let’s focus here on the Git workflow. We can have a separate topic on public repositories.

Christoph Deil wrote:

The only git workflow I have used so far is the one used in the open source world as e.g. on github or bitbucket:
http://docs.astropy.org/en/latest/development/workflow/index.html

Basically every developer forks the main repo, creates as many branches there as she likes, and makes pull requests against the main repo when she thinks something is ready.
Then the main repo maintainer reviews the contribution and then typically after a few iterations merges it into the main repo.

Having all branches in the main repo will not work very well even for 10 developers I think, but I never tried it.

What I would like to have as a developer is the ability to have unit tests run on one of my branches or pull requests and to have the issue / pull request discussions connected to the code nicely.

Jürgen, would it be technically possible to have forked repos for developers or is that too hard to set up (at least now as long as there are only few developers)?

Technically, what does this mean? Having a fully copy of the developer repository on our server?

Jürgen Knödlseder wrote:

Christoph Deil wrote:

The only git workflow I have used so far is the one used in the open source world as e.g. on github or bitbucket:
http://docs.astropy.org/en/latest/development/workflow/index.html

Basically every developer forks the main repo, creates as many branches there as she likes, and makes pull requests against the main repo when she thinks something is ready.
Then the main repo maintainer reviews the contribution and then typically after a few iterations merges it into the main repo.

Having all branches in the main repo will not work very well even for 10 developers I think, but I never tried it.

What I would like to have as a developer is the ability to have unit tests run on one of my branches or pull requests and to have the issue / pull request discussions connected to the code nicely.

Jürgen, would it be technically possible to have forked repos for developers or is that too hard to set up (at least now as long as there are only few developers)?

Technically, what does this mean? Having a fully copy of the developer repository on our server?

Yes.
I never looked at non-hosted git / user management systems and I don’t know what you use now, so I have no idea how easy or hard this is.
But diskspace shouldn’t be an issue, right?

Christoph Deil wrote:

Jürgen Knödlseder wrote:

Christoph Deil wrote:

The only git workflow I have used so far is the one used in the open source world as e.g. on github or bitbucket:
http://docs.astropy.org/en/latest/development/workflow/index.html

Basically every developer forks the main repo, creates as many branches there as she likes, and makes pull requests against the main repo when she thinks something is ready.
Then the main repo maintainer reviews the contribution and then typically after a few iterations merges it into the main repo.

Having all branches in the main repo will not work very well even for 10 developers I think, but I never tried it.

What I would like to have as a developer is the ability to have unit tests run on one of my branches or pull requests and to have the issue / pull request discussions connected to the code nicely.

Jürgen, would it be technically possible to have forked repos for developers or is that too hard to set up (at least now as long as there are only few developers)?

Technically, what does this mean? Having a fully copy of the developer repository on our server?

Yes.
I never looked at non-hosted git / user management systems and I don’t know what you use now, so I have no idea how easy or hard this is.
But diskspace shouldn’t be an issue, right?

We can certainly investigate this issue, but implementation would take some time. Diskspace is probably not the (primary) issue, it’s more the handling of access rights.

Yesterday we had an interesting meeting with people from other projects (e.g. Euclid) on community development frameworks. I heard about something called gitlab which could be an interesting solution: http://gitlabhq.com/. Maybe not yet mature, but something to check out.

In https://cta-redmine.irap.omp.eu/issues/624 Jürgen the issue came up when unit tests are run on Jenkins.

Ideally as a developer I would have to option to run them for a given branch before I ask for my code to be merged into one of the major branches.
IMO the next best thing is that Jürgen triggers the tests for my branch when I ask him to merge (i.e. make a “pull request”) into devel.
I think first merging into devel and then seeing the test failures and have a second pull request with fixes or more work on the existing branch is not a good workflow.

One more thing: ideally as a developer I’d like ssh access to the machine where my test failed to be able to debug it.
I realize this is probably not possible to set up for security reasons.
But that means that if I break the Solaris build Jürgen has to fix the issue.

Christoph Deil wrote:

In https://cta-redmine.irap.omp.eu/issues/624 Jürgen the issue came up when unit tests are run on Jenkins.

Ideally as a developer I would have to option to run them for a given branch before I ask for my code to be merged into one of the major branches.
IMO the next best thing is that Jürgen triggers the tests for my branch when I ask him to merge (i.e. make a “pull request”) into devel.
I think first merging into devel and then seeing the test failures and have a second pull request with fixes or more work on the existing branch is not a good workflow.

The CI suite is quite computing intensive, so it can only be fully deployed on a very restricted number of branches. In the actual setup, a nightly test pipeline is run on the devel branch. We could add an integration branch on which we run the full test pipeline (or a subset?) manually during integration. But this would make the integration process quite long.

One more thing: ideally as a developer I’d like ssh access to the machine where my test failed to be able to debug it.
I realize this is probably not possible to set up for security reasons.
But that means that if I break the Solaris build Jürgen has to fix the issue.

I understand, but indeed it’s not possible in the actual setup as all the VMs are behind the firewall on a local network. The fixes have indeed to be done by the integration manager.

I moved the following over from the “Repository copies” discussion, as the item is more related to the git workflow.

Christoph Deil wrote:

DON’T:
- issue a pull request on github, because it would mean discussions and thus project history are in a second place in addition to the Redmine at IRAP.
Unfortunately github doesn’t allow turning off pull requests, I just complained to them about it, but so have many others in the past year and so far they didn’t change it.
For now we just have to ignore github pull requests against github.com:gammalib/gammalib should there ever be one (this is e.g. what Linus or everyone else does https://github.com/torvalds/linux/pulls )

DO:
- use github.com/gammalib/gammalib to browse code, commits, view diffs between branches, ... or what the repo to get notifications for any new commits if you want.
- fork github.com/gammalib/gammalib to github.com/yourname/gammalib and use your repo on github for the 100 new feature branches you have instead of putting them all in the IRAP repo.
- If you think something is ready to be merged in devel, create a Redmine ticket and ask Jürgen to merge your branch, i.e. make a “pull request”. Typically there will be a Redmine ticket already, because we first discuss what we plan to do before actually implementing it.

Jürgen, do you think this is a good way to operate? Or would you prefer to review and eventually merge branches in the IRAP repo only?

I think we should offer both options.

For the moment we’re a very small number of developers, so it should be possible to work without forks. However, if someone prefers github and the forking model, we should also support this. In the meantime, I will start a discussion here with the informatics team to see whether we can setup a fork model at IRAP. But this will certainly need some time before it could be realized.

I propose to draft the git workflow guideline on a Wiki page named Proposal of Git workflow. Following the discussion on this forum, we can tune this page until we converge on a given model. Once agreed, we copy this model over to the Using Git for GammaLib development page.

In short, I would propose to protect the devel branch on our server for external pushes. People that want to work directly with our repo will create feature branches as needed. Once a feature branch is ready, the developer sends a pull (or merge) request to the integration manager. The integration manager will merge the feature branch in the integration branch, executes the test suite, and when successful, merges the integration branch into the devel branch. The integration manager then deletes the feature branch from the IRAP repo. I guess this will keep (for the moment) the number of branches at a manageable level.

A developer that wants to use github creates a fork of gammalib on github. Similar to the case above, a feature branch will be created in the fork, and once the feature is ready, he issues a pull request on Redmine. The integration manager will then merge the change into the integration branch of the IRAP repo, and perform the integration actions as above.

By the way: I added a tracker “Pull request” to Redmine, so that we can handle them as separate issues. The Pull request has some additional mandatory fields (“Related issue”, “Branch”, “Repository”). Only the manager is allows to modify pull requests (but developers should be able to create them).

I think any links to code in branches from Redmine tickets to help discussions would break anyways as soon as the branch is deleted, so I don’t see the advantage of copying my branch from github to IRAP before asking you to review and pull.

Redmine allows links to commits, so as long as the commits are in the repository, the link will exist.

Here’s an example of what a large feature branch with a month of discussion looks like on github in the end, I don’t think we can do something similarly nice with Redmine:
https://github.com/astropy/astropy/pull/471

Hi Christoph,

I started to describe the Git workflow for GitHub: https://cta-redmine.irap.omp.eu/projects/gammalib/wiki/Proposal_of_Git_workflow. I tested all steps on my machine.

Comments and suggestions are welcome.

Cheers,
Jürgen

Nice writeup!

I think we should add a reference that teaches what rebasing on devel means and how to resolve conflicts, since this is quite hard:
http://docs.astropy.org/en/latest/development/workflow/development_workflow.html#rebasing-on-trunk
(and links there)

One thing that is nice about github is that there is an easy way to turn an issue into a pull request, i.e. to attach commits to an issue:
http://docs.astropy.org/en/latest/development/workflow/development_workflow.html#converting-a-github-issue-to-a-pull-request
This keeps the discussion in one place, whereas you propose to open a second issue in Redmine to signal the pull request.

Is it possible to add a “Pull Request” state to the issue “Status”, or maybe we can use the existing “Review” state for this?

Then add a link to the github branch compare view against devel:
https://github.com/cdeil/gammalib/compare/devel...travis

Actually this is non-ideal, because the developer devel branch will often be out of sync with the main repo devel branch, so you as the reviewer often see an out dated diff.
Using actual pull requests would be better:
https://github.com/cdeil/gammalib/pull/2

But I realize this is moving more and more in the “We use github” direction and you probably don’t want this because there is already a lot of content in the Redmine, and github doesn’t allow free, non-public repos, which we might want / need.

Maybe this could work well?
gammalib devs are allowed to just issue a github pull request against devel at github.com/gammalib/gammalib, so that they immediately see if they have to rebase or not, and you get a nice diff to review, but all discussion must take place on the original Redmine issue?

Christoph Deil wrote:

Nice writeup!

I think we should add a reference that teaches what rebasing on devel means and how to resolve conflicts, since this is quite hard:
http://docs.astropy.org/en/latest/development/workflow/development_workflow.html#rebasing-on-trunk
(and links there)

Agree.

One thing that is nice about github is that there is an easy way to turn an issue into a pull request, i.e. to attach commits to an issue:
http://docs.astropy.org/en/latest/development/workflow/development_workflow.html#converting-a-github-issue-to-a-pull-request
This keeps the discussion in one place, whereas you propose to open a second issue in Redmine to signal the pull request.

Is it possible to add a “Pull Request” state to the issue “Status”, or maybe we can use the existing “Review” state for this?

That’s an alternative possibility. I checked that I can filter on this, and I agree that this is better. I removed the pull request tracker.

Then add a link to the github branch compare view against devel:
https://github.com/cdeil/gammalib/compare/devel...travis

Actually this is non-ideal, because the developer devel branch will often be out of sync with the main repo devel branch, so you as the reviewer often see an out dated diff.
Using actual pull requests would be better:
https://github.com/cdeil/gammalib/pull/2

But I realize this is moving more and more in the “We use github” direction and you probably don’t want this because there is already a lot of content in the Redmine, and github doesn’t allow free, non-public repos, which we might want / need.

Maybe this could work well?
gammalib devs are allowed to just issue a github pull request against devel at github.com/gammalib/gammalib, so that they immediately see if they have to rebase or not, and you get a nice diff to review, but all discussion must take place on the original Redmine issue?

I would prefer not to issue anything at GitHub. The integration manager will anyways have to rebase for security. The request that the developer rebases will just simplify the live for the integration manager, so that the developer fixes already possible conflicts.

I added some text to Proposal of Git workflow to describe the work for the integration manager. I’m not sure that everything on the rebase / merge side is fully correct, but at least it worked for the test case.

Any comments and suggestions are welcome.

I now have to work on the case where developers work directly on the central repository.

Based on the past discussion, I now compiled a number of Wiki pages to describe the Git workflow for GammaLib.

Please read these pages carefully and provided comments and suggestions.

If still have a problem after rebasing using the central repository. Check Central repository workflow for developers for more information. While on GitHub, pushing after rebasing works without problems, I have to pull first when I use the central repository. Otherwise, I get a message

remote: error: denying non-fast-forward refs/heads/007-my-new-feature (you should pull first)