Managing Release and Integration Cycles

Utilizing Instability and Breakage to Promote DSpace Evolution, Code Review and Developer Participation

To Give DSpace Developers opportunities to work on the trunk without restrictions that everything going into it be completely perfected gives the developer community and the codebase opportunities to evolve. We recognize that sometime disturbance and breakage has a stimulating effect on developer participation.  Therefor it is proposed that the DSpace Guidelines for Committing to trunk allow for period of instability and stability with clear deadlines as to allow for the introduction and testing of new features which have been decided to be included into the next release by the community.

Commiter Rights and Google Summer of Code Participation

This proposal originally arose out of the Google Summer of Code project with the need to actually have GSoC students help with the code merge of their projects into Trunk. But, as students are not "Committers", they currently are limited in receiving rights to Trunk (under our current 'policies').  It is recommended that we hold a Special Topic meeting next week, and try and determine what to do in this scenario. Obviously, this proposal then leads to other questions of
how we may view committer rights.

Issues In Relation to Asynchronous Release and Modularity

Robin Taylor says "Having completely forgotten about yesterdays developers meeting (doh !) I just read the log and was interested in the chat about committer rights etc. I don't want to preempt any further discussion but just wanted to say that it might be logical to have the discussion about modularisation and asynchronous releases before the discussion on committing to trunk. The envisaged shape of the svn repo might affect how we view committer rights eg if Sword lived in its own module with its own release cycle then people might be less concerned about an individual with an interest in Sword having commit rights just to that module".

Original Dialog In IRC Meeting

[20:03] <span style="color: #000000"><tdonohue> mdiggory: are there any GSoC upcoming meetings to know about (I know this was mentioned as still being worked on at/before OR10)?</span>[20:03] <span style="color: #000000"><mdiggory> Monday, August 9 Suggested 'Pencils Down'</span>
[20:04] <span style="color: #000000"><mdiggory> Friday, August 20 Final Evaluation Deadline/Final Payment Issued</span>
[20:04] <span style="color: #000000"><mdiggory> Monday, August 23 Final Results Announced</span>
[20:04] <span style="color: #000000"><mdiggory> Monday, August 30 Students Submit Code Samples</span>
[20:04] <span style="color: #000000"><tdonohue> (FYI for all -- this schedule is also embedded on our GSoC wiki page: </span>[https://wiki.duraspace.org/display/DSPACE/Google+Summer+of+Code)\|../../../../../../../../../display/DSPACE/Google+Summer+of+Code)|]
[20:04] <span style="color: #000000"><mdiggory> All students did receive passing grades for the midterm.</span>
[20:05] <span style="color: #000000"><tdonohue> that's great news all around!</span>
[20:05] <span style="color: #000000"><mdiggory> But, now I think begins the challenging period of us... how do we finish out this years to have the projects follow through to codebase / trunk where appropriate</span>
[20:05] <span style="color: #000000"><mhwood> JIRA incident to track each one?</span>
[20:07] <span style="color: #000000"><mdiggory> Well, we started out with some failure to setup a JIRA project for GSOc, now I'm feeling that was maybe a bad choice...</span>
[20:08] <span style="color: #000000"><mdiggory> so, now that we have material to bring into various levels of core, it would be good to see tickets in appropriate places for those activities "DSpaceJIRA" , "Services JIRA" etc</span>
[20:09] <span style="color: #000000"><tdonohue> yea -- it's better to split up into manageable "chunks"</span>
[20:10] <span style="color: #000000"><mdiggory> But, simply getting everyone used to creating tickets around the GSOC projects is the first challenge</span>
[20:10] <span style="color: #000000"><tdonohue> it'd be harder to "approve" a JIRA issue if it's really just a placeholder for the whole project -- I imagine for some projects only "part" may be trunk-ready, and the rest may need a bit more work</span>
[20:10] <span style="color: #000000"><mdiggory> I see Unit testing as a good first candidate</span>
[20:11] <span style="color: #000000"><tdonohue> well -- couldn't we consider the creation of final JIRA issues as part of the "final project report" (i.e. the final deliverable)</span>
[20:11] <span style="color: #000000"><mhwood> Binding each project more tightly to its place in DSpace than to GSoC also emphasizes that this is "real", not just an exercise. We want to see this stuff become production-ready.</span>
[20:12] <span style="color: #000000"><tdonohue> mhwood +1</span>
[20:13] <span style="color: #000000"><mdiggory> I agree, that not all the projects will be good candidates for trunk, but ticketing for other projects needs to target appropriate modules projects</span>
[20:14] <span style="color: #000000"><mdiggory> so for storage and semeantic web projects we have the "DSpace Services" JIRA project that might be a good home</span>
[20:15] <span style="color: #000000"><tdonohue> mdiggory -- yes, I agree with that. That's what I was trying to get at by saying that a "JIRA placeholder" is less useful than splitting up the project and adding separate (but related) JIRA issues into each of the appropriate modules in JIRA</span>
[20:15] <span style="color: #000000"><mhwood> Might even want several tickets for a single project if it usefully breaks down into components in varying states of readiness.</span>[20:16] <span style="color: #000000"><tdonohue> exactly, mhwood -- in my opinion, several tickets is much better than one ticket (as most of these projects touch several parts of the codebase)</span>[20:18] <span style="color: #000000"><mdiggory_> ok, I was going to say that the REST work, if we are truly targeting 1.</span>
[20:18] <span style="color: #000000"><mhwood> Time to go fishing in syslog.</span>
[20:18] <span style="color: #000000"><mdiggory_> 1.7 with it, might have a home in the dspace 1.x ticketing</span>
[20:19] <span style="color: #000000"><tdonohue> yea -- I was hoping that both REST and some of the very early Unit Testing could start to be pushed towards Trunk (and DSpace 1.7)</span>
[20:19] <span style="color: #000000"><mdiggory_> (notes this will open up a discussion into async release process etc)</span>
[20:20] <span style="color: #000000"><tdonohue> (but whether my hopes match with reality is another matter -- in any case, JIRA tickets should help us all get a better sense of how "ready" each of these projects are)</span>
[20:21] <span style="color: #000000"><kshepherd> morning all, sorry for lateness</span>
[20:21] <span style="color: #000000"><tdonohue> hi kshepherd -- still talking GSoC right now</span>
[20:21] <span style="color: #000000"><mdiggory_> I think a goal with the REST work is that it support legacy DSpace at the same time as new services portings... IMO it should be pluggable to that degree and I know that Aaron would probably recommend such a target.</span>
[20:22] <span style="color: #000000"><mdiggory_> I think there is a point where some destabilization of trunk needs to happen to get the work in... and we need to aggree on that period as much as on the feature freeze dates etc</span>
[20:23] <span style="color: #000000"><mdiggory_> sso, for instance, if destabilization happens in sept then feature freeze could happen in oct</span>
[20:23] <span style="color: #000000"><mdiggory_> or august/sept</span>
[20:23] <span style="color: #000000"><tdonohue> that all makes sense -- I still think the best first step is to have someone (preferrably the students, perhaps with mentor help) to create the appropriate JIRA tickets for each project -- that way we can push them for review and try and come to a quick consensus on next steps</span>
[20:24] <span style="color: #000000"><mdiggory_> I agree, but we need to agree on who does that commit work, and I will suggest students over mentors</span>
[20:25] <span style="color: #000000"><tdonohue> mdiggory_ you mean actually having the students commit the code to trunk? (or am i misunderstanding)</span>
[20:25] <span style="color: #000000"><mdiggory_> yes</span>
[20:26] <span style="color: #000000"><mdiggory_> We talked about temporary commit rights etc at the or10 meeting...</span>
[20:26] <span style="color: #000000"><tdonohue> hmmm... as much as I'd like to say "yes", that does make me slightly nervous (in that we haven't had a more thorough review of the code by committers to help decide which parts are 'non-controversial')</span>
[20:26] <span style="color: #000000"><mdiggory_> my point in suggesting it is to aleaviate yet another gatekeeper role / bottleneck in the process</span>
[20:27] <span style="color: #000000"><tdonohue> yes, we did talk about the temporary commit rights at OR10 -- but if i recall, it was tabled for a full-meeting discussion (special topic mtg)</span>
[20:27] <span style="color: #000000"><mhwood> Commit to a branch and then merge slightly later?</span>
[20:27] <span style="color: #000000"><tdonohue> yea, i definitely see the benefits mdiggory -- i completely agree with minimizing bottleneck (just worried at whether this is controversial amongst our committers)</span>
[20:28] <span style="color: #000000"><kshepherd> it makes me slightly nervous </span>
[20:28] <span style="color: #000000"><tdonohue> commit to a branch first may be less controversial</span>
[20:28] <span style="color: #000000"><mdiggory_> I challenge that this community needs to change its perception of trunk.</span>
[20:28] <span style="color: #000000"><kshepherd> on the other hand, i've argued that broken trunks aren't the end of the world</span>
[20:28] <span style="color: #000000"><mdiggory_> but and do so respectfully, I know why the first blush is to be conservative.</span>
[20:28] <span style="color: #000000"><tdonohue> mdiggory_ i understand your challenge (and honestly, I agree to a point) -- it's just something that needs to be voted on</span>
[20:29] <span style="color: #000000"><mdiggory_> of course</span>
[20:29] <span style="color: #000000"><tdonohue> so -- if you want to propose this (likely via email, we don't have enough here now), we can bring to a vote and see what happens</span>
[20:29] <span style="color: #000000"><kshepherd> if we really can't live with broken trunks, we'll just keep our own branches on our own svn repos anyway</span>
[20:30] <span style="color: #000000"><kshepherd> so i think i'd vote for it.. might need to think more though </span>
[20:30] <span style="color: #000000"><mdiggory_> mhwood: unless there is a review between the commit to the branch and the merge... I don't see usefullness in that strategy.</span>
[20:30] <span style="color: #000000"><tdonohue> I see three options: (1) allow commit to trunk , (2) allow commit to a branch, and later merge to trunk (after more review/testing), (3) wait to move to trunk until review/testing (the old way of doing things -- which sometimes leaves things languishing)</span>
[20:30] <span style="color: #000000"><mhwood> The branch is to create space for review, with the code checked into the SCM so we can use all the usual SCM tools for inspecting and manipulating it.</span>
[20:31] <span style="color: #000000"><mdiggory_> thats already happening</span>
[20:31] <span style="color: #000000"><mdiggory_> but we need to be very cautions about how long those branches sit... deviating from trunk</span>
[20:31] <span style="color: #000000"><tdonohue> mdiggory_ it's happening separately right now though (each project has it's own 'branch') -- I think mhwood is suggesting a "merged" branch</span>
[20:32] <span style="color: #000000"><mdiggory_> the longer they do so, the lower their probability for inclusion</span>
[20:32] <span style="color: #000000"><mhwood> Agreed: *prompt* review and decisions.</span>
[20:32] <span style="color: #000000"><tdonohue> agreed -- I think the goal would be that the branch is very temporary -- a place for immediate review and then hopefully move to trunk before 1.7</span>
[20:32] <span style="color: #000000"><mdiggory_> tdonohue: I just think thats a not such a good idea, this is really what a trunk is for</span>
[20:33] <span style="color: #000000"><mhwood> So where should we be looking? The SCM tree seems to have grown like kudzu lately.</span>
[20:33] <span style="color: #000000"><mdiggory_> and I agree with kshepherd, breakage on the trunk is actually a good thing</span>
[20:33] <span style="color: #000000"><mdiggory_> is the disturbance that promotes change/improvement</span>
[20:33] <span style="color: #000000"><kshepherd> yeah we do need to get less precious about trunk, i think </span>
[20:33] <span style="color: #000000"><tdonohue> mdiggory_ yes -- but, do these students have experience with merging? I'm worried that a merge could accidentally break more than we bargained for (but I could just be paranoid)</span>
[20:33] <span style="color: #000000"><kshepherd> and we have more code review tools to help with post-commit reviews, too</span>
[20:34] <span style="color: #000000"><mdiggory_> tdonohue: that is why we are here as mentors, to assist them in learning such things if they do not</span>
[20:34] <span style="color: #000000"><mdiggory_> ie mentoring = helping</span>
[20:34] <span style="color: #000000"><mdiggory_> not mentoring = judging</span>
[20:34] <span style="color: #000000"><grahamtriggs> depends on the way you want to view it - yes, we could get less precious and more experimental with trunk... but that would mean we should be branching early for releases</span>
[20:35] <span style="color: #000000"><mhwood> Yes, be as experimental as we want, so long as there is an island of relative stability *somewhere*.</span>
[20:35] <span style="color: #000000"><mdiggory_> grahamtriggs: or we work in phases/periods of stability /instability</span>
[20:35] <span style="color: #000000"><mdiggory_> ie... controlled chaos</span>
[20:36] <span style="color: #000000"><tdonohue> so, i'll also note here that "breaking trunk" currently is against our recently approved "Guidelines for Committing" -- </span>[https://wiki.duraspace.org/display/DSPACE/Guidelines+for+Committing\|../../../../../../../../../display/DSPACE/Guidelines+for+Committing|]<span style="color: #000000"> (obviously, we can change these guidelines as we see fit, but we need to have larger discussion)</span>
[20:36] <span style="color: #000000"><mdiggory_> So, table for special topic meeting that needs to happen asap?</span>
[20:36] <span style="color: #000000"><mdiggory_> so more developers that have an interst in such a policy have time to respond</span>
[20:37] <span style="color: #000000"><tdonohue> mdiggory_ : yes, we could even schedule mtg for next week -- but, we need to also send an email proposing these changes (for those unable to attend)</span>
[20:38] <span style="color: #000000"><mdiggory_> I volunteer kshepherd </span>
[20:38] <span style="color: #000000"><tdonohue> mdiggory_ : will you take lead on proposing this to dspace-devel today/tomorrow? We can put on agenda for next week's mtg.</span>
[20:38] <span style="color: #000000"><kshepherd> hmm? for what?</span>
[20:38] <span style="color: #000000"><tdonohue> haha -- you are the one who brought this all up, mdiggory_ </span>
[20:39] <span style="color: #9c009c">* kshepherd runs svn delete on trunk and runs away cackling</span>
[20:39] <span style="color: #000000"><tdonohue> haha </span>
[20:39] <span style="color: #000000"><mdiggory_> to write an email about proposing to change the Guidelines to allow more breakage or destabilization phases on trunk</span>
[20:39] <span style="color: #000000"><mdiggory_> ok ok ok...</span>
[20:40] <span style="color: #000000"><tdonohue> yes -- and explain the background as to *why you propose this* -- which is that the proposal is that the GSoC students commit ready code to trunk</span>
[20:40] <span style="color: #000000"><tdonohue> those are two separate issues, I know -- but we might be able to cover both in one special topic mtg</span>
[20:40] <span style="color: #000000"><kshepherd> my "break trunk" opinions have mostly been in the context of the way we tend to leave bugfixes going stale in JIRA instead of just committing.. this is a slightly different context so i'd like to read/think more</span>[20:42] <span style="color: #000000"><tdonohue> ok -- anything else immediately on GSoC? I notice we only have 20mins left (not that there's much more substantial on the agenda though)</span>
[20:42] <span style="color: #000000"><mhwood> kshepherd, that is a bit different. I think many of us tend to want more comment than we give, so changes rot....</span>
[20:42] <span style="color: #9c009c">* kshepherd nods</span>
[20:43] <span style="color: #000000"><mdiggory_> but I think it is related</span>
[20:43] <span style="color: #000000"><kshepherd> no better way to enourage peer review than accidentally breaking trunk, though </span>
[20:43] <span style="color: #000000"><mdiggory_> because it has to do with getting things into trunk, then hardening them when errors arise</span>
[20:43] <span style="color: #000000"><kshepherd> so like mdiggory_ says, it can actually force us into action</span>
[20:43] <span style="color: #000000"><tdonohue> true -- but we also don't want to harm productivity of our widely distributed group by having a broken trunk for too long </span>
[20:43] <span style="color: #000000"><mdiggory_> rather than waiting long periods of time and only exceppting the most perfect merges</span>
[20:43] <span style="color: #000000"><kshepherd> and will get us used to using atlassian code review tools more</span>
[20:44] <span style="color: #000000"><mdiggory_> which raises the bar for contribution</span>
[20:44] <span style="color: #000000"><mhwood> Do those tools work now? Must go look again.</span>
[20:44] <span style="color: #000000"><kshepherd> tdonohue: heh true, of course.. bear in mind that i think 95% of commits won't actually result in a broken trunk </span>
[20:44] <span style="color: #000000"><tdonohue> yea agreed </span>
[20:44] <span style="color: #000000"><kshepherd> mhwood: heh, i hope so! i think there is a test review in crucible</span>
[20:44] <span style="color: #000000"><mdiggory_> yes, crucible is still working</span>