Timestamps are in GMT/BST.
[6:29] -orwell.freenode.net- *** Looking up your hostname...
[6:29] -orwell.freenode.net- *** Checking Ident
[6:29] -orwell.freenode.net- *** Found your hostname
[6:29] -orwell.freenode.net- *** No Ident response
[6:29] * DuraLogBot (~PircBot@webster.duraspace.org) has joined #duraspace
[6:29] * Topic is '[Welcome to DuraSpace - This channel is logged - http://irclogs.duraspace.org/]'
[6:29] * Set by cwilper!ad579d86@gateway/web/freenode/ip.173.87.157.134 on Fri Oct 22 01:19:41 UTC 2010
[11:49] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) has joined #duraspace
[12:08] * mhwood (mwood@mhw.ulib.iupui.edu) has joined #duraspace
[12:28] * tdonohue (~tdonohue@c-98-226-113-104.hsd1.il.comcast.net) has joined #duraspace
[13:19] * peterdietz (uid52203@gateway/web/irccloud.com/x-tlwveoijuuburswz) has joined #duraspace
[14:54] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[15:29] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 272 seconds)
[15:41] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[15:46] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 252 seconds)
[15:56] * srobbins (~srobbins@libstfsdg02.library.illinois.edu) has joined #duraspace
[15:59] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[16:04] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 252 seconds)
[16:16] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[16:23] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 246 seconds)
[16:34] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[16:44] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 246 seconds)
[16:50] * mhwood (mwood@mhw.ulib.iupui.edu) Quit (*.net *.split)
[16:50] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) Quit (*.net *.split)
[16:50] * misilot (~misilot@p-body.lib.fit.edu) Quit (*.net *.split)
[16:56] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[16:59] * mhwood (mwood@mhw.ulib.iupui.edu) has joined #duraspace
[16:59] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) has joined #duraspace
[16:59] * misilot (~misilot@p-body.lib.fit.edu) has joined #duraspace
[17:02] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 272 seconds)
[17:07] * mhwood (mwood@mhw.ulib.iupui.edu) Quit (*.net *.split)
[17:07] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) Quit (*.net *.split)
[17:07] * misilot (~misilot@p-body.lib.fit.edu) Quit (*.net *.split)
[17:10] * mhwood (mwood@mhw.ulib.iupui.edu) has joined #duraspace
[17:10] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) has joined #duraspace
[17:10] * misilot (~misilot@p-body.lib.fit.edu) has joined #duraspace
[17:13] * airborn_ (~airborn@unaffiliated/airborn) has joined #duraspace
[17:19] * airborn_ (~airborn@unaffiliated/airborn) Quit (Ping timeout: 250 seconds)
[17:25] * pbecker (~pbecker@ubwstmapc098.ub.tu-berlin.de) Quit (Quit: Leaving)
[18:00] * hpottinger (~hpottinge@mu-160190.dhcp.missouri.edu) has joined #duraspace
[19:51] * KevinVdV (~KevinVdV@d54C5104D.access.telenet.be) has joined #duraspace
[20:00] <tdonohue> Hi all, it's time for our weekly DSpace DevMtg. Agenda: https://wiki.duraspace.org/display/DSPACE/DevMtg+2015-10-14
[20:00] <kompewter> [ DevMtg 2015-10-14 - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/DevMtg+2015-10-14
[20:01] <tdonohue> Two main topics today (as usual)... 5.4 and 6.0
[20:02] <tdonohue> First, starting with 5.4, it seems like we are now to a point where we have a "decent number" of important fixes to get into 5.x. So, I'm wondering if we should think about scheduling a 5.4 release in the near(ish) future
[20:02] <tdonohue> Here's the list of things scheduled for 5.4 right now: https://jira.duraspace.org/browse/DS/fixforversion/12942/
[20:02] <kompewter> [ DSpace: 5.4 - DuraSpace JIRA ] - https://jira.duraspace.org/browse/DS/fixforversion/12942/
[20:03] <tdonohue> We essentially have 4 still open issues... 3 of which have plausible PRs (may need minor refactoring on a few). and then there's DS-2788 which is "unsolved"
[20:03] <kompewter> [ https://jira.duraspace.org/browse/DS-2788 ] - [DS-2788] Performance degradation of SOLR Discovery indexing in DSpace 5 - DuraSpace JIRA
[20:04] <tdonohue> It seems like if we can tackle these 4 tickets in the near(ish) term, we have ourselves a decent 5.4 release
[20:05] <tdonohue> Anyone from @mire here know of the status of 2788? KevinVdV or cwilper perhaps?
[20:05] <terry-b> We are seeing an issue in 5.3 that we are likely to report. We cannot get /rest to run in tomcat8. It runs in tomcat7. Have others experienced this?
[20:05] * aschweer (~andrea@d157.itstaff.waikato.ac.nz) has joined #duraspace
[20:06] <KevinVdV> I’m not really up to date on this particular issue, I know of its existence but that is it. My main focus is 6.0 at the moment
[20:06] <aschweer> hi all, sorry I'm late
[20:07] <tdonohue> terry-b: I feel like peterdietz (or someone else) was recently working on Tomcat8 + REST...but it may have been on the "master" branch.
[20:08] <peterdietz> hi all. late, and not all that present, have a call starting any minute.
[20:08] <hpottinger> I've been tinkering with Solr performance a bit, have made some optimizations in the server config, and have done a bit of tinkering with the search solr_config to turn on cache autowarming
[20:09] <peterdietz> on localhost rest is fine on tomcat8
[20:09] <tdonohue> Regarding a 5.4, I'll also note that Google Scholar has discovered DS-2679, and are "pestering" me (nicely though), about when it'll be fixed. It's a "fixed" ticket, but unreleased as of yet
[20:09] <peterdietz> does catalina.out have anything?
[20:09] <kompewter> [ https://jira.duraspace.org/browse/DS-2679 ] - [DS-2679] Google Scholar ordering of metadata tags with multiple values (like authors) broken - DuraSpace JIRA
[20:10] <hpottinger> re Solr: I've got "average" (actually "mode") query response time down to about 251MS at at the worst, and more typically at 20-80MS
[20:10] <aschweer> I'm just going through "code review needed" issues to see whether there are others that might qualify for 5.4
[20:10] <terry-b> peterdietz, we have not yet seen anything helpful. I will encourage my colleague to put in the ticket and any data that might be helpful.
[20:11] <tdonohue> Here's some other PRs that might qualify for 5.4...they are flagged as such in GitHub: https://github.com/DSpace/DSpace/milestones/5.4
[20:11] <kompewter> [ Issues · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/milestones/5.4
[20:12] <tdonohue> Essentially here, what I'm getting at is that (1) we seem to have some pretty significant bug fixes already into the 5.x branch (and more that could/should be added soon). (2) We really should think about a 5.4.... (3) So, do we schedule it, or continue to "wait" around for things like 2788?
[20:14] <tdonohue> (crickets) ;)
[20:14] <hpottinger> I'll add a comment to 2788 re: cache autowarming, though that's probably just a win for my repository... we have facets on our home page, and other links to very broad searches... which means keeping the cache around is good for us
[20:15] <tdonohue> Ok..well, I'll say that I'd like to see a 5.4 soon. I think the Google Scholar bug fix is very important (and GS is complaining), the Solr logging fix is important...both of these are already in. Also, DSPR#1045 seems important
[20:15] <kompewter> [ https://github.com/DSpace/DSpace/pull/1045 ] - DS-2699 improve search by aschweer
[20:15] <aschweer> agreed, search + Scholar are pretty important use cases
[20:16] <hpottinger> is DSPR#1045 something that can go into 5.4?
[20:16] <kompewter> [ https://github.com/DSpace/DSpace/pull/1045 ] - DS-2699 improve search by aschweer
[20:16] <tdonohue> So, if there's anyone out there who'd like to help get a 5.4 release, I'd do my best to help out
[20:17] <cwilper> err, re:2788, i think we could still chip away at improving things there. i do plan to run some re-indexing perf. tests soon-ish and i suspect a PR may come out of that within the next couple weeks.
[20:17] <hpottinger> I will also promise to help out, I can't RC any more, I've promised people I won't, but I can help
[20:17] <tdonohue> hpottinger: good point on 1045... we do need to discuss the Boolean operator change (from OR to AND). But, there is two fixes there..and the initial fix *definitely* applies to 5.4. The question is only whether the boolean operator change should be rescheduled for 6.0
[20:18] <hpottinger> so, we need two PRs and two issues, then?
[20:19] <aschweer> I'm happy to split the PR if that helps
[20:19] <mhwood> I think it should be split.
[20:19] <tdonohue> yea, 1045 probably should be a split PR with two tickets. The first commit actually *fixes* the bug that is DS-2699. The second commit changes the boolean operator
[20:19] <kompewter> [ https://jira.duraspace.org/browse/DS-2699 ] - [DS-2699] Search not working as expected - DuraSpace JIRA
[20:20] <tdonohue> I think the first commit *needs* to be in 5.4. The second one is more questionable (but the idea is a good one) and may need to be rescheduled for 6.0
[20:20] <aschweer> yeah well as I've said before, as far as I'm concerned OR default is a bug, but fair enough that this may not be the case for everyone (especially if there are local customisations that rely on OR being default)
[20:20] <aschweer> and I know OR has been the default for a long time
[20:21] <tdonohue> aschweer: I actually agreed with you until I realized that DSpace has been using boolean OR since *forever* (possibly since the beginning).
[20:21] <aschweer> tdonohue: thanks for looking that up
[20:21] <tdonohue> aschweer: so it made me wonder if we really need to wait for a major release to do full testathon & announcement of "DSpace, now with boolean AND" ;)
[20:22] <aschweer> yes
[20:22] <aschweer> what should the operator change do with the dspace.cfg setting? just remove it from dspace.cfg (since it's unused) and we make sure to put instructions in the upgrade docs?
[20:23] <tdonohue> The dspace.cfg search settings only apply to Lucene-based search. They should be removed once Lucene is removed (hopefully in 6.0)
[20:23] <aschweer> oh also, I think the bugfix commit in my PR is only for XMLUI, it'd be great if someone could look at JSPUI (I'll be away next week)
[20:23] <aschweer> ah good point re Lucene still being in the codebase
[20:23] <aschweer> might be a good idea to go through dspace.cfg at that point and check what else is Lucene-only
[20:24] <tdonohue> yep, agreed. The config will be undergoing a major review shortly in my DSPR#1104...I plan to rip it a part a bit as I already found some obsolete configs in there
[20:24] <kompewter> [ https://github.com/DSpace/DSpace/pull/1104 ] - DS-2654: Reloadable Configurations via Apache Commons Configuration by tdonohue
[20:25] <tdonohue> In any case, back to 5.4. Is there anyone who'd be willing to help do the 5.4 release (as part of a mini-release team)?
[20:26] <hpottinger> it's fun, honest
[20:26] <aschweer> I'm tempted but I'll be away next week and again mid-November, I fear I just won't have the time
[20:27] <mhwood> Do we have a date for this? My schedule gets messy as year's end draws near.
[20:27] <tdonohue> we can work it around your schedule, aschweer, if you are still "willing" :) I'm also feeling swamped, but am willing to chip in
[20:28] <tdonohue> mhwood: I'm trying to tease a date out of us.. That's what I'm worried about.. I feel like 5.x needs a 5.4, and I'd rather it not "slide" too long (into holidays/travel)
[20:28] <aschweer> Is there a list of what's involved in a bugfix release? And/or an idea of how much work this might be?
[20:29] <tdonohue> Mostly it's helping get PRs tested a bit (or at least the key ones), which others can help with too (self included). Then we schedule a deadline, cut the release & announce. To lead the release, you'd mostly just be the person cutting the release, and helping draft the announcement (I'll chip in though too)
[20:30] <tdonohue> we'd set the deadline based on when the release will be "cut". Then we'll work in these meetings to ensure PRs get tested/reviewed based on that deadline
[20:30] <aschweer> That doesn't sound so bad (famous last words)
[20:31] <tdonohue> It shouldn't be bad. I would do it, but I feel like I'm way behind on my 6.0 contribution (reloadable configs) + UI Prototype
[20:31] <aschweer> It would have to be the first week of November for me, or after Nov 17
[20:32] <tdonohue> First week of November seems reasonable to strive for. If we miss it, we can reschedule (and find someone else, or push to after Nov 17)
[20:32] <mhwood> I can probably help out a bit in that timeframe.
[20:32] <hpottinger> note, DSpace 6 schedule works into this same timeframe a bit
[20:32] <aschweer> Ok I guess I just volunteered then :) Yes help would be much appreciated, I've never even been on a release team.
[20:33] <hpottinger> you were conscripted to the 5.0 RT, if memory serves, and you helped quite a bit
[20:33] <tdonohue> Thanks aschweer! Welcome to the RT. By the way, once you are on the RT, you are always on the RT (kidding)
[20:33] <aschweer> Yeah but I didn't push any buttons
[20:33] <aschweer> haha
[20:34] <hpottinger> the button pushing is the easiest and most fun part
[20:34] <mhwood> And there's a written procedure on the wiki.
[20:34] <aschweer> until something breaks...
[20:34] <hpottinger> nothing ever breaks
[20:34] <aschweer> yeah right!
[20:34] <tdonohue> Ok. So, we have a tentative 5.4 schedule. First week of November.
[20:35] <hpottinger> https://wiki.duraspace.org/display/DSPACE/Release+Procedure#ReleaseProcedure-AdviceforfutureReleaseCoordinators
[20:35] <kompewter> [ Release Procedure - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/Release+Procedure#ReleaseProcedure-AdviceforfutureReleaseCoordinators
[20:35] <tdonohue> aschweer: we can schedule the date you plan to "cut" the release at a time where more of us will be lurking in IRC... (i.e. morning for you, afternoon for USA folks)
[20:35] <tdonohue> aschweer: that way if something "goes wrong" there's folks there to help out
[20:35] <aschweer> aschweer: That scheduling sounds like a good plan
[20:35] <aschweer> tdonohue obviously
[20:35] <aschweer> sorry
[20:35] <aschweer> see that's what happens during my mornings :)
[20:35] <mhwood> :-)
[20:35] <tdonohue> (feel free to talk to yourself)
[20:35] <aschweer> yeah well at least that'd probably be in English these days
[20:36] <hpottinger> we should add that advice to the wiki
[20:36] <aschweer> what, have >1 coffee before coming to the dev meeting?
[20:36] <hpottinger> "feel free to talk to yourself"
[20:36] <aschweer> oh right
[20:36] <aschweer> sure
[20:37] <tdonohue> Ok, so I'm satisfied that we have enough of a 5.4 "plan" for now. Hopefully we can all work together to get more of the bug fixes ready in the next few weeks
[20:37] <tdonohue> (and we can discuss them in upcoming meetings as needed, as well)
[20:38] <aschweer> Cool. I won't be in next week's, since I'm away and it's also the middle-of-the-night one for me
[20:38] <aschweer> But I'll have a look over PRs/issues not yet scheduled for 5.4 that look like good candidates for it, and will also see which PRs/issues for 5.4 need chasing up in terms of testing
[20:38] <hpottinger> aschweer: the procedure page has stuff you can do beforehand, you can do the dry run, and test your GPG keys and log into Sonatype, all that stuff
[20:38] <mhwood> The usual suspects will be around on IRC most days.
[20:39] <aschweer> Great, thanks for the reassurance hpottinger and mhwood and tdonohue
[20:39] <tdonohue> thanks, aschweer. sounds good!
[20:39] <tdonohue> Next topic for today then is 6.0 updates, where I'd like to open up the floor... Anyone have major 6.0 updates to share or issues needing discussion?
[20:40] <terry-b> I migrated a large database instance to 6.0. The migration completed, but my web apps would not start.
[20:40] <tdonohue> terry-b: any specific errors from the webapps? We'd probably need more info on what "went wrong"
[20:40] <terry-b> When I migrated a smaller database with the same code, web apps started. I put details into Jira
[20:41] <terry-b> The CPU hit 100% and none of the apps were responding to requests.
[20:41] <terry-b> I am concerned that the box was trying to cache lots and lots of objects.
[20:41] <terry-b> We plan to beef up the test server to see if we can draw any better conclusions.
[20:42] <terry-b> The error logs were failing on MultiPartFilter request. Very vague errors.
[20:42] <terry-b> My goal was to performance test the PR I have submitted for /rest.
[20:42] <tdonohue> terry-b: You might also want to check the tomcat logs. Sometimes if the dspace logs are vague, there may be a more specific error in Tomcat
[20:43] <tdonohue> terry-b: but honestly, thanks for all this awesome scalability testing. I'm hoping we can more easily determine if what you are seeing is a large cache, or if it's actually even a bug hiding in the code
[20:44] <terry-b> We looked and could not find anything useful yet.
[20:44] <terry-b> We'll add more to JIRA after we try on more powerful server.
[20:44] <mhwood> I wonder if turning up the log level would help.
[20:45] <terry-b> mhwood, I tried that as well.
[20:45] <KevinVdV> Well you can always output all sql queries by adding a single line to the config (if that helps)
[20:45] <mhwood> Maybe run with a profiler? At least have some idea of where it's spending all this time.
[20:46] <terry-b> We have new relic running on the server. It was showing "IO Wait" taking up 90+% of the CPU
[20:46] <mhwood> Much swapping?
[20:46] <tdonohue> terry-b: I'd also be curious just to see a sample from the logs (of the errors when you try to startup one or more webapps). Even if the errors are vague to you, it's possible it might trigger an idea from one of us
[20:48] <terry-b> mhwood, it swas not showing swapping, but something seemed odd. I presume that postgres was struggling. tdonohue, I will pull some data out of the logs and post it into jira
[20:48] <terry-b> https://jira.duraspace.org/browse/DS-2798
[20:48] <kompewter> [ [DS-2798] Serious performance issue upgrading large database to DSpace 6 schema - DuraSpace JIRA ] - https://jira.duraspace.org/browse/DS-2798
[20:48] <tdonohue> terry-b: another idea is to see if DSpace responds via the commandline, or if it also "hangs"... So, simply running "./dspace database info" or other basic commands just to see if the database itself is "healthy"
[20:48] <mhwood> Dalibo has some nice PostgreSQL performance monitoring tools.
[20:48] <kompewter> [ https://jira.duraspace.org/browse/DS-2798 ] - [DS-2798] Serious performance issue upgrading large database to DSpace 6 schema - DuraSpace JIRA
[20:50] <mhwood> https://github.com/dalibo/pgbadger
[20:50] <kompewter> [ dalibo/pgbadger · GitHub ] - https://github.com/dalibo/pgbadger
[20:51] <tdonohue> In any case, we'll be looking forward to anything else you discover terry-b. Hopefully we can figure out what is going on & how to fix it.
[20:51] <terry-b> Thanks. I will keep you all posted
[20:52] <mhwood> http://pgcluu.darold.net/
[20:52] <kompewter> [ pgCluu :: PostgreSQL performances monitoring and auditing tool ] - http://pgcluu.darold.net/
[20:53] <tdonohue> I'll give a general update. My contribution (Apache Commons Config, DSPR#1104) is starting to "snowball" a bit with the new API refactor. In essentially replacing ConfigurationManager with ConfigurationService, I'm having to also replace some of the other "static" *Manager classes that depend heavily on ConfigurationManager (e.g. PluginManager).
[20:53] <kompewter> [ https://github.com/DSpace/DSpace/pull/1104 ] - DS-2654: Reloadable Configurations via Apache Commons Configuration by tdonohue
[20:54] <tdonohue> I'm *nearly* there (had to replace PluginManager also with PluginService)...but my PR is going to be rather massive.
[20:54] <hpottinger> pull on one thread
[20:54] <tdonohue> hpottinger: exactly.
[20:54] <mhwood> We should look over the whole plugin/service area. Much duplication of function. Much mixing of only superficially related concepts in both.
[20:55] * hpottinger vaguely hopes AuthN/Z gets caught up in it all :-)
[20:55] <tdonohue> Unfortunately the "thread" seems to have become more prominent in the refactored API (as everything is more service based). In the 5.x API, I was able to "work around" PluginManager easier
[20:56] <tdonohue> mhwood: yes, agreed. For now, I'm not doing that (as that's a massive project in its own right). I'm just turning PluginManager into a bean/service... so the ugly "plugin.*" configs still will exist (for now)
[20:56] <mhwood> Understood. We'll get those sorted later!
[20:57] <tdonohue> Anyone else have updates to share on 6.0-related stuff?
[20:57] <mhwood> Alas, no, not now.
[20:58] <KevinVdV> I would love to get some testers on: https://github.com/DSpace/DSpace/pull/1105, very tricky query stuff going on there (since I’m not really a DB wizard)
[20:58] <KevinVdV> The oracle one scares me a bit :)
[20:59] <tdonohue> it seems like something worth testing at the same level as terry-b has done (although, we'd also want to try to find an Oracle tester, obviously)
[21:00] <hpottinger> why's everyone looking at me? :-)
[21:00] <KevinVdV> Starts looking at hpottinger
[21:01] <hpottinger> OK, fine, I'll help
[21:01] <tdonohue> KevinVdV: one thing that stands out as slightly odd to me in that PR... It looks like there'd now be two columns... "bitstream_order" and "bitstream_order_legacy"? I don't see any dropping of the old column? https://github.com/DSpace/DSpace/pull/1105/files
[21:01] <kompewter> [ [Task: DS-2807] Bundle bitstream ordering fix for hibernate by KevinVdV · Pull Request #1105 · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/pull/1105/files
[21:01] <KevinVdV> Yeah I should drop it perhaps, but wanted to keep in there for testing purposes
[21:01] <tdonohue> Oh, I see. So, it's still a work in progress, slightly
[21:01] <KevinVdV> So you can see what it was before & after, but I will make a note to the PR to drop it
[21:02] <tdonohue> definitely do :)
[21:03] <tdonohue> hpottinger: thanks for volunteering to ehlp
[21:03] <KevinVdV> There add a log + in progress
[21:03] <tdonohue> help ;)
[21:03] <KevinVdV> & thx hpottinger, I did a small test myself but a second pair of eyes is always helpfull
[21:04] <tdonohue> Any other general questions/ updates for today? (I notice we just passed the 1 hour mark)
[21:05] <mhwood> Sorry, I have to leave. 'bye all!
[21:05] <KevinVdV> Need to run as well, until next time
[21:05] <tdonohue> Ok, hearing no other updates, we'll close the meeting for today
[21:06] <tdonohue> thanks all!
[21:06] * mhwood (mwood@mhw.ulib.iupui.edu) has left #duraspace
[21:06] <terry-b> take care!
[21:06] * KevinVdV (~KevinVdV@d54C5104D.access.telenet.be) Quit (Quit: KevinVdV)
[21:07] <aschweer> right, time for that second coffee. thanks all!
[21:07] * aschweer (~andrea@d157.itstaff.waikato.ac.nz) Quit (Quit: leaving)
[21:14] * hpottinger (~hpottinge@mu-160190.dhcp.missouri.edu) Quit (Quit: Leaving, later taterz!)
[21:41] * tdonohue (~tdonohue@c-98-226-113-104.hsd1.il.comcast.net) has left #duraspace
These logs were automatically created by DuraLogBot on irc.freenode.net using the Java IRC LogBot.