Timestamps are in GMT/BST.
[5:31] * eddies (~eddies@unaffiliated/eddies) Quit (Quit: Leaving.)
[5:37] * eddies (~eddies@cm18.epsilon183.maxonline.com.sg) has joined #duraspace
[5:37] * eddies (~eddies@cm18.epsilon183.maxonline.com.sg) Quit (Changing host)
[5:37] * eddies (~eddies@unaffiliated/eddies) has joined #duraspace
[6:29] -wolfe.freenode.net- *** Looking up your hostname...
[6:29] -wolfe.freenode.net- *** Checking Ident
[6:29] -wolfe.freenode.net- *** Found your hostname
[6:30] -wolfe.freenode.net- *** No Ident response
[6:30] * DuraLogBot (~PircBot@atlas.duraspace.org) has joined #duraspace
[6:30] * Topic is '[Welcome to DuraSpace - This channel is logged - http://irclogs.duraspace.org/]'
[6:30] * Set by cwilper!ad579d86@gateway/web/freenode/ip.173.87.157.134 on Fri Oct 22 01:19:41 UTC 2010
[8:08] * Asger (~abr@nat.statsbiblioteket.dk) has joined #duraspace
[12:25] * mhwood (mwood@mhw.ulib.iupui.edu) has joined #duraspace
[13:08] * tdonohue (~tdonohue@c-67-177-108-221.hsd1.il.comcast.net) has joined #duraspace
[14:28] * tdonohue (~tdonohue@c-67-177-108-221.hsd1.il.comcast.net) Quit (Read error: Connection reset by peer)
[14:30] * tdonohue (~tdonohue@c-67-177-108-221.hsd1.il.comcast.net) has joined #duraspace
[15:18] * Asger (~abr@nat.statsbiblioteket.dk) Quit (Quit: Leaving)
[17:02] * eddies (~eddies@unaffiliated/eddies) Quit (Quit: Leaving.)
[17:06] <tdonohue> Hi all... my normal weekly Office Hours are going on from now until our Devel Mtg. Let me know if there's anything you'd like to chat about: https://wiki.duraspace.org/display/~tdonohue/DSpace+Office+Hours
[17:06] <kompewter> [ DSpace Office Hours - Tim Donohue - DuraSpace Wiki ] - https://wiki.duraspace.org/display/~tdonohue/DSpace+Office+Hours
[18:52] * hpottinger (~hpottinge@mu-162198.dhcp.missouri.edu) has joined #duraspace
[19:16] * sands (~sandsfish@18.189.31.239) has joined #duraspace
[19:24] <sands> Is anyone out there running Shibboleth via Apache with a DSpace 1.8.x instance?
[19:24] * sands crosses fingers
[19:47] <tdonohue> not sure, sands. Are you having an issue with it? Perhaps ask on dspace-devel or dspace-tech?
[19:48] <sands> my colleague is deploying 1.8.2 with Shib and is having a difficult time determining if the Apache config for shib is correct or if it is causing some weird behavior.
[19:48] <sands> just figured i'd ask here quickly. he'll probably hit the lists if there's no immediate resolution.
[19:49] <tdonohue> https://wiki.duraspace.org/display/DSDOC18/Authentication+Plugins#AuthenticationPlugins-ConfiguringShibbolethAuthentication%28DSpace1.8.2%29
[19:49] <kompewter> [ Authentication Plugins - DSpace 1.8 Documentation - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSDOC18/Authentication+Plugins#AuthenticationPlugins-ConfiguringShibbolethAuthentication%28DSpace1.8.2%29
[19:49] <tdonohue> Docs say: "With Shibboleth DSpace will require that you use Apache installed with the mod_shib module acting as a proxy for all HTTP requests for your servlet container (typically Tomcat). DSpace will receive authentication information from the mod_shib module through HTTP headers."
[19:49] <tdonohue> That's as much as I know :) sorry
[19:50] <sands> no problem, thank you tim
[19:52] * helix84 (~a@kou-street209-48.pks.muni.cz) has joined #duraspace
[19:53] <tdonohue> Hi All, reminder that at the top of the hour we have our DSpace Developers Mtg here: https://wiki.duraspace.org/display/DSPACE/DevMtg+2012-08-15
[19:53] <kompewter> [ DevMtg 2012-08-15 - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/DevMtg+2012-08-15
[19:56] * ryscher (98033b9f@gateway/web/freenode/ip.152.3.59.159) has joined #duraspace
[20:00] <tdonohue> Hi all, time for our weekly DSpace Devel Mtg. Agenda: https://wiki.duraspace.org/display/DSPACE/DevMtg+2012-08-15
[20:00] <kompewter> [ DevMtg 2012-08-15 - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/DevMtg+2012-08-15
[20:01] * bollini (~chatzilla@host90-209-dynamic.116-80-r.retail.telecomitalia.it) has joined #duraspace
[20:01] * richardrodgers (~richardro@pool-173-48-255-197.bstnma.fios.verizon.net) has joined #duraspace
[20:01] <tdonohue> As with the last few weeks, I figured we'd bypass JIRA reviews so that we can concentrate on the 3.0 Release (and maybe even call some votes on some of these outstanding features / pull requests)
[20:01] * robint (52292725@gateway/web/freenode/ip.82.41.39.37) has joined #duraspace
[20:02] <tdonohue> First though, just wanted to again publicize the new 3.0 Schedule: https://wiki.duraspace.org/display/DSPACE/DSpace+Release+3.0+Notes#DSpaceRelease3.0Notes-TimelineandProcessing
[20:02] <kompewter> [ DSpace Release 3.0 Notes - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/DSpace+Release+3.0+Notes#DSpaceRelease3.0Notes-TimelineandProcessing
[20:02] <tdonohue> Does anyone have any comments on that new schedule? (will pause for a brief moment)
[20:02] * mdiggory (~mdiggory@rrcs-74-87-47-114.west.biz.rr.com) has joined #duraspace
[20:03] <bollini> happy to see postponed the freeze deadline of 1 week (and also added more details about what this mean - pull request in github)
[20:04] <tdonohue> yep. Also to be clear, the RT decided to "rename" the "freeze" to be a "contribution deadline"...not really a "freeze" anymore...just the deadline to getting your stuff into a pull request
[20:05] <bollini> yes, very useful naming change
[20:05] <mdiggory> Hello everyone.
[20:05] <tdonohue> Ok. Sounds like no more comments on the 3.0 Schedule, so we'll move along for now. If there are suggestions for more changes to the schedule, please email dspace-devel/-release
[20:05] <richardrodgers> do we think this push-out will be sufficient, or more time still needed (we can still deliver in 2012 with further push)?
[20:06] * tdonohue notes the Release Team is more than welcome to chime in on questions :)
[20:06] <mdiggory> I just want to correct in my last email, I did mistake that the contrib deadline was set to next friday, not this friday
[20:07] <mdiggory> so strike my point (1) to push it our further ATM.
[20:07] <tdonohue> richardrodgers: I think the schedule is always a bit of a guess and is even more so this time around with our move to GitHub Pull Requests. But, we do still have flexibility in the Schedule. We've only ever announced that 3.0 will be out "sometime in Oct/Nov". Currently we've pushed back to Oct 26
[20:07] <robint> Just out of interest, does anyone present have any contributions still without a pull request ?
[20:08] <bollini> yes
[20:08] <richardrodgers> tdonohue: fair point, but my concern was with potential submitters, as robint is just asking….
[20:08] <sands> tdonohue, +1 it is not hard and fast
[20:08] <mdiggory> tdonohue / richardrodgers That would seem to leave some room for consideration
[20:08] <helix84> yes, a minor rewrite of LDAP auth
[20:09] <bollini> SOLR DAO (I will make a pull request probably tomorrow), Pubmed and Statistics (not sure to be able to contribute with the current deadline)
[20:09] <helix84> and also DS-1193
[20:09] <kompewter> [ https://jira.duraspace.org/browse/DS-1193 ] - [#DS-1193] display bitstream read access permissions on item page - DuraSpace JIRA
[20:09] <mdiggory> @Mire --> Statistics, Configurable Workflow and Item Level Versioning contribs
[20:10] <robint> Quite a lot then, on the positive side its going to be great release !
[20:10] <tdonohue> So, it sounds like we still do have quite a few pull requests that are still coming :)
[20:10] * mhwood was hoping for some dspace-devel discussion of harmonizing Pubmed, Total-Impact etc. altmetrics work.
[20:10] <helix84> yep, everyone is on a crunch :)
[20:10] <hpottinger> robint: that's the spirit!
[20:10] <tdonohue> +1 robint -- may shape up to be one of the biggest releases in terms of features
[20:11] <PeterDietz> Taking a quick look at our local customizations, and see if there are any other goodies worth pushing upstream. We've got things like facebook opengraph metadata, rss feed improvements for itunes (U), and a curation task to get the duration of audio bitstream.
[20:11] <tdonohue> Ok. So, that all being said, it sounded like (via email discussions) that we wanted to call several *votes* today on some of the outstanding pull requests
[20:11] <mdiggory> tdonohue: that was my worry addressed in the proposal, its clear that there are a great deal of motivated parties involved with contributing to the release, the more significant issue is getting the review process to be efficient enough to get things process and moving forward.
[20:12] <tdonohue> Shall I go ahead an call those votes? (I know of votes requested on @mire pull requests and on XOAI...I can call them one by one, if we are ready)
[20:12] <bollini> mhwood: yes, I will like to work with other on these topics but I will need that my colleague come back from holiday to be more reactive (27th August, so just after feature submission deadline)
[20:12] <helix84> I'd like to call a vote on putting XOAI into 3.0 (hitting the merge button). Of course, if there are any comments, there is still time to address them before release, this is just about getting this feature in.
[20:12] <mdiggory> I think its important to clarify what the nature of the vote is initially.
[20:12] <helix84> i'll start with my own +1 to XOAI
[20:12] <bollini> +1 to XOAI
[20:13] <mdiggory> +1 to XOAI
[20:13] * mdiggory shuts up and gets started
[20:13] <sands> +1 to XOAI
[20:13] <mhwood> +1 XOAI
[20:13] <hpottinger> +1 XOAI
[20:13] <richardrodgers> +1 to XOAI
[20:14] <robint> +1 XOAI
[20:14] <tdonohue> +1 to XOAI being accepted to 3.0 (note the language here -- I think we may need to call a second vote as to whether it *Replaces* OAI-1.0 completely)
[20:14] <helix84> tdonohue: right
[20:14] <mdiggory> So what I wanted to get clarity on is that we are voting to accept a pull request for a feature.
[20:15] <mhwood> That was my understanding.
[20:15] <tdonohue> OK. XOAI final vote: +9 to accept for 3.0. No votes against.
[20:15] <robint> tdonohue: well noted as I think the pul request may replace the existing dspace-oai module ?
[20:15] <sands> if i understand things correctly, the OAI 1.0 module is completely decoupled, so I would be +1 for releasing it separately. (if we're going to vote on that.)
[20:15] <helix84> If I may be so bold to call the second vote of XOAI replacing OAI (the current pull request has this form). There is still the option of releasing the old oai separately,they can be even stalled side-by-side, but this vote isn't about that.
[20:15] <mdiggory> robint: likewise... This pull request cannot be automatically merged.
[20:15] <helix84> https://github.com/DSpace/DSpace/pull/64
[20:15] <kompewter> [ Pull Request #64: OAI 2.0 (a.k.a XOAI) integrated into DSpace 3.0 by lyncodev · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/pull/64
[20:16] <mdiggory> meaning theres corrections that are needed in the branch before it can be pulled
[20:16] <helix84> this is the new pull request - squashed
[20:16] <helix84> s/stalled/installed/
[20:16] <kompewter> helix84 meant to say: If I may be so bold to call the second vote of XOAI replacing OAI (the current pull request has this form). There is still the option of releasing the old oai separately,they can be even installed side-by-side, but this vote isn't about that.
[20:16] <mdiggory> helix84: still some conflict blocking auotmatically merging it
[20:16] <bollini> I think that we should try to move ahead! replace oai 1.0 and if problem or other arise we can decide to release a oai 1.0 update after the 3.0
[20:16] <PeterDietz> but it looks like something happened in master that makes these unable to auto-magically merge in. Mine broke too.
[20:17] <mdiggory> contributions happened.
[20:17] <helix84> mdiggory: I see. I'll see to that it gets addressed.
[20:18] <tdonohue> PLEASE VOTE: Would you agree to XOAI completely replacing OAI-1.0 in 3.0? This would mean that the old "oai" module would be entirely removed from "master/trunk". (As mentioned, we could move it to a different project if folks still wanted to use it instead of XOAI.)
[20:18] <mdiggory> after reviewing a number of the smaller requests from others that were not significant in nature (bug fixes and minor api changes) that could be accepted, those PR were accepted
[20:18] <mdiggory> Yes +1
[20:18] <helix84> +1 to replacing
[20:18] <robint> Might be best to decide what we do with old OAI before merging
[20:18] <mhwood> How hard will replacement be on people who are running the old one?
[20:18] <bollini> +1 to replacing
[20:18] <mdiggory> XOAI uses the original OAI code for its default
[20:19] <helix84> mdiggory: i'mnot sure what you mean but i don't think you're right
[20:19] <sands> +1 to replacing, as long as the original OAI module is made easily accessible for those who still want to run it.
[20:19] <mdiggory> Ok perhaps I misunderstood lyncode
[20:19] <hpottinger> +1 replace, as long as it's documented
[20:19] <robint> +1 to replacing
[20:19] <helix84> hpottinger: noted
[20:19] <mdiggory> but wasn't there an iteration where the old oai code was added and the default was to use it?
[20:20] <PeterDietz> +1 to replacing. Is it default to running via DB or via solr?
[20:20] <tdonohue> To clarify -- XOAI does NOT use the original OAI-1.0 code. However, XOAI can be configured to use DB queries like OAI-1.0 (By default XOAI uses Solr though)
[20:20] <richardrodgers> +1 to sands point - since OAI can be locally extended, we don't want to strand people
[20:20] <helix84> PeterDietz: default is solr, DB means changing a single option
[20:20] <mdiggory> does it use those db queries in its default configuration?
[20:21] <bollini> we should talk about SOLR as default in general...
[20:21] <tdonohue> +1 to replacing -- agree with sands (OAI-1.0 needs to be made still accessible for folks wanting it) and hpottinger (XOAI needs to be documented)
[20:22] <tdonohue> ok. Looks like the vote for XOAI to replace OAI-1.0 is : +9 in favor. None opposed
[20:22] <tdonohue> (However, we need to be sure to move current OAI-1.0 to a separate project for folks that need it, and XOAI needs documentation)
[20:22] <bollini> please note that imho SOLR/DB default, where put the old OAI module etc. are questions that come after hit the merge button
[20:22] <mdiggory> tdonohue: how can OAI be provided and replaced at the same time?
[20:22] <mdiggory> OAI 1.0 that is
[20:23] <robint> tdonohue: might be easiest to do that before the merge of the new one
[20:23] <helix84> so, to explain what OAI 2.0 is: it's written from scratch. OAI 2.0 is the DSpace data provider which uses the XOAI library (external maved dependency, bsd-licensed, developed by lyncode) to do the heavy lifting (similiar to the way the old oai webapp used OAIcat)
[20:23] <tdonohue> mdiggory -- move to code to a new 'dspace-oai-1.0' GitHub project *before* it gets replaced
[20:23] <helix84> s/maved/maven/
[20:23] <kompewter> helix84 meant to say: so, to explain what OAI 2.0 is: it's written from scratch. OAI 2.0 is the DSpace data provider which uses the XOAI library (external maven dependency, bsd-licensed, developed by lyncode) to do the heavy lifting (similiar to the way the old oai webapp used OAIcat)
[20:24] <tdonohue> So, we have 3 actions coming out of this vote: (1) Someone needs to move current OAI-1.0 code to a new GitHub project (for folks who need it), (2) XOAI pull request gets committed (replacing OAI-1.0), and (3) XOAI needs to be documented.
[20:24] <mdiggory> some questions may arise as to if we want to see that library be in the dspace codebase or not
[20:24] <hpottinger> helix84 has the start for a wiki page right there. ;-)
[20:24] <helix84> mdiggory: today i have trouble understanding your questions
[20:25] <mhwood> Can it not get into Maven Central?
[20:25] <helix84> actually, there is https://wiki.duraspace.org/display/DSPACE/OAI%202.0
[20:25] <kompewter> [ OAI 2.0 - DSpace - DuraSpace Wiki ] - https://wiki.duraspace.org/display/DSPACE/OAI%202.0
[20:25] <helix84> mhwood: it's already in maven central
[20:26] <mdiggory> to clarify, I'm asking do we want to see DSpace OAI be dependent on an external XOAI library, or should the external OAI library be contributed to the codebase as well, so that it can be maintained in unison.
[20:26] <tdonohue> XOAI in Maven Central: http://search.maven.org/#search|ga|1|XOAI
[20:26] <kompewter> [ The Central Repository Search Engine ] - http://search.maven.org/#search|ga|1|XOAI
[20:26] <helix84> mdiggory: it's maintained by lyncode and will be used in other projects as well. it's quite new and developed actively
[20:26] <mdiggory> this isn't about accessability in Maven Central
[20:27] <mdiggory> This is about dependency on externally managed code
[20:27] <robint> mdiggory: good question
[20:27] <bollini> mdiggory: I think that this is not a decision that we can take. It is owned by Lyncode
[20:27] <helix84> note: we didn't control OAICat, either
[20:27] <robint> Some of the Greek code is also based on jars they maintain
[20:28] <tdonohue> OAI-1.0 already depended on externally managed code -- OAICat by OCLC
[20:28] <mdiggory> for instance, we accepted dspace-services originally with dependencies on some librarye that aaron maintained and released, but felt that over time, this wasn't best because we could not maintain those ourselves and correct them directly
[20:28] <helix84> it's BSD licensed, so we could fork it. Not that I have any reason to recomend it.
[20:29] <robint> I think it is reasonable to consider the context of any jar we choose to depend on
[20:29] <helix84> clarification: there's no reason to fork the code, but it's an option if the need arises in future.
[20:29] <bollini> I agree with helix84. Just use it now as is, in future we will decide
[20:30] <tdonohue> I think it is definitely worth discussing any JAR we choose to depend on. I'm just pointing out that both XOAI and OAI-1.0 depended on externally managed dependencies. Ideally though, I agree that we'd hopefully want to manage such code ourselves when it makes sense to.
[20:30] <helix84> spoiler alert: i'm going to call a vote on accepting lyncode as a commiter soon
[20:31] <mhwood> Is there some specific reason to suppose that XOAI will fall into disrepair or evolve away from what we need?
[20:31] <sands> Q: the OAI module is in the 1.8.x branch. We can grab it and manage it somewhere, or point people to that branch, yes? Lots of other discussion to happen before the end of the meeting, what with XOAI being accepted already...
[20:31] <bollini> could lyncode be interested in move their library under the duraspace hat?
[20:31] <mdiggory> to be clear this isn't a judgement on lyncode or helix84, this is about assuring that DSpace depends either on stable, available, well maintained libraries and that if we want to develop the XOAI further, we are not bound to getting approval from a third party to make those changes
[20:31] <helix84> mhwood: on the contrary, it's in lyncode's direct interest to keep it working great with dspace
[20:32] <mdiggory> because the code is under the ownership of tha third party
[20:32] <mdiggory> this is a dialog about copyright
[20:32] <helix84> mdiggory: actively maintained: check; bsd-licensed = no approval: check
[20:32] <helix84> actually apach-licensed, sorry for the mistake
[20:33] * aschweer (~schweer@schweer.its.waikato.ac.nz) has joined #duraspace
[20:33] <robint> hello aschweer
[20:33] <mdiggory> but under copyright of lyncode, we cannot override classes and put them into dspace github without changing the copyright to duraspace?
[20:33] <aschweer> good morning, sorry I'm late
[20:34] <mhwood> I still don't see why this situation is different from, say, Log4J
[20:34] <tdonohue> I personally think XOAI has a better chance of "not falling into disrepair" than OAICat, just because of the fact that XOAI source code is published on GitHub (& can be forked at will). Though I do think it's worth asking Lyncode if he'd want to submit it all to DuraSpace & get the benefits of broader community support.
[20:35] <mhwood> Ah, code is public: check.
[20:35] <helix84> mdiggory: Copyright 2012 Lyncode
[20:35] <helix84> example (see header): https://github.com/lyncode/xoai/blob/master/src/main/java/com/lyncode/xoai/dataprovider/OAIDataProvider.java
[20:35] <kompewter> [ xoai/src/main/java/com/lyncode/xoai/dataprovider/OAIDataProvider.java at master · lyncode/xoai · GitHub ] - https://github.com/lyncode/xoai/blob/master/src/main/java/com/lyncode/xoai/dataprovider/OAIDataProvider.java
[20:36] <PeterDietz> I'm with mhwood on this one. We don't ask solr to move under the duraspace umbrella, though we've made a custom version of their code
[20:36] <helix84> +1 mhw
[20:36] * tdonohue would like to suggest we bring this discussion to either email or to XOAI JIRA ticket itself. Otherwise, I worry we won't get to other votes
[20:36] <mdiggory> Solr is a big project with many dendencies and users
[20:36] <hpottinger> +1 to moving XOAI to DuraSpace, though I don't actually get a vote on that, just consider this me cheerleading for my favorite community :-)
[20:36] <mdiggory> this has one use at this time, dspace xoai
[20:37] <tdonohue> mdiggory: I suggest asking Lyncode that question. I don't think this is a subject we can decide upon today without feedback from him
[20:37] <mhwood> Seconded: move details of XOAI discussion to ML.
[20:37] <helix84> mdiggory: the plan is to make other data providers (e.g. Fedora Commons)
[20:37] <helix84> mdiggory: it's just new, 's all
[20:37] * tdonohue is going to move on to other votes... prepare yourselves!
[20:38] <tdonohue> VOTE: Should we include @mire's Advanced Embargo Code in 3.0: https://github.com/DSpace/DSpace/pull/43 ?
[20:38] <kompewter> [ Pull Request #43: [DS-895] Advanced Embargo Project by mdiggory · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/pull/43
[20:39] <helix84> +1 to advanced embargo
[20:39] <mhwood> +1 #43
[20:39] <hpottinger> +1 Pull 43
[20:39] <kshepherd> +1
[20:39] <bollini> 0 I haven't checked it
[20:40] <robint> +1
[20:40] <aschweer> 0 only just got back from leave, haven't had a chance to look yet
[20:40] <sands> 0, haven't been able to review
[20:40] <tdonohue> +1 on #43 -- I think this is a great example of pulling together feedback from many in the community to make a better feature (see all the feedback on : https://wiki.duraspace.org/display/DSPACE/Advanced+Embargo+Support)
[20:41] <tdonohue> (Though, at the same time, I wonder if this Advanced Embargo work is going to *break* AIP Backup & Restore in some places.)
[20:41] <mhwood> We have some time to heal it.
[20:41] <tdonohue> yes, we do
[20:41] <mdiggory> I'm sorry, minor emergency in the office, I was distracted. Yes, thank you for moving forward on the vote, I'm still for XOAI even with my points/concerns
[20:41] <helix84> tdonohue: i think we have the testing period to figure that out. And even if we won't make it, 3.1, 3.2 to fix that...
[20:42] <hpottinger> +1 tdonohue, there's tons of community support for Pull 43, I think it's worth the merging effort
[20:42] <mdiggory> tdonohue: we are working to assure that it does not break OAI
[20:42] <mdiggory> sorry... AIP
[20:42] <bollini> mdiggory: we have simulated the emergency in your office to move ahead
[20:42] <helix84> :D
[20:42] <mdiggory> brilliant
[20:43] <kshepherd> ok my bus is arriving at its stop now, gotta go, please randomly assign my votes for the rest ;)
[20:43] <tdonohue> getting your signals crossed, mdiggory :) Good to heard that AIP stuff should still work
[20:43] <tdonohue> kshepherd votes +1 on assigning all tickets to kshepherd
[20:43] <tdonohue> :)
[20:43] <mdiggory> the last bit of work was to assure that use of the legacy metadata fields (terms and lift date) are still functional even when using the Advanced Features
[20:43] <mhwood> float rand0() {return 1.0;}
[20:44] <helix84> side question: are we expecting DCAT to show up today?
[20:44] <tdonohue> Ok, looks like the vote on Pull #43 (Advanced Embargo) passes: +6 in favor, 3 obstained, 0 against
[20:44] <mdiggory> In this manner, SWORD, ItemImport, LNI and AIP should still be able to accept items with embargo terms in metadata
[20:45] <tdonohue> helix84: Not to my knowledge. I don't think DCAT will be attending today. But, I think they plan to attend the week of Sept 5 (after their next mtg)
[20:45] <mdiggory> excellent, thank you
[20:45] <tdonohue> VOTE: Should we include @mire's Discovery enhancements (for XMLUI) in 3.0: https://github.com/DSpace/DSpace/pull/45 ?
[20:45] <kompewter> [ Pull Request #45: [DS-1229] @mire discovery contribution by KevinVdV · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/pull/45
[20:46] <bollini> +1
[20:46] <helix84> +1 hell yes
[20:46] <mhwood> +1 #45
[20:46] <richardrodgers> haven't reviewed, but seems like a clear win +1
[20:47] <sands> +1
[20:47] <robint> +1
[20:47] <tdonohue> +1 #45 -- though I do want to stress we may need to document Discovery differences between XMLUI and JSPUI (assuming the JSPUI stuff gets in too)
[20:47] <aschweer> +1 based on what Kevin told me about it at OR
[20:47] <sands> +1 to that tdonohue
[20:47] <PeterDietz> +1 sounds like some interesting enhancements
[20:47] <hpottinger> +1 Pull 45
[20:48] <bollini> tdonohue: I don't think that we will have much difference (hopefully none - also making additional change to DS-1217 after to have accepted it)
[20:48] <kompewter> [ https://jira.duraspace.org/browse/DS-1217 ] - [#DS-1217] DSpace Discovery for JSPUI - DuraSpace JIRA
[20:48] <mdiggory> +0 - I'm just abstaining based on my affiliation, but very much support the contribution in #45.
[20:49] <tdonohue> Ok, vote on Discovery Enhancements (#45) passes: +10 in favor, 1 abstain, 0 against. We need documentation for these new features (obviously) and may need to document any differences between XMLUI & JSPUI Discovery (if any)
[20:49] <mdiggory> agreed
[20:49] <tdonohue> Ok, anything else that is ready for an official vote?
[20:49] <mhwood> https://github.com/DSpace/DSpace/pull/41
[20:49] <kompewter> [ Pull Request #41: [COMPLETE] [DS-861] Salt PasswordAuthentication by mwoodiupui · DSpace/DSpace · GitHub ] - https://github.com/DSpace/DSpace/pull/41
[20:49] <mdiggory> mhwood: you beat me too it
[20:50] <tdonohue> +1 to #41 , especially since the update is "invisible" to end users (on their next login, their password is re-hashed). Great work, mhwood!
[20:50] <mdiggory> +1 on #41
[20:50] <mhwood> Thanks, I received some good suggestions.
[20:50] <bollini> +1
[20:51] <helix84> +1 #41 (great work, indeed)
[20:51] <sands> +1
[20:51] <aschweer> +1 - haven't reviewed, but looks like there was some great discussion going on in the pull request
[20:51] <hpottinger> +1 Pull 41
[20:52] <robint> +1
[20:52] <sands> Side-note: Documentation due date: August 31st.
[20:52] <robint> Does this mean I can no longer insert cuta and pasted passwords ??
[20:52] <tdonohue> (Additional Docs sidenote -- please put your docs over in the 3.0 area, preferrably: https://wiki.duraspace.org/display/DSDOC3x/DSpace+3.x+Documentation)
[20:53] <helix84> (I just remembered I have one more feature in pipeline for 3.0 without pull request at this time - DS-1084)
[20:53] <kompewter> [ https://jira.duraspace.org/browse/DS-1084 ] - [#DS-1084] Handle authority and confidence fields in Bulk Editing - DuraSpace JIRA
[20:53] <tdonohue> Ok, vote on Salt PasswordAuthN (#41) passes: +8 in favor, 0 against
[20:53] <tdonohue> Anything else ready to be voted on today? Speak up now :)
[20:53] <hpottinger> I haven't had a chance to review, but PeterDietz made a pull request for DS-1241 (Pull #63 https://github.com/DSpace/DSpace/pull/63) earlier this week, wanna vote on it?
[20:53] <kompewter> [ https://jira.duraspace.org/browse/DS-1241 ] - [#DS-1241] Statistics implementation in Elastic Search - DuraSpace JIRA
[20:53] <mhwood> Hash + salt + algorithm should be copyable.
[20:54] <PeterDietz> btw mhwood, thats a great looking pull request with continuous improvements based on feedback.
[20:54] <helix84> i'm calling a vote on free ice cream for everyone!
[20:54] <PeterDietz> Git / GitHub doesn't really help facilitate distributed ice cream
[20:54] <tdonohue> although helix84's vote is tempting, hpottinger has called a vote on #63
[20:54] <sands> helix84 No ice cream for anyone until the 3.0 release!
[20:54] <sands> ;)
[20:55] <PeterDietz> The elastic search code is good as-is. There is some idea, that I'd like to make it easier to test by adding a import-stats-data-to-ES launcher
[20:55] <richardrodgers> Are you running in production PeterDietz ?
[20:55] <mdiggory> I only want to verify if it impacts any of the existing statistics api
[20:55] <sands> 0 - have not looked at it yet
[20:56] <helix84> PeterDietz: would that importer be in 3.0? 3.1?
[20:56] <PeterDietz> we run elastic search in production, no problems to report
[20:56] <bollini> 0 - have not looked at it yet
[20:56] <aschweer> 0 - haven't looked yet and I'm wondering about including in 'official' DSpace vs some sort of add-on status
[20:57] <tdonohue> Is there a stats-importer for folks wanting to upgrade/switch from Solr? Or do you have to start your stats fresh?
[20:57] <PeterDietz> Thus far, I've only built a dspace.log -> ElasticSearch tool.
[20:57] <PeterDietz> uncommitted
[20:58] <mdiggory> PeterDietz: I notice updates in StatisticsTransformer, do these impact the existing Solr Stats behavior at all?
[20:59] <PeterDietz> I initially wanted to make this an add-on module, but ran into several difficult questions if it would be a module
[20:59] <tdonohue> ok. I'd suggest we likely need some form of importer (minimally from dspace.log). Otherwise, it seems like it'd only be useful for new DSpace installs (or folks willing to toss out their existing stats)
[21:00] <PeterDietz> it only enhances StatsTransformer. adds a new constructor to pass in your date limit (start, end)
[21:00] <mdiggory> I'd prefer to keep a common interface / services for statistics for DSpace with common log processing, but with different backends and views
[21:00] <PeterDietz> ...and some additional growth-statistics data (DB driven), that go with the stats view
[21:00] <robint> mdiggory +1
[21:01] * tdonohue notes we've hit the one hour mark
[21:01] <bollini> the UI improvements (dashboard) work also with the SOLR provider?
[21:01] <PeterDietz> I agree with making a stats interface. I started that work, but you make tons of bad trade-offs forcing the data to fit into a new container.
[21:01] <tdonohue> good question from bollini... I think the UI improvements (screenshots) are AWESOME :)
[21:01] <PeterDietz> ES data is not the same as SOLR response data
[21:02] <hpottinger> I'm a +1 for Pull 63, contingent on either a tool for importing Solr stats, or documentation on a manual conversion process.... I've got a number of users chomping at the bit for stats UI improvements, this will keep my people happy
[21:02] <mdiggory> PeterDietz: can you elaborate?
[21:03] <helix84> i don't think the vote on ES has been called yet
[21:03] <mdiggory> TBH, our stats contribution should be reviewed in concert with Pull #63, there may be needed features there...
[21:04] <mhwood> Can the UI improvements be easily pulled out?
[21:05] <aschweer> (I need to go to work -- bye all)
[21:05] <PeterDietz> UI improvements do not work with solr. I'm having ES convert the output data to JSON, and giving that to the view, which uses JS to convert/massage into pretty charts
[21:05] * aschweer (~schweer@schweer.its.waikato.ac.nz) Quit (Quit: leaving)
[21:06] * sands needs to leave on time today. bye all!
[21:06] * sands (~sandsfish@18.189.31.239) Quit (Quit: sands)
[21:06] <PeterDietz> If one could get solr to output to same JSON, than the google charts / graphs would work for both
[21:06] <richardrodgers> I have to run also - bye all
[21:06] * richardrodgers (~richardro@pool-173-48-255-197.bstnma.fios.verizon.net) Quit (Quit: richardrodgers)
[21:07] <bollini> mmm... I have a general concern about reusing the JSON output directly form a search engine as is
[21:07] <tdonohue> Ok, sounds like folks needing to leave here...so, we may just have to leave it at that. It seems like most folks still have questions about Pull #63.
[21:07] <bollini> I want to suggest to introduce a common JSON object that we can fill with data from different search engine
[21:07] <bollini> BTW SOLR can produce JSON (it is used by the XMLUI autocomplete feature)
[21:08] <helix84> PeterDietz: solr already has json as an output format, just the contents is probably not what the same. The question is if the solr response has everything you need.
[21:08] <PeterDietz> mmm. I doubt that the response of ES and the response of SOLR would be identical.
[21:08] <bollini> in the JSON porting we have simply serialized in JSON the DSpace Discovery "model" (FacetValue)
[21:08] <mdiggory> PeterDietz: I think we should continue a dialog with KevinVdV and assure that we identify features being brought to the table in the stats contrib and look for alignment. Yes Solr can both produce json and we can customize Solr to produce different outputs if neccessary.
[21:09] <helix84> PeterDietz: likely not in format, byt what about the data?
[21:09] <bollini> sorry s/JSON porting/JSPUI porting/
[21:09] <mdiggory> you still didn't answer my question PeterDietz is the data stored in a similar format in ES?
[21:09] <tdonohue> mdiggory +1 -- I like the idea of continuing this discussion with KevinVdV to see if there's better alignment with Solr Stats possible
[21:09] <mdiggory> meaning 1 ES record = 1 DSpace event
[21:10] <mdiggory> tdonohue: thanks
[21:10] <tdonohue> With better "alignment" between the two statistics backends (Solr or ES), I think this would make a good feature. I just worry right now that the current ES work is almost a new stats system -- I think we need them to stay aligned
[21:11] <PeterDietz> The schema / mapping is very similar
[21:11] <tdonohue> (Or, the other option is to make this a separate module altogether, if there is no way to keep them "aligned")
[21:11] <PeterDietz> i.e. lat/long, type, id, time, I started from solr.
[21:13] <PeterDietz> I was thinking I didn't want to build a concrete interface that wraps ES and SOLR data, when I could still be evolving the output of ES
[21:13] <tdonohue> by "alignment", I also mean aligned behind the same UI. That's the area I worry about. It could get hard to support two different stats UIs in all of the various XMLUI Themes, etc.
[21:13] <mdiggory> PeterDietz: good, thanks
[21:13] <bollini> I still think that we should try to introduce abstraction also in the format sent to the browser (JSON not tired to a search engine but derived from a DSpace "model").
[21:14] * tdonohue notes this is just my initial opinion -- I'm willing to be swayed/overruled & I fully admit that I haven't had a chance to do a thorough review of everything in the ES work yet.
[21:15] <mhwood> Sorry, I have to go. Lots of good things today!
[21:15] <tdonohue> (but that UI work that PeterDietz has done is AWESOME -- sorry, had to say it again)
[21:15] <PeterDietz> So some interesting things to ask from your statistics are number of bitstream downloads within a Collection, broken down over month
[21:15] * mhwood (mwood@mhw.ulib.iupui.edu) has left #duraspace
[21:16] <PeterDietz> Which countries are your visitors coming from.
[21:16] * tdonohue notes that if others have to leave, feel free. We won't be able to call any more votes today, but you are welcome to hang around for discussion. If there's something you want to call a vote on, please feel free to do so on dspace-devel!
[21:16] * robint (52292725@gateway/web/freenode/ip.82.41.39.37) Quit (Quit: Page closed)
[21:16] <PeterDietz> Top downloaded bitstreams in this collection during this time-period.
[21:16] <hpottinger> PeterDietz: yes, very much interest from my folks in doing just that (downloads per community/collection)
[21:16] <bollini> this looks very similar to the work that we have done for the HKU and that we initially plan to contribute for 3.0
[21:17] <PeterDietz> So, different queries can respond with different output. The trouble of forcing this to go through some type of clearinghouse (abstract model), is that creating a new query probably means creating another method in your model
[21:18] <bollini> I think that we could have a chance to review this work and make some alignments (JSPUI/XMLUI SOLR/ES) after the 27th August
[21:19] <tdonohue> PeterDietz -- but if all you're doing is pulling out JSON (from ES or Solr), couldn't their just be a generic JSON which most query results could go into? It's not like you're gathering completely different results from new queries? (Or am I misunderstanding your point?)
[21:19] <tdonohue> s/their/there/
[21:19] <kompewter> tdonohue meant to say: PeterDietz -- but if all you're doing is pulling out JSON (from ES or Solr), couldn't there just be a generic JSON which most query results could go into? It's not like you're gathering completely different results from new queries? (Or am I misunderstanding your point?)
[21:19] <bollini> PeterDietz: I think that it is natural that the abstract model can evolve overtime but having a common layer will help to keep things in sync (ES / SOLR)
[21:20] <helix84> tdonohue: my thinking exactly
[21:22] <PeterDietz> "top_countries" : {
[21:22] <PeterDietz> "_type" : "terms",
[21:22] <PeterDietz> "missing" : 234,
[21:22] <PeterDietz> "total" : 1776,
[21:22] <PeterDietz> "other" : 0,
[21:22] <PeterDietz> "terms" : [ {
[21:22] <PeterDietz> "term" : "United States",
[21:22] <PeterDietz> "count" : 900
[21:22] <PeterDietz> }, {
[21:22] <PeterDietz> "term" : "United Kingdom",
[21:22] <PeterDietz> "count" : 415
[21:22] <PeterDietz> }, {
[21:22] <PeterDietz> "term" : "France",
[21:22] <PeterDietz> "count" : 72
[21:22] <PeterDietz> }, {
[21:22] <PeterDietz> "term" : "Spain",
[21:22] <PeterDietz> "count" : 69
[21:22] <PeterDietz> }, {
[21:22] <PeterDietz> "term" : "Canada",
[21:22] <PeterDietz> "count" : 43
[21:22] <PeterDietz> }
[21:23] <PeterDietz> So the JSON can be the common ground, its very clean, nothing specific. The UI should be able to be build against this common data
[21:24] <tdonohue> Right...if ES & Solr could be setup to export a common JSON structure, it could be used to then drive the cool UI stuff (like you've already done with your work)
[21:25] <PeterDietz> I am unaware of anyone working on dspace SOLR stats, so its an extra step to make this change also touch SOLR in the same move as contributing ES
[21:26] <PeterDietz> I would guess that anything engine1 can do, so can engine2, its just a matter of if you want to get around to implementing it. So you could implement a query such as visitor's from *.edu hostnames in Elastic Search, then add a data-graphic to your view.
[21:26] <tdonohue> PeterDietz -- KevinVdV & mdiggory are good contacts on the SOLR stats part. As mdiggory said earlier, you could work with them to see if there's a way to align the ES stuff with Solr stuff (and that way you only need to do the ES part of it)
[21:26] <helix84> peter, my thinking here is: is this similiar enough to the (JSON) results format Solr already provides so that it can be shared? Solr response format is mature and well-defined, so I'd suggest to adopt it if you can. OTOH, if your intermediate formats are still developing, which is a good thing, maybe it's just not ready to be distributed in DSpace (again, not necessarily a bad thing, because it gives you freedom to break stuff to make it
[21:26] <helix84> better). This is all just my thinking out loud, not saying that I'm in any way opposed to anything about ES.
[21:26] <PeterDietz> If someone comes along and also implements that for SOLR, then great, it shouldn't hold up the show.
[21:30] <helix84> and peter, i know how it feels to work on something and want to push it upstream to a wide audience, but the consequence of that is that you'll have to keep it stable (or proveide an upgradepath) once it adopts users
[21:30] <bollini> I renew my offer to take a look for JSPUI/SOLR, but not starting before 27th August (this is in conflict with current deadline)
[21:31] <hpottinger> It would be nice to see if the interested parties for supporting Solr-based stats could collaborate with PeterDietz and align the interface improvements, if the only way to get the shiny is to go with ES-based stats, I know how my users will want me to vote. :-)
[21:32] <helix84> disclaimer: stats is not currently my area of interest, just giving peter some food for thought :)
[21:32] <bollini> sorry to use a private chat message: just want to check that my messages are received
[21:33] <tdonohue> +1 hpottinger. That's also why I'd like to see Peter's great work also get in front of KevinVdV (who's been doing a lot of the Solr Stats work). I know this UI stuff is something that users will *WANT*
[21:33] <PeterDietz> Here is solr's facet output. Different facet query.
[21:33] <PeterDietz> {
[21:33] <PeterDietz> facet_queries: { },
[21:33] <PeterDietz> facet_fields: {
[21:33] <PeterDietz> id: [
[21:33] <PeterDietz> "2481",
[21:33] <PeterDietz> 294,
[21:33] <PeterDietz> "285",
[21:33] <PeterDietz> 219,
[21:33] <PeterDietz> "2480",
[21:33] <PeterDietz> 210,
[21:33] <PeterDietz> "60656",
[21:33] <PeterDietz> 77,
[21:33] <PeterDietz> its gross
[21:33] <tdonohue> bollini -- yes, your messages are coming across. I'm not sure I understand what you meant by "I renew my offer to take a look for JSPUI/Solr"
[21:34] <bollini> I can take a look to the PeterDietz work and I can check if it is possible to improve/implement it in SOLR (implementing also the JSPUI part)
[21:35] <bollini> ... but starting from 27th August
[21:36] * ryscher (98033b9f@gateway/web/freenode/ip.152.3.59.159) Quit (Quit: Page closed)
[21:37] <tdonohue> bollini -- ok, then I think you'd want to see if the Release Team would be OK with that (I'd suggest sending an email, since I think most have left for today). Currently, their schedule is set up so that new feature code must be in a Pull Request by Aug. 24, so that it can be reviewed before Sept 7.
[21:38] <bollini> tdonohue -- ok, I will write to the release list
[21:40] <tdonohue> PeterDietz -- from my understanding with Solr, you're only going to get out as "pretty" of data as you put in. So, that data my only look gross cause it wasn't really indexed initially in a "human understandable" way. It could be a matter of just reindexing the data in a different way to get a different looking output
[21:41] <PeterDietz> Also, one fundamental difference between solr and ES, is that for each facet in Solr, you have to make another query, which gives its response in json. ES can do a massive query that has multiple different distinct facets inside of it. Thus, your Javascript that has to parse the json whichever ways its coming out
[21:41] <PeterDietz> the "gross" i meanted a bit ago was this difference:
[21:41] <PeterDietz> "terms" : [ {
[21:41] <PeterDietz> 17:22 PeterDietz: "term" : "United States",
[21:41] <PeterDietz> 17:22 PeterDietz: "count" : 900
[21:41] <PeterDietz> 17:22 PeterDietz: },
[21:42] <PeterDietz> vs
[21:42] <PeterDietz> facet_fields: {
[21:42] <PeterDietz> id: [
[21:42] <PeterDietz> 2481",
[21:42] <PeterDietz> 294,
[21:42] <PeterDietz> "285",
[21:42] <PeterDietz> 219,
[21:42] <PeterDietz> term and count, vs string/int/string/int
[21:44] <PeterDietz> I'm not adding any additional indexing / schema properties to Elastic Search. Its also allowing me to facet upon any field I want
[21:44] <tdonohue> PeterDietz -- I'm just pointing out that Solr actually has many different options for Indexing. Here -- take a look at the sample JSON response format on the Solr Wiki -- it's much more readable: http://wiki.apache.org/solr/SolJSON
[21:44] <kompewter> [ SolJSON - Solr Wiki ] - http://wiki.apache.org/solr/SolJSON
[21:45] <PeterDietz> I made a request like:
[21:45] <PeterDietz> http://localhost:8080/solr/statistics/select?q=*:*%20AND%20type:0%20AND%20-isBot:true%20AND%20owningComm:20&facet=true&facet.field=id&wt=json
[21:45] <helix84> gotta go, bye
[21:45] * helix84 (~a@kou-street209-48.pks.muni.cz) Quit (Quit: helix84)
[21:45] <PeterDietz> yep. Sorry the ES show/tell isn't going smooth.
[21:46] <PeterDietz> I'm not sure what my take-away action is, that can be done before the cut-off. Work with Kevin to determine a common data format for SOLR and ES to share.
[21:46] <bollini> I'm confident that we can get the same response starting from data read from SOLR
[21:46] <tdonohue> Right -- PeterDietz. I'm not arguing for or against ES vs. Solr. As far as I can tell, they are actually both quite similar. All I'm pointing out is that the "ugly" Solr output you are seeing is happening because of how DSpace is *indexing* things in Solr. So, we can always change how we index the data in Solr to get a better output.
[21:47] <bollini> please note that I have say "data read" from SOLR and NOT the same response from SOLR
[21:47] <PeterDietz> I was thinking for output, you were talking about html, css, chart libraries
[21:48] <bollini> tdonohue: no, the standard json format from SOLR is different than the ES format (shown by Peter). It is not related to the stored data
[21:49] <bollini> tdonohue: this is why I'm asking to define a dspace format (that eventually can start from the current ES format) and support it for the future
[21:50] <tdonohue> PeterDietz: I think the next step is to chat with Kevin & perhaps bring discussions to the dspace-release/dspace-devel list. You can always ask for an "extension".
[21:50] <bollini> we are not tired to the JSON SOLR response to supply JSON data to the browser... we can query SOLR, manipulate the response and build a JSON response
[21:51] <PeterDietz> Ahh, okay good. Since ES and SOLR both natively output their data into some serialized form of JSON, but they're different formats.
[21:51] <tdonohue> unfortunately I need to head out now...will catch up more later. bollini, if you have suggestions on ways to get Solr to support better JSON, definitely feel free to let us all know (email is probably best at this point) -- If we can find ways to get this all merged in better it'd be great
[21:52] <PeterDietz> So, maybe there is a DSpaceStatsJSON object, and I do DSpaceStatsJSON.parseES(es_response), and also DSpaceStatsJSON.parseSOLR(solr_response)
[21:52] * tdonohue (~tdonohue@c-67-177-108-221.hsd1.il.comcast.net) Quit (Read error: Connection reset by peer)
[21:54] <bollini> no, I'm talking more about the opposite side
[21:54] <bollini> we need an object (javabean) DSpaceStats that can be serialized in a standard way in JSON
[21:55] <hpottinger> they're kicking us all out of the library again, I've gotta run, will check back in on the transcript later tonight
[21:55] * hpottinger waves
[21:55] <PeterDietz> bye hardy
[21:55] * hpottinger (~hpottinge@mu-162198.dhcp.missouri.edu) has left #duraspace
[21:55] <bollini> our API should only return the DSpaceStats object, serialization can be provided by an utility class (common to JSPUI, XMLUI, SOLR and ES implementation)
[21:56] <PeterDietz> I was at this point a couple of months ago, thinking of the DSpaceStats common object, I can't visualize it very well
[21:57] <bollini> so the browser layer (javascript) only need to know about the structure of the DSpaceStats object (as JSON object)
[21:57] <bollini> sorry to have used DSpaceStats, I mean a *new* object that we have to define
[21:58] <PeterDietz> right, some new not-yet-existing class to be the common object
[21:59] <bollini> I think that we can start using the actual format returned by ES, I want only be sure that our API define a return object with a know structure and not a "string" where the contract about structure is keep on the UI layer (javascript)i
[22:00] <bollini> are you working on a porting of the discovery search stuff to ES?
[22:02] <PeterDietz> Our librarians don't want to use discovery, so solr-discovery is unpopulated on production. Thus we haven't pushed towards moving that to ES
[22:02] <PeterDietz> So.. facets are complicated. I think that is the type of data we want returned by statistics.
[22:03] <PeterDietz> The alternative to a facet is raw data. Which would only be useful if you want to display the raw data of the last 10 visits to this collection
[22:03] <PeterDietz> ES supports various TYPES of facets. Terms, Range, Histogram, Date-Histogram, Filter, Query. Statistical. Terms-Stats, Geo-Distance
[22:04] <PeterDietz> Terms->{term: USA, count: 900, ... }
[22:05] <PeterDietz> "monthly_downloads" : {
[22:05] <PeterDietz> "_type" : "date_histogram",
[22:05] <PeterDietz> "entries" : [ {
[22:05] <PeterDietz> "time" : 1272672000000,
[22:05] <PeterDietz> "count" : 651
[22:05] <PeterDietz> }, {
[22:05] <PeterDietz> "time" : 1275350400000,
[22:05] <PeterDietz> "count" : 183
[22:05] <PeterDietz> }
[22:06] <PeterDietz> They are quite related, and similar in structure. A list of something
[22:07] <PeterDietz> So one question about doing this the proper way, and using the DSpaceStats interface. What does that gain us, when say we migrate to using MongoDB-Stats, and we have to modify the dspace-stats objects structure again to fit the new provider, which might do data differently
[22:08] <bollini> no we need to think about the structure that we need to provide the features that the users ask for
[22:09] <bollini> for your new UI improvements we need an object with two properties? a name and a count, just define it
[22:09] <bollini> to the ES / SOLR / MongoDB providers will be asked to fill this object to support "current" features
[22:10] <bollini> a question: ES response format is only JSON? or there is a "javabin" format?
[22:12] <bollini> in your implementation the browser access to ES or the JSON ES response is streamed to the browser by a "proxy" servlet/trasformer?
[22:13] <PeterDietz> browser does not get the raw response. ES runs at localhost:9200, so DSpace will perform the query, and get a response object back. And pull off a piece of that response, and hide it as a hidden element in the html.
[22:14] <PeterDietz> I could have gone the route of proxying it. But I don't like the idea of the browser making direct calls
[22:14] <PeterDietz> hmm. I'm guessing that ES only outputs JSON...
[22:14] <bollini> agree. I'm against direct call from the browser
[22:15] <PeterDietz> It has its own response processor you could use within java against that response object (if you wanted to do all manipulation from Java)
[22:17] <PeterDietz> TermsFacet countryFacet = resp.getFacets().facet(TermsFacet.class, "top_countries");
[22:17] <PeterDietz> addTermFacetToTable(countryFacet, division, "Country", "Top Country Views (all time)");
[22:17] <bollini> exactly, I think that you should return a new dspacestat java object build using the "java" response from ES, so that we do the same using SOLR and the differences between the JSON format will be not more important
[22:18] <bollini> next, the new dspacestat java object can be serialized in JSON for make the life of the browser layer easier using some external library
[22:19] <bollini> for example: http://flexjson.sourceforge.net/
[22:19] <kompewter> [ JSON Serialization Usage ] - http://flexjson.sourceforge.net/
[22:20] <PeterDietz> I use jquery to convert a value stuffed into a hidden HTML element into javascript/json.
[22:20] <PeterDietz> // Get data from elastic that has been dumped on the page.
[22:20] <PeterDietz> var elasticJSON = $.parseJSON($('#aspect_statisticsElasticSearch_ElasticSearchStatsViewer_field_response').val());
[22:21] <bollini> wait
[22:21] <bollini> I'm talking about introduce a new level of abstraction
[22:22] <PeterDietz> ok sorry, i didn't check the flexjson, and now i see its a Java thing
[22:23] <bollini> if I have understand your architecture now you have XMLUI code - call ES -> get JSON -> embbed JSON in HTML response -> jquery get the json from the html and parse i
[22:23] <bollini> I propose to change in
[22:24] <bollini> XMLUI code - call ES -> get something -> build a JAVA DSpace stat object -> serialize this object in JSON -> embed JSON in HTML response -> jquery...
[22:25] <bollini> this mean that if we want replace ES with SOLR we only need to implement the step 2, 3 and 4 (query the new provider build a common dspace java object)
[22:25] <bollini> the new dspace stat object is a DTO
[22:27] <bollini> I'm sorry it is very late in Italy... I need to go
[22:27] <PeterDietz> ok, thanks for staying up to chat
[22:28] <PeterDietz> i realize its past bed time for you.
[22:29] <bollini> I will write an email to the release list for take ahead this discussion, I'm very interested in have this contribution included in 3.0 and merged with the work that we have done for HKU
[22:29] <PeterDietz> ok, I'm on the same page as you. I'll just need to hash out (i.e. in code / UML) a structure for stats-object that works for both ES data and SOLR data
[22:31] <bollini> good! I go to bed. See you next time. bye
[22:31] <PeterDietz> bye
[22:31] <kompewter> bye!
[22:31] * bollini (~chatzilla@host90-209-dynamic.116-80-r.retail.telecomitalia.it) Quit (Quit: ChatZilla 0.9.88.2 [Firefox 14.0.1/20120713134347])
These logs were automatically created by DuraLogBot on irc.freenode.net using the Java IRC LogBot.