Evaluating Personal Search. An ECIR 2011 [Half Day] Workshop. April 18, 2011. Dublin, Ireland.

Personal Search refers to the process of searching within one’s personal space of digital information, e.g., searching one’s desktop or mobile phone for required data items or information. While some recent advancements have been made in this domain, research acceleration is hindered by the lack of established or standardized baselines and evaluation metrics, and lack of commonly available test collections. There is a clear consensus within the research community of the need for standardized repeatable evaluation techniques in the PS space, perhaps in the form of a TREC track for example. However, there are a number of significant challenges associated with this, not the least of which is the fact that the data associated with this domain is personal to the individual, multimedia in nature, and different users will have different forms of collections, differing information needs and different memories of required information.

This workshop aims to bring together researchers interested in working towards standardized evaluation approaches for the personal search space. Due to the large space that this covers, as a first step towards overall standardized personal search evaluation this workshop will focus on evaluation for the textual elements within personal desktop collections and known item keyword queries for these elements.  An interactive forum for researchers to share ideas and initiate collaborations will be provided, with the explicit goal of establishing a means to evaluate personal search algorithms and solutions in a comparable and repeatable way. It is also intended to form a consortium of people at the workshop interested in bringing this evaluation solution to fruition in the subsequent months.

The topics of the workshop will be evaluation focused and include but not be limited to:

• Understanding personal search evaluation
• Interesting target tasks and explanations of their importance
• Critiques or comparisons of existing evaluation methods
• How we can combine existing evaluation methods
• Improving on previously suggested methods
• Making evaluation  more realistic
• Reducing the cost of evaluation
• Proposal of new evaluation methods
• Using human computation games for evaluation
• User study approaches
• Simulation approaches

Datasets

A number of resources are available prior to the workshop. We expect interested participants to be able to use these resources in various ways -- analyzing the characteristics of the data, evaluating various retrieval methods using the queries and result sets, suggesting a better data set for evaluating personal search, etc. Further information and download instructions are available at http://www.cdvp.dcu.ie/iCLIPS/EPS2011/datasets.html

Paper Submissions

The workshop is now accepting paper submissions. Short position papers (max. 2 pages) describing creative use of / modifications to provided datasets and approaches or ideas / challenges for the domain are invited. Submissions should be in ACM SIGIR format. LaTeX and Word templates are available at http://www.acm.org/sigs/publications/proceedings-templates (for LaTeX, use the "Option 2" style).

Papers should be anonymised and submitted in pdf format through the EasyChair system http://www.easychair.org/conferences/?conf=eps2011 no later than midnight Pacific Daylight Time on February 28, 2011. Submissions will be subject to double-blind reviewing.  Accepted papers will be included in the Evaluating Personal Search Workshop 2011 proceedings.

Important Dates

February 28, 2011: Deadline for paper submission (midnight Pacific Daylight Time)
March 18 , 2011: Notification to authors
March 25, 2011: Camera-ready copy due
April 18, 2011: Workshop

Further Information

Further information is available on the workshop website at http://www.cdvp.dcu.ie/iCLIPS/EPS2011/ or by emailing the workshop organizers.

Workshop Organizers

David Elsweiler – University of Erlangen, Germany (david@elsweiler.co.uk)
Liadh Kelly – Dublin City University, Ireland (lkelly@computing.dcu.ie)
Jinyoung Kim – UMass Amherst, USA (jykim@cs.umass.edu)

Program Committee

Leif Azzopardi – University of Glasgow, UK
Daragh Byrne – Dublin City University, Ireland
Robert Capra – University of North Carolina, USA
Yi Chen – Dublin City University, Ireland
Sergey Chernov – L3S, Germany
Bruce Croft – Univ. of Massachusetts, Amherst, USA
Ronald Fernandez – University of Santiago de Compostela, Spain
Cathal Gurrin – Dublin City University, Ireland
Karl Gyllstrom – Katholieke Universiteit Leuven, Beligium
Donna Harman – NIST, USA
David Hawking – Funnelback, Australia
Sara Javanmardi – Bing, USA
Gareth Jones – Dublin City University, Ireland
Noriko Kando – National Institute of Informatics, Japan
Diane Kelly – University of North Carolina, USA
David Losada – University of Santiago de Compostela, Spain
Ian Ruthven – University of Strathclyde, Glasgow, UK
Alan Smeaton – Dublin City University, Ireland
Jaime Teevan – Microsoft Research, Redmond, USA
Paul Thomas – CSIRO ICT Centre, Australia