#89 ✓resolved
Sandy

Location based search sort doesn't work with subsequent narrowing down with filters

Reported by Sandy | March 10th, 2010 @ 05:57 PM | in Bugs

Using:

near([lat, lng], :sort => true)
With:
with(:some_field).between(start_range .. end_range)
When there are matches with both the near() and with() methods the following error is returned:
Solr Response: nulljavalangNullPointerExceptionat_orgapachelucenespatialtierDistanceFieldComparatorSource$DistanceScoreDocLookupComparatorcopyDistanceFieldComparatorSourcejava105at_orgapachelucenesearchTopFieldCollector$OneComparatorScoringMaxScoreCollectorcollectTopFieldCollectorjava301at_orgapachesolrsearchDocSetDelegateCollectorcollectDocSetHitCollectorjava102at_orgapachelucenesearchIndexSearchersearchWithFilterIndexSearcherjava294at_orgapachelucenesearchIndexSearchersearchIndexSearcherjava254at_orgapachelucenesearchSearchersearchSearcherjava171at_orgapachesolrsearchSolrIndexSearchergetDocListAndSetNCSolrIndexSearcherjava1101at_orgapachesolrsearchSolrIndexSearchergetDocListCSolrIndexSearcherjava880at_orgapachesolrsearchSolrIndexSearchersearchSolrIndexSearcherjava341at_orgapachesolrhandlercomponentQueryComponentprocessQueryComponentjava182at_orgapachesolrhandlercomponentSearchHandlerhandleRequestBodySearchHandlerjava195at_orgapachesolrhandlerRequestHandlerBasehandleRequestRequestHandlerBasejava131at_orgapachesolrcoreSolrCoreexecuteSolrCorejava1316at_orgapachesolrservletSolrDispatchFilterexecuteSolrDispatchFilterjava338at_orgapachesolrservletSolrDispatchFilterdoFilterSolrDispatchFilterjava241at_orgmortbayjettyservletServletHandler$CachedChaindoFilterServletHandlerjava1089at_orgmortbayjettyservletServletHandlerhandleServletHandlerjava365at_orgmortbayjettysecuritySecurityHandlerhandleSecurityHandlerjava216at_orgmortbayjettyservletSessionHandlerhandleSessionHandlerjava181at_orgmortbayjettyhandlerContextHandlerhandleContextHandlerjava712at_orgmortbayjettywebappWebAppContexthandleWebAppContextjava405at_orgmortbayjettyhandlerContextHandlerCollectionhandleContextHandlerCollectionjava211at_orgmortbayjettyhandlerHandlerCollectionhandleHandlerCollectionjava114__at_orgmortbayjetty
When :sort => false the error is resolved, however results are not ordered by location

Comments and changes to this ticket

  • Sandy

    Sandy March 10th, 2010 @ 06:00 PM

    Keyword search is compatible with location based search.

    keywords(params[:q]) do
       highlight :description, {:max_snippets => 5}
    end
    
    near([lat, lng], :sort => true)
    

    Works fine if there are hits for both conditions.

  • mat

    mat March 11th, 2010 @ 10:22 AM

    • Milestone set to Bugs
  • mat

    mat March 11th, 2010 @ 10:34 AM

    I created the following test case, which passes:

    it 'should perform a radial search with attribute scoping and distance sorting' do
      search = Sunspot.search(Post) do |query|
        query.near(ORIGIN, :sort => true)
        query.with(:title, 'teacup')
      end
      search.results.should == [@posts[1], @posts[4]]
    end
    

    The test passes. What version of solr-spatial-light do you have in your lib? There were NPE problems with earlier versions...

    Or maybe my test case isn't capturing the problem you describe?

  • mat
  • Sandy

    Sandy March 11th, 2010 @ 07:47 PM

    Sorry for all this and thanks for the help! Sorry to confuse things further, but keyword search is not working either (i turned the near filter off and didn't realize i had so i thought it was working alongside location based search).

    I now think it's got something to do with my setup / configuration.

    From Solr Admin Page

    cwd=/Users/sandy/.rvm/gems/ruby-1.9.1-p378/gems/sunspot-1.0.3/solr SolrHome=/Users/sandy/.rvm/gems/ruby-1.9.1-p378/gems/sunspot-1.0.3/solr/solr/

    In the local solr directory there is only:
    /data /pid

    The directory is generated when i start sunspot for the first time:

    rake sunspot:solr:start
    

    Is the script/generate sunspot command redundant? If i generate it i can't connect to the solr instance - connection refused. So i just trashed it and i can connect but should Solr be running from the gem directory? Shouldn't it be running from the project directory. I used your example application on github as a basis to build from. The /solr directory in the example contains the /lib and /conf directories, whereas mine does not.

    I'm going to try and replicate the same behavior in your example application.

  • Sandy

    Sandy March 11th, 2010 @ 08:04 PM

    I ran

    sunspot-installer -v ./solr -f
    

    And now inside /solr i have

    /conf
    /lib/lucene-spatial-2.9.1.jar
    /lib/solr-spatial-light-0.0.4.jar
    
  • Sandy

    Sandy March 11th, 2010 @ 08:51 PM

    I added similar functionality based on the Regression spec to the sample github application (updated to sunspot 1.0.3) - sunspot-installer -v ./solr

    It worked with filters but the same failure occurred when searching by keywords when :sort => true

    class SearchesController < ApplicationController
      def show
        origin = [47.661557 , -122.349938]
        @search = Post.search do
          
          near(origin, :distance => 20, :sort => true)
          
          if params[:q].present?
            keywords(params[:q]) do
              highlight :body
            end
          end
          if params[:cat].present?
            with(:category_id, params[:cat].to_i)
          end
          if params[:tags].present?
            with(:tags).all_of(params[:tags])
          end
          with(:published_at).less_than(Time.now)
          facet :category_id
          facet :tags, :limit => 10
          paginate(:page => params[:page], :per_page => 10)
          data_accessor_for(Post).include = :category
        end
      end
    end
    

    and in the model

    class Post < ActiveRecord::Base
      belongs_to :category
    
      searchable do
        text :title, :default_boost => 2
        text :tags
        text :body, :stored => true
        integer :category_id, :references => Category
        time :published_at
        string :tags, :multiple => true do
          tags.split(/\W+/)
        end
        coordinates { [lat , lng] }
      end
    end
    

    Now when i query:

    http://localhost:3000/search?q=odit&amp;tags[]=reintermediate&amp;t...

     RSolr::RequestError in SearchesController#show
    
    <html>
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
    <title>Error 500 </title>
    </head>
    <body><h2>HTTP ERROR: 500</h2><pre>null
    
    java.lang.NullPointerException
        at org.apache.lucene.spatial.tier.DistanceFieldComparatorSource$DistanceScoreDocLookupComparator.copy(DistanceFieldComparatorSource.java:105)
        at org.apache.lucene.search.TopFieldCollector$OneComparatorScoringMaxScoreCollector.collect(TopFieldCollector.java:301)
        at org.apache.solr.search.DocSetDelegateCollector.collect(DocSetHitCollector.java:102)
        at org.apache.lucene.search.IndexSearcher.searchWithFilter(IndexSearcher.java:294)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:254)
        at org.apache.lucene.search.Searcher.search(Searcher.java:171)
        at org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:1101)
        at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:880)
        at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341)
        at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182)
        at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
        at org.mortbay.jetty.Server.handle(Server.java:285)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
    </pre>
    <p>RequestURI=/solr/select</p><p><i><small><a href="http://jetty.mortbay.org/">Powered by Jetty://</a></small></i></p><br/>
    
  • Sandy
  • Sandy

    Sandy March 12th, 2010 @ 09:03 PM

    Ok, i've had a mini breakthrough. I still don't know why my app won't work with a sunspot.yml file. Or why the solr home directory is in the gem directory and not the solr project directory.

    Since each Hit instance is now returning distance values, i have noticed that when using the :sort => true parameter, the distance is returned for each hit only upon the first instance of each search (this may be a problem with my overall logic - i think i will move everything into the model), however, the same problem exists when replicating the behaviour in the sample app from github.

    If i simply reload the page with the same query parameters, a distance value is not returned to each hit. If i add another query parameter, to further scope or facet the search, that is when i get the null pointer error. But if the center point changes, then it will work as expected.

    Any ideas?

  • Sandy

    Sandy March 13th, 2010 @ 10:33 PM

    • Title changed from “Location based search sorting isn't compatible with other Filters” to “Location based search sort doesn't work with subsequent narrowing down with filters”
  • mat

    mat March 19th, 2010 @ 02:49 PM

    Just wanted to let you know that I've been able to reproduce this in a test in solr-spatial-light and it's fixed in that project's master. Your detailed reports were very helpful and you were on to something with the empty distances -- the NPE and the empty distances actually had the same underlying cause, and the same fix. Thanks very much!

    I'll resolve this ticket once I pull the new solr-spatial-light version into sunspot.

  • mat

    mat March 19th, 2010 @ 02:57 PM

    • State changed from “new” to “resolved”

    (from [7498abb5351a895c05b75e947de9711298c2bd5d]) Update solr-spatial-light to 0.0.5

    Fixes NullPointerException on second search with same spatial param;
    missing distances on second search with same spatial param; and
    irrelevant distances bug.

    [#95 state:resolved] [#89 state:resolved] http://github.com/outoftime/sunspot/commit/7498abb5351a895c05b75e94...

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Awesome Solr interaction for Ruby

People watching this ticket

Referenced by

Pages