#197 new
Jimmy Secretan

Problem with sharded sunspot search

Reported by Jimmy Secretan | April 26th, 2013 @ 10:01 PM | in Wishlist

I am trying to get sunspot to search a sharded/multicore index. When I use the following initializer:

shard_sessions = (0..3).map do |i|
    session = Sunspot::Session.new
    session.config.solr.url = "http://host/solr/core#{i}"
    Sunspot::SessionProxy::ThreadLocalSessionProxy.new(session)
end

Sunspot.session = Sunspot::SessionProxy::IdShardingSessionProxy.new(Sunspot.session, shard_sessions)

I get the following error (full trace):

sunspot (1.3.3) lib/sunspot/session_proxy/sharding_session_proxy.rb:157:in `block in new_search'
sunspot (1.3.3) lib/sunspot/session_proxy/sharding_session_proxy.rb:157:in `map'
sunspot (1.3.3) lib/sunspot/session_proxy/sharding_session_proxy.rb:157:in `new_search'
sunspot (1.3.3) lib/sunspot/session_proxy/sharding_session_proxy.rb:174:in `search'
sunspot (1.3.3) lib/sunspot.rb:345:in `search'
app/controllers/search_phrases_controller.rb:66:in `results'
actionpack (3.2.13) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (3.2.13) lib/abstract_controller/base.rb:167:in `process_action'
actionpack (3.2.13) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (3.2.13) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
activesupport (3.2.13) lib/active_support/callbacks.rb:425:in `_run__2100136803045882264__process_action__2354738279958607022__callbacks'
activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback'
activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks'
activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.2.13) lib/abstract_controller/callbacks.rb:17:in `process_action'
actionpack (3.2.13) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
activesupport (3.2.13) lib/active_support/notifications.rb:123:in `block in instrument'
activesupport (3.2.13) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (3.2.13) lib/active_support/notifications.rb:123:in `instrument'
actionpack (3.2.13) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
actionpack (3.2.13) lib/action_controller/metal/params_wrapper.rb:207:in `process_action'
activerecord (3.2.13) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (3.2.13) lib/abstract_controller/base.rb:121:in `process'
actionpack (3.2.13) lib/abstract_controller/rendering.rb:45:in `process'
actionpack (3.2.13) lib/action_controller/metal.rb:203:in `dispatch'
actionpack (3.2.13) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
actionpack (3.2.13) lib/action_controller/metal.rb:246:in `block in action'
actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:73:in `call'
actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:73:in `dispatch'
actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:36:in `call'
journey (1.0.4) lib/journey/router.rb:68:in `block in call'
journey (1.0.4) lib/journey/router.rb:56:in `each'
journey (1.0.4) lib/journey/router.rb:56:in `call'
actionpack (3.2.13) lib/action_dispatch/routing/route_set.rb:612:in `call'
warden (1.2.1) lib/warden/manager.rb:35:in `block in call'
warden (1.2.1) lib/warden/manager.rb:34:in `catch'
warden (1.2.1) lib/warden/manager.rb:34:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
rack (1.4.5) lib/rack/etag.rb:23:in `call'
rack (1.4.5) lib/rack/conditionalget.rb:25:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/head.rb:14:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/flash.rb:242:in `call'
rack (1.4.5) lib/rack/session/abstract/id.rb:210:in `context'
rack (1.4.5) lib/rack/session/abstract/id.rb:205:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/cookies.rb:341:in `call'
activerecord (3.2.13) lib/active_record/query_cache.rb:64:in `call'
activerecord (3.2.13) lib/active_record/connection_adapters/abstract/connection_pool.rb:479:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `_run__2108545853073247874__call__3668802813255819695__callbacks'
activesupport (3.2.13) lib/active_support/callbacks.rb:405:in `__run_callback'
activesupport (3.2.13) lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
activesupport (3.2.13) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.2.13) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/reloader.rb:65:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
airbrake (3.1.10) lib/airbrake/rails/middleware.rb:13:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
railties (3.2.13) lib/rails/rack/logger.rb:32:in `call_app'
railties (3.2.13) lib/rails/rack/logger.rb:16:in `block in call'
activesupport (3.2.13) lib/active_support/tagged_logging.rb:22:in `tagged'
railties (3.2.13) lib/rails/rack/logger.rb:16:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/request_id.rb:22:in `call'
rack (1.4.5) lib/rack/methodoverride.rb:21:in `call'
rack (1.4.5) lib/rack/runtime.rb:17:in `call'
activesupport (3.2.13) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
rack (1.4.5) lib/rack/lock.rb:15:in `call'
actionpack (3.2.13) lib/action_dispatch/middleware/static.rb:63:in `call'
airbrake (3.1.10) lib/airbrake/user_informer.rb:16:in `_call'
airbrake (3.1.10) lib/airbrake/user_informer.rb:12:in `call'
railties (3.2.13) lib/rails/engine.rb:479:in `call'
railties (3.2.13) lib/rails/application.rb:223:in `call'
railties (3.2.13) lib/rails/railtie/configurable.rb:30:in `method_missing'
rack (1.4.5) lib/rack/lint.rb:48:in `_call'
rack (1.4.5) lib/rack/lint.rb:36:in `call'
rack (1.4.5) lib/rack/showexceptions.rb:24:in `call'
rack (1.4.5) lib/rack/commonlogger.rb:33:in `call'
rack (1.4.5) lib/rack/chunked.rb:43:in `call'
rack (1.4.5) lib/rack/content_length.rb:14:in `call'
unicorn (4.6.2) lib/unicorn/http_server.rb:552:in `process_client'
unicorn (4.6.2) lib/unicorn/http_server.rb:632:in `worker_loop'
unicorn (4.6.2) lib/unicorn/http_server.rb:500:in `spawn_missing_workers'
unicorn (4.6.2) lib/unicorn/http_server.rb:142:in `start'
unicorn (4.6.2) bin/unicorn:126:in `<top (required)>'
/Users/jsecretan/.rvm/gems/ruby-1.9.3-p194/bin/unicorn:23:in `load'
/Users/jsecretan/.rvm/gems/ruby-1.9.3-p194/bin/unicorn:23:in `<main>'

Any help would be greatly appreciated. Thanks!

Comments and changes to this ticket

  • Jimmy Secretan

    Jimmy Secretan April 27th, 2013 @ 07:20 AM

    I managed to make a little bit of progress. Originally for the initializer, I had used some code I had found here:

    Link

    but I'm not sure that was correct. This is what I have now:

    shard_sessions = (0..3).map do |i|
        config = Sunspot::Configuration.build
        config.solr.url = "host/solr/core#{i}"
        Sunspot::SessionProxy::ThreadLocalSessionProxy.new(config)
    end
    
    base_session = Sunspot::Session.new
    base_session.config.solr.url = "http://host/solr/core0"
    
    Sunspot.session = Sunspot::SessionProxy::IdShardingSessionProxy.new(base_session, shard_sessions)
    

    One thing I noted was that I had to leave out the http in the shard session URLs.

    Now when I do a full text search, this doesn't return an error like the previous setup, but instead seems to blank out the query. The query works fine when I just specify a normal core URL.

    When I submit the following query

    verb_search = Sunspot.search(Patent) do
          fulltext search_text do
            boost_fields :title => 2.0
            highlight :abstract
            highlight :description
            query_phrase_slop slop_factor
            minimum_match 1
          end
          paginate :page => current_page, :per_page => page_length
        end
    

    the output looks as follows

    fq=type%3APatent&start=0&rows=30&shards=host%2Fsolr%2Fcore0%2Chost%2Fsolr%2Fcore1%2Chost%2Fsolr%2Fcore2%2Chost%2Fsolr%2Fcore3&q=%2A%3A%2A
    

    But, as I mentioned, when I do a non-sharded setup, the query comes out fine.

  • Jimmy Secretan

    Jimmy Secretan April 29th, 2013 @ 08:55 AM

    I managed to fix the code after forking sunspot:

    git@github.com:jsecretan/sunspot.git

    Not sure if I was just setting this up wrong or this just never worked. It seemed the block was not getting passed properly to the delegated search. Hopefully I can clean this up and pull request if it is a genuine problem.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Awesome Solr interaction for Ruby

People watching this ticket

Pages