view · edit · sidebar · attach · print · history

< Niklaus.20161220-davaz-com-multithreaded | Index | Niklaus.20161214-sbsm-without-drb >>

20161219-error-TextInfoPlugin

Summary

  • Fix error in import swissmedic TextInfoPlugin
  • Davaz.com should support heavy workload
  • Keep in Mind

Commits

Index

Fix error in import swissmedic TextInfoPlugin

Found the following error this morning from import_daily

Plugin: ODDB::TextInfoPlugin
Error: NoMethodError
Message: undefined method `packages' for ["rel", "stylesheet"]:Array
Backtrace:
/var/www/oddb.org/src/model/registration.rb:241:in `block in packages'
/var/www/oddb.org/src/model/registration.rb:240:in `each'
/var/www/oddb.org/src/model/registration.rb:240:in `inject'
/var/www/oddb.org/src/model/registration.rb:240:in `packages'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:1334:in `block (2 levels) in report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1332:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1332:in `block in report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1330:in `open'
/var/www/oddb.org/src/plugin/text_info.rb:1330:in `report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1402:in `parse_aips_download'
/var/www/oddb.org/src/plugin/text_info.rb:1447:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:554:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:514:in `call'
/var/www/oddb.org/src/util/updater.rb:514:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:548:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:335:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:202:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

This was again the result of an out-of-memory error as shown by the following lines from log/oddb/debug/2016/12.log

2016-12-08 08:29:06 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:762:in `update_active_agent': update_inactive_agent 00701/02 Polysorbatum 80  false dose 0  agent.pointer :!registration,00701!sequence,02!composition,29041064!inactive_agent,Polysorbatum 80.
2016-12-08 08:29:07 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:163:in `trace_memory_useage': Using 16744 MB of memory. Limit is 16384. Swissmedic_do_tracing true
2016-12-08 08:29:07 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:168:in `block in trace_memory_useage': Aborting as using 16744 MB of memory > than limit of 16384
2016-12-08 08:29:07 CETlog notify Error: swissmedic: start outgoing process ["log"]. Must attach 0 files and 0 parts.
2016-12-08 08:29:07 CETUtil.send_mail list_and_recipients ["log"]
2016-12-08 08:29:07 CET Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Error: swissmedic - 12/2016 size 2884 with 0 attachments. Plugin: ODDB::SwissmedicPlugin
Error: SystemExit
Message: SystemExit
Backtrace:

We must correct the e-mail text sent in this case.

Pushed commits Meaningfull error message for out of memory and Memory error when > 16 GB, not 2GB

Davaz.com should support heavy workload

Updating davaz.com to work with SBSM 1.3.5.

Asking myself why in src/state/works/movies.rb the init method cannot call @session.load_movies even when the @session is a DaVaz::Util::Session.

Before I started migrating to Rack there were many actions, loading, counting defined in src/util/app.rb which was a class not derived from a SBSM class. Now I made it derive from SBSM::App. Created and pushed a tag before_rack to https://github.com/ngiger/davaz.com/tree/before_rack see more easily the difference. Examining the diff generated via git diff -w before_rack > rack.diff

The load_movies method is currently accessible via @session.app.load_movies instead of the earlie @session. Shall we make it accessible by implmenting a method_missing to SBSM::Session? Looks like a good idea. I even got the first time into the extend methods of lib/sbsm/state.rb where SBSM looks for @viral_modules. And indeed it detects the mod::EVENT_MAP.

Next question is now: The login rejects wrong username/password combinations, but why do we still see the login button after the login?

Using pry I see that I messed up the cookie handling. When stopping after the login I have in lib/sbsm/app.rb @ line 112

request.cookies  {"davaz.com-preferences" =>"{:language=>\"en\"}", "_session_id"=>"10e524151d7f0da819f4222ecc1"}
request.env['rack.request.cookie_hash']   => {"davaz.com-preferences"=>"{:language=>\"en\"}", "_session_id"=>"10e524151d7f0da819f4222ecc1"}
request.env['rack.request.cookie_string'] => "davaz.com-preferences=%7B%3Alanguage%3D%3E%22en%22%7D; _session_id=10e524151d7f0da819f4222ecc1; davaz.com-preferences=%7B%3Aremember%3D%3E%22cc42f585f093931b54da7831f0ed0fd066bddf245d869a2cd5f0bbe9b1c1c644%22%2C+%3Aname%3D%3E%22juerg%40davaz.com%22%2C+%3Alanguage%3D%3E%22en%22%7D; _session_id=98003e3199d3e50976a9c54352"

this results in lib/sbsm/session.rb @ line 217 to

rack_request.params
=> {"language"=>"en", "zone"=>"works", "event"=>"movies"}
[16] pry(#<DaVaz::Util::Session>)> @cookie_input
=> {:language=>"en"}

Adding a unit test for this failure in sbsm. But also testing davaz.com. Looking at how Rack checks for cookies by looking at /home/niklaus/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rack-2.0.1/lib/rack/request.rb

204       def cookies
205         hash = fetch_header(RACK_REQUEST_COOKIE_HASH) do |k|
206           set_header(k, {})
207         end
208         string = get_header HTTP_COOKIE
209 
210         return hash if string == get_header(RACK_REQUEST_COOKIE_STRING)
211         hash.replace Utils.parse_cookies_header get_header HTTP_COOKIE
212         set_header(RACK_REQUEST_COOKIE_STRING, string)
213         hash
214       end

Searching for a solution where I can use Rack methods to set and retrieve the indivual cookie items. Wondering why I don't see state_id in the second line when looking at the log file from test/feature/login_out_test.rb

::1 - - [19/Dec/2016:16:03:13 +0100] "GET /en/admin?login_email=juerg%40davaz.com&login_password= &remember_me=1&flavor=&language=en&event=login&state_id=47327897449080&fragment= HTTP/1.1" 200 16 0.0635
D, [2016-12-19T16:03:13.393949 #15679] DEBUG -- : app.rb:108:in `call' starting session_id 102f4e878f9b29c39d75a5f2077  session DaVaz::Util::Session /en/works/movies/: cookies  are {"language"=>"en", "_session_id"=>"102f4e878f9b29c39d75a5f2077", "davaz.com-preferences"=>"102f4e878f9b29c39d75a5f2077", "remember"=>"5c044e2134236fd263cca584634d2fbf14b575b1de0ca4c77f1b03dc28c0cf28", "name"=>"juerg@davaz.com"} @cgi CGI

Okay. Rack displays in its GET log also the content of request.env['QUERY_STRING']. This is a rack parameter I ignored till now. In my debug session its content was login_email=juerg%40davaz.com&login_password=&remember_me=1&flavor=&language=en&event=login&state_id=47126673689520&fragment=.

The test/feature/login_out_test.rb continues to fail but davaz.ngiger.ch allows now a login and logout. Looks like handling fragments in rack/test work differently than I expected.

I solved the original problem, that we got exception when running wrk -t12 -c40000 -d30s http://davaz.ngiger.ch by using a module global mutex around session.process_rack as the davaz.com application is apparently not thread safe in the current state. Number of request per second is around 30, which is much less than the 3280 seen when calling the same test on davaz.com.

Running all watir tests shows 6 Failures

  1) Error:
TestGallery#test_gallery:
Selenium::WebDriver::Error::UnknownError: unknown error: Element <a name="serie_link" href="javascript:void(0);" class="serie-link even" id="ABD" onclick="return toggleShow('show', 'http://localhost:11090/en/gallery/ajax_rack/serie_id/ABD', null,
  'show_container', 'ABD');
">...</a> is not clickable at point (570, 146). Other element would receive the click: <div data-dojo-attach-point="thumbContainer" class="thumb-container"></div>
--
  1) Failure:
TestShop#test_shopping_cart_calculation_with_publications [test/feature/shop_test.rb:60]:
Expected: "Title of ArtObject 112"
--
  2) Failure:
TestShop#test_checkout [test/feature/shop_test.rb:204]:
browser text should match Your order has been succesfully sent. but is 
--
  3) Failure:
TestShop#test_checkout_fails_with_validation_error [test/feature/shop_test.rb:132]:
browser text should match Please fill out the fields that are marked with red. but is 
--
  4) Failure:
TestShop#test_checkout_fails_without_user_info [test/feature/shop_test.rb:118]:
browser text should match Please fill out the fields that are marked with red. but is 
--
  5) Failure:
TestShop#test_checkout_with_error [test/feature/shop_test.rb:49]:
Should show correct total of 568 CHF, but is Total       CHF 342.- / $ 273.- /  204.- Remove all items
--
  1) Error:
TestFamily#test_family_slide_show_widget:
Net::ReadTimeout: Net::ReadTimeout
--
  1) Error:
TestGuestbook#test_zoo_boo:
Net::ReadTimeout: Net::ReadTimeout
view · edit · sidebar · attach · print · history
Page last modified on December 30, 2016, at 05:37 PM