view · edit · sidebar · attach · print · history

Index>

20161130-cleanupActiveAgents

Summary

  • Get rid of inactive_agents which have is_active_agent == nil
  • Make davaz.com running under Rack
  • Keep in Mind

Commits

Index

Get rid of inactive_agents which have is_active_agent == nil

Started the modified swissmedic importer yesterday. Analysing its output. Looks like the output got stuck very early. Found in log/oddb/debug/2016/11.log 

2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:127:in `verify_packages': verify_packages from /mnt/oddb-ci2/git/oddb.org/test/var/xls/Packungen-2016.11.29.xlsx 10 registrations and 6 @known_packages from ["00278", "01", "002"] till ["57678", "03", "024"]
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:144:in `block (2 levels) in verify_packages': verify_packages delete 00278 01 001
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:271:in `update':  cp /mnt/oddb-ci2/git/oddb.org/test/var/xls/Packungen-2016.11.29.xlsx /mnt/oddb-ci2/git/oddb.org/test/var/xls/Packungen-latest.xlsx after 0 minutes
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:313:in `update':  done. 0 export_registrations @update_comps was  with 9 changes
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:314:in `update':  done0. $swissmedic_do_tracing is true. Having 1 threads
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:316:in `update':  done1. $swissmedic_do_tracing is false. Having 1 threads
2016-11-29 17:29:58 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:318:in `update':  done2. $swissmedic_do_tracing is false. Having 1 threads
2016-11-29 17:29:59 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:321:in `update':  done3. $swissmedic_do_tracing is false. Having 1 threads
2016-11-29 17:29:59 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:323:in `update':  done4. $swissmedic_do_tracing is false. Having 1 threads
2016-11-29 17:29:59 +0100: /mnt/oddb-ci2/git/oddb.org/src/plugin/swissmedic.rb:327:in `update':  done5 ensure. $swissmedic_do_tracing is false. Having 1 threads

And in the stdout I see 

59267/01:  :!registration,59267!sequence,01!composition,22638432.: Must correct is 1 agents. Has  1 active_agents
Deleting wrong_agent odba_isolated_store :!registration,59267!sequence,01!composition,22638432!active_agent,Influenza Virus Surface Antigen (haemagglutinin And Neuraminidase) (a/h1n1; A/h3n2; B)%: A/brisbane/59/2007. is 24364827 Influenza Virus Surface Antigen (Haemagglutinin and Neuraminidase) (A/h1N1; A/h3N2; B): A/brisbane/59/2007 15 µg/ml
59267/01: :!registration,59267!sequence,01!composition,22638432.: has now 0 active_agents
59267/02:  :!registration,59267!sequence,02!composition,25001627.: Must correct is 6 agents. Has  6 active_agents
Deleting wrong_agent odba_isolated_store :!registration,59267!sequence,02!composition,25001627!active_agent,Haemagglutininum Influenzae A (h1n1). is 25001630 Haemagglutininum Influenzae a (H1N1)
Deleting wrong_agent odba_isolated_store :!registration,59267!sequence,02!composition,25001627!active_agent,Neuraminidasum Inactivatum (virus-stamm A/california/7/2009 (h1n1)-derived Strain%, Used Nymc X-181). is 26760485 Neuraminidasum Inactivatum (Virus-Stamm A/california/7/2009 (H1N1)-Derived Strain, used nymc X-181) 15 µg/ml
message type 0x5a arrived from server while idle

Transforming the new puts-Statements into LogFile.debug and activating the trace_msg. There were 2088 active_agents deleted. And what means the line message type 0x5a arrived from server while idle. Did the server die? Now my bin/admin statement $res= active_sequences.collect{|s| s.compositions.collect{|c| c.active_agents.find_all {|x| x.is_active_agent == nil} }} hangs. Waiting 5 minutes to be sure that it is really stuck. Then I will restart the oddbd daemon. After another 3 minutes I got 

ch.oddb> $res= active_sequences.collect{|s| s.compositions.collect{|c| c.active_agents.find_all {|x| x.is_active_agent == nil} }}
-> $res.size
Array
ch.oddb> $res.size
-> 11151

Import is running nicely and seems to produce correct result. See

While the report is running, asking myself why sometimes the Globulina Equina is recognized by the parse as active_agent and sometimes not. Eg. I find in oddb_calc.xml 

           <SUBSTANCE_NAME>Globulina Equina (immunisé Avec Tissu Ostéocartilagineux, Conjonctif, Séreuses, Ganglions Lymphatiques Porcins)</SUBSTANCE_NAME>
            <IS_ACTIVE_AGENT>false</IS_ACTIVE_AGENT>
and 
            <SUBSTANCE_NAME>Globulina Equina (immunisé Avec Muqueuse Sinusinale Porcins)</SUBSTANCE_NAME>
            <IS_ACTIVE_AGENT>true</IS_ACTIVE_AGENT>

The answer is that the recognition fails, if we have inside a bracket one or more occurrences of a comma. This is contradiction to the the result, where a comma delimits a new substance. I think I will just a special if /Globulina Equina/ matches to catch these 24 errors.

This was easy pushed the following commit to oddb2xml Added a hack for 24 globulina equina which commas between brackets

Travis-CI reminded me that I have a few errors. Eg. 

 Oddb2xml::MedregbmDownloader person download person txt should return valid String FIXED
     Expected pending 'Should handle SSL issues' to fail. No Error was raised.

This fix was particularly easy. Just remove the "Pending" line! But this provoked other errors as Medreg-Info for pharmacies and medical persons und is now distributed as XLSX and no longer as XLS/txt.

Porting the parslet changes to oddb.org with commit Added a hack for 24 compositions globulina equina with commas between brackets

My swissmedic job to update the composition is running since 3,25 h and has only update 3539 of 11151 active_sequences.

At 15:20 the import finished and sent me an email containing 

ODDB::SwissmedicPlugin - Report 30.11.2016
Total time to update: 399.00 [m]
Checked compositions: 11096
New compositions: 11096
Deleted compositions: 2
Updated agents: 9936
New agents: 60

Verified the following pages (after restarting all oddb-daemons):

As the patch for the Globulina Equina was not yet applied, I still have the wrong information there. Rerunning the import.

Pushed the commit Display corresponding chemical substances correctly.

Rerunning the import locally produced an error, as swissmedic update tried to update the field is_active_agent, which is no longer writeable.

When looking at the code and some items in the database I find cases, where inactive_agents are wrongly marked as is_active, e.g. 

ch.oddb> registration('00277').sequence('01').compositions.first.inactive_agents.first.pointer
-> :!registration,00277!sequence,01!composition,23826342!active_agent,Globulina Equina (immunisé Avec Coeur%, Endothélium Vasculaire Porcins).
ch.oddb> registration('00277').sequence('01').compositions.first.inactive_agents.first.oid
-> 30656124

Added some code to fix this problem now. Running the imported deletes (temporarily) a lot of inactive-agents, which still have the ActiveAgent class. But once this one time patch done. It will be possible to remove the is_active_agent attributes.

Reverted the patch on thinpower to wait for my local import to finish.

Received email with 

ODDB::SwissmedicPlugin - Report 01.12.2016
Total time to update: 563.00 [m]
Checked compositions: 11096
New compositions: 11096
Deleted compositions: 0
Updated agents: 9565
New agents: 5135

Looks good. Checked (after restarting oddbd)

Will start the update job on thinpower on the evening of Dec. 1. Pushed commit Do not update is_active_agent and Delete inactive_agent with wrong is_active_agent

Make davaz.com running under Rack

Working on the problem, why after a login we don't display the correct page.

Testing manually, I was able to login and the page http://davaz.ngiger.ch/en/gallery/new_art_object/ looks very similar to davaz.com. But after entering some data and clicking "Save" the URL got redirected to http://davaz.ngiger.ch/en/gallery, but no new picture was displayed. Going back and clicking on Logout, did throw an exception 

NoMethodError at /en/admin/logout/fragment/Rack_AFVfragment/Rack_AFV
undefined method `movies' for "/en/works/design/#Rack_AFV":String
Ruby 	/var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb: in block in component, line 59
Web 	GET davaz.ngiger.ch/en/admin/logout/fragment/Rack_AFVfragment/Rack_AFV
Traceback (innermost first)
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb: in block in component
                args = [model.send(key), session || @session, self]...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb: in create
        					self.send(component, model)...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb: in block in compose
        				div.push(label(create(component, model), component))...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb: in each
        			}.each { |matrix, component|...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb: in compose
        			}.each { |matrix, component|...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb: in init
        			compose()...
    /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/component.rb: in initialize
        			init()...
    /home/niklaus/git/sbsm/lib/sbsm/state.rb: in new
 /home/niklaus/git/sbsm/lib/sbsm/state.rb: in new
          view = klass.new(model, @session)...
/home/niklaus/git/sbsm/lib/sbsm/state.rb: in view
          view = klass.new(model, @session)...
/home/niklaus/git/sbsm/lib/sbsm/state.rb: in to_html
          name = view...
/home/niklaus/git/sbsm/lib/sbsm/session.rb: in to_html
    			@state.to_html(cgi)...
/home/niklaus/git/sbsm/lib/sbsm/session.rb: in block in drb_process
            to_html

After restarting the jobs, I made a login and after pressing logout I got another time an empty screen with the URL http://davaz.ngiger.ch/en/admin/logout/fragment/. After going back I had the followin exception 

/var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb:157: warning: Hash#index is deprecated; use Hash#key
NoMethodError: undefined method `movies' for "/en/gallery/gallery/":String
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb:59:in `block in component'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb:145:in `create'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb:33:in `block in compose'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb:15:in `each'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/divcomposite.rb:15:in `compose'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/composite.rb:72:in `init'
        /var/www/davaz.com/vendor/ruby/2.3.0/gems/htmlgrid-1.1.3/lib/htmlgrid/component.rb:139:in `initialize'
        /home/niklaus/git/sbsm/lib/sbsm/state.rb:239:in `new'
        /home/niklaus/git/sbsm/lib/sbsm/state.rb:239:in `view'
        /home/niklaus/git/sbsm/lib/sbsm/state.rb:173:in `to_html'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:441:in `to_html'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:175:in `block in drb_process'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:173:in `synchronize'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:173:in `drb_process'
        /usr/local/lib/ruby/2.3.0/drb/drb.rb:1137:in `method_missing'
        /home/niklaus/git/sbsm/lib/sbsm/app.rb:106:in `call'

Added a test for login/logout. But must first figure out, why I cannot login when running the tests, whereas I am able to login in the real app.

Analysing error- and access_log for login/logou 

# access_log
127.0.0.1 - - [30/Nov/2016:11:37:41 +0100] "GET /resources/javascript/widget/login.js HTTP/1.1" 200 1792 "http://davaz.ngiger.ch/en/gallery/gallery/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"              
127.0.0.1 - - [30/Nov/2016:11:37:41 +0100] "GET /resources/javascript/widget/templates/login.html HTTP/1.1" 200 224 "http://davaz.ngiger.ch/en/gallery/gallery/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"   
127.0.0.1 - - [30/Nov/2016:11:37:41 +0100] "GET /en/admin/login_form/ HTTP/1.1" 200 1096 "http://davaz.ngiger.ch/en/gallery/gallery/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"                              
127.0.0.1 - - [30/Nov/2016:11:37:45 +0100] "GET /en/admin?login_email=juerg%40davaz.com&login_password=&remember_me=1&flavor=&language=en&event=login&state_id=69943609856680&fragment= HTTP/1.1" 200 16 "http://davaz.ngiger.ch/en/
127.0.0.1 - - [30/Nov/2016:11:37:45 +0100] "GET /en/gallery/gallery/ HTTP/1.1" 200 41826 "http://davaz.ngiger.ch/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"                                                 
<..>
127.0.0.1 - - [30/Nov/2016:11:37:45 +0100] "GET /resources/css/gallery_admin.css HTTP/1.1" 200 1224 "http://davaz.ngiger.ch/en/gallery/gallery/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"                   
27.0.0.1 - - [30/Nov/2016:11:35:07 +0100] "GET /en/admin/logout/fragment/Rack_AFV HTTP/1.1" 200 - "http://davaz.ngiger.ch/en/works/design/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
127.0.0.1 - - [30/Nov/2016:11:35:22 +0100] "GET /en/admin/logout/fragment/Rack_AFVfragment/Rack_AFV HTTP/1.1" 500 101682 "http://davaz.ngiger.ch/en/works/design/" "Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"

# error_log
[Wed Nov 30 11:35:07 2016] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //localhost:8007/en/admin/logout/fragment/Rack_AFV
[Wed Nov 30 11:35:07 2016] [debug] proxy_util.c(1535): [client 127.0.0.1] proxy: *: found reverse proxy worker for http://localhost:8007/en/admin/logout/fragment/Rack_AFV, referer: http://davaz.ngiger.ch/en/works/design/
[Wed Nov 30 11:35:07 2016] [debug] mod_proxy.c(1032): Running scheme http handler (attempt 0)                               
[Wed Nov 30 11:35:07 2016] [debug] mod_proxy_http.c(1995): proxy: HTTP: serving URL http://localhost:8007/en/admin/logout/fragment/Rack_AFV

When running the test the cuki_str does not contain :remember= after the login. Why?

view · edit · sidebar · attach · print · history
Page last modified on December 30, 2016, at 06:55 PM