After reinstalling the correct swissmedic-diff running import-daily completed yesterday without problem. Why? Will continue tomorrow. Therefore getting dump of yesterday an verifying that I have the same latest files on oddb-ci2 as on thinpower. on thinpower
ywesee@thinpower /var/www/oddb.org $ ls -lrt /var/www/oddb.org/data/*/*latest* -rw-r--r-- 1 apache apache 113665 21. Mai 2010 /var/www/oddb.org/data/xls/vaccines-latest.xls -rw-r--r-- 1 apache apache 711548 13. Dez 2011 /var/www/oddb.org/data/pdf/Swissmedic-Journal-latest.pdf -rw-r--r-- 1 apache apache 250151 13. Jan 2012 /var/www/oddb.org/data/pdf/narcotics-de-latest.pdf -rw-r--r-- 1 apache apache 250151 13. Jan 2012 /var/www/oddb.org/data/pdf/narcotics-fr-latest.pdf -rw-r--r-- 1 apache apache 4551596 8. Apr 2012 /var/www/oddb.org/data/xml/XMLSwissindexPharma-latest.xml -rw-r--r-- 1 apache apache 101735 13. Okt 2014 /var/www/oddb.org/data/xls/analysis_de_latest.xlsx -rw-r--r-- 1 apache apache 104382 13. Okt 2014 /var/www/oddb.org/data/xls/analysis_fr_latest.xlsx -rw-r--r-- 1 apache apache 266307 1. Dez 2014 /var/www/oddb.org/data/xls/companies_latest.xlsx -rw-r--r-- 1 apache apache 2734603 16. Dez 2014 /var/www/oddb.org/data/xls/doctors_latest.xlsx -rw-r--r-- 1 apache apache 4749986 2. Jun 11:01 /var/www/oddb.org/data/xml/XMLSwissindexPharma-DE-latest.xml -rw-r--r-- 1 apache apache 2994009 8. Jul 00:18 /var/www/oddb.org/data/xls/Packungen-latest.xlsx -rw-r--r-- 1 apache apache 4806151 4. Aug 07:26 /var/www/oddb.org/data/xml/XMLPublications-latest.zip -rw-r--r-- 1 apache apache 5718445 4. Aug 10:42 /var/www/oddb.org/data/xml/XMLRefdataPharma-latest.xml -rw-r--r-- 1 apache apache 2203563 14. Aug 07:26 /var/www/oddb.org/data/xls/Präparateliste-latest.xlsx -rw-r--r-- 1 apache apache 10623854 18. Aug 07:01 /var/www/oddb.org/data/csv/interactions_de_utf8-latest.csv -rw-r--r-- 1 apache apache 832700648 18. Aug 07:15 /var/www/oddb.org/data/xml/AipsDownload_latest.xml
on oddb-ci2 (after manually copying XMLRefdataPharma-latest.xml from thinpower
niklaus@oddb-ci2 /v/w/oddb.org> ls -lrt /var/www/oddb.org/data/*/*latest* -rw-r--r-- 1 apache apache 4787424 18. Nov 2014 /var/www/oddb.org/data/xml/XMLSwissindexPharma-de-latest.xml -rw-r--r-- 1 apache apache 4749986 9. Jun 09:47 /var/www/oddb.org/data/xml/XMLSwissindexPharma-DE-latest.xml -rw-r--r-- 1 apache apache 2994008 6. Jul 22:59 /var/www/oddb.org/data/xlsx/Packungen-latest.xlsx -rw-r--r-- 1 apache apache 2235611 6. Jul 22:59 /var/www/oddb.org/data/xlsx/Präparateliste-latest.xlsx -rw-r--r-- 1 apache apache 2994009 7. Jul 11:57 /var/www/oddb.org/data/xls/Packungen-latest.xlsx -rw-r--r-- 1 apache apache 4806151 11. Aug 02:32 /var/www/oddb.org/data/xml/XMLPublications-latest.zip -rw-r--r-- 1 apache apache 10623854 17. Aug 15:59 /var/www/oddb.org/data/csv/interactions_de_utf8-latest.csv -rw-r--r-- 1 apache apache 2203563 17. Aug 18:17 /var/www/oddb.org/data/xls/Präparateliste-latest.xlsx -rw-r--r-- 1 apache apache 831483455 17. Aug 21:10 /var/www/oddb.org/data/xml/AipsDownload_latest.xml -rw-r--r-- 1 apache apache 5718445 18. Aug 07:49 /var/www/oddb.org/data/xml/XMLRefdataPharma-latest.xml
Ignoring AipsDownload_latest.xml as it will be downloaded as of today. Dates are not always exactly the same, but the sizes match.
Found the following entry on oddb-ci2 oddb/debug/2015/08.log 2015-08-18 07:32:03 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:648 remove_active_agents_that_are_nil 46111/01 composition.oid 9479306 7 active_agents. 30 mg substance.oid 34127211 substance.pointer :!registration,46111!sequence,01!composition,9479306!active_agent,-hydricum,false. and now at thinpower I found 2015-08-18 08:22:07 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:651 remove_active_agents_that_are_nil 46111/01 composition.oid 9479306 7 active_agents. 30 mg substance.oid 34123339 substance.pointer :!registration,46111!sequence,01!composition,9479306!active_agent,-hydricum,false.
I tried to catch the problem by changing the line def delete_active_agent(substance)
def delete_active_agent(substance, dummy = nil)
require 'pry'; binding.pry if dummy
Finally I got into pry and it was easy to diagnose the problem
From: /var/www/oddb.org/src/model/composition.rb @ line 62 ODDB::Composition#delete_active_agent:
61: def delete_active_agent(substance, dummy = nil)
=> 62: require 'pry'; binding.pry if dummy
63: if(active = active_agent(substance))
64: @active_agents.delete(active)
65: @active_agents.odba_isolated_store
66: active
67: end
68: end
3] pry(#<ODDB::Composition>)> substance
=> "Enterococcus Faecium Vivus (typus Sf 68)"
[4] pry(#<ODDB::Composition>)> dummy
=> true
But it looks like deleting this active_agent produces problem as I see now an empty Wirkstoff. Looking at the code I see, that delete_active_agents really does need a second parameter and that I must consider whether it is a Wirkstoff or a Hilfsstoff. Done. Reimporting database.
Looking and debugging with the unit test I see, that I run into this situation when running the second import. Therefore it looks as if I did not create correctly the active_agents the first time. Looks like handling the excipiens is completly wrong, as it decides to destroy doses instead. Narrowing error down to method update_active_agent where a dose gets added instead of the substance. Fixed unit-tests by specifying that :delete should never be called. Now my unit test work fine.
Okay now import worked find. Pushed commit Fix must_check and update_active_agent in import swissmedic
Resuming work to do (relative to my workspace):
First a small commit, because ruby 2.1.2 is no longer the most recent ruby for Ruby 2.x Avoid fixing to ruby 2.1.2
I think I must add a proper Composition not only a CompositionList. Should I add one single CompositionList for active and inactive agents? But how can I add in this case the needed title "Wirk|Hilfsstoffe"? Example is IKSNR 00603 "Hiberix, Stechampulle mit Lyophilisat und Solvens.
Working on symplifying the views. Found one error. But must wait for jobs/update_swissmedicinfo_only update_compositions to finish. Will continue tomorrow.