view · edit · sidebar · attach · print · history

Index>

20140708-Fix-importDaily

Summary

  • Fix import daily (problems with gems)

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • One unit-test for searchbar fails and might be a clue why searching does not work correctly.
  • Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?
  • Make ext/swissindex/test/test_swissindex.rb and skipping tests in migel-gem pass.

---

Remove interactions (hayes, cyp450)

bin/admin shows a few cyp450 objects, that we probably should delete

ch.oddb> cyp450s.collect{|x|x.cyp_id}
-> ["1A2", "2A6", "2B6", "2C8", "2C9", "2D6", "2E1", "3A4", "2C18", "2C19", "3A5-7"]
ch.oddb> cyp450s.each{|x| delete_cyp450(x.cyp_id) }
-> Array
ch.oddb> cyp450s.collect{|x|x.cyp_id}
-> []

Also we must remove the old @substrate_connections in substances. It might be a good strategy to have two commits.

  1. Remove all usages of cpy450 and substrate_connections (except the accessor and the delete-methods)
  2. restart oddb and check that everything is okay
  3. remove all cpy450 and substrate_connections
  4. restart oddb and check that everything is okay
  5. second commit which removes the accessor and delete methods

Fix export-generics undefined method `sequences' for "format_type":String

Analysing the error

Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.
(druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:119:in `comparables': undefined method `sequences' for "format_type":String (NoMethodError)
	from (druby://localhost:10005) /var/www/oddb.org/src/model/package.rb:214:in `comparables'
	from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:96:in `block in export_generics'
	from (druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each'
	from (druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each_package'
	from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
	from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:149:in `block in each_package'
	from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_value'
	from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_package'
	from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
	from (druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:664:in `block in each_package'
	from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `each_value'
	from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
	from (druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:663:in `each_package'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:94:in `export_generics'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:113:in `block in export_generics_xls'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `call'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `block in safe_export'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/tempfile.rb:316:in `open'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:285:in `safe_export'
	from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:111:in `export_generics_xls'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
	from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
	from /var/www/oddb.org/src/plugin/xls_export.rb:22:in `export_generics'
	from /var/www/oddb.org/src/util/exporter.rb:154:in `export_generics_xls'
	from /var/www/oddb.org/src/util/updater.rb:416:in `update_swissmedic_followers'
	from /var/www/oddb.org/src/util/updater.rb:198:in `run'
	from jobs/import_daily:13:in `block in <module:Util>'
	from /var/www/oddb.org/src/util/job.rb:40:in `call'
	from /var/www/oddb.org/src/util/job.rb:40:in `run'
	from jobs/import_daily:12:in `<module:Util>'
	from jobs/import_daily:11:in `<module:ODDB>'
	from jobs/import_daily:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted

Add debug output to see which registration/sequence causes this problem. See http://dev.ywesee.com/Niklaus/20140701-Fix-invalid-byte-sequence-in-import-daily Fachinfo displays correctly http://ch.oddb.org/de/gcc/fachinfo/swissmedicnr/17233, but not the patinfo. http://ch.oddb.org/de/gcc/patinfo/reg/17233/seq/00

Verifying amount of damage on thinpower via bin/admin

ch.oddb> registration('17233').sequence('02').package('001').atc_class
-> format_type
ch.oddb> registration('17233').sequence('02').package('068').atc_class
-> format_type
ch.oddb> $all = []; packages.collect {|x| $all << x if x.atc_class.is_a?(String)}; $all.size
->2

A few seconds later (while import_daily was running) I could not reproduce this error anymore. Did Zeno delete the registration 17233?

Fix import daily (problems with gems)

Waiting to see whether we have the same problem with oddb-ci2. Copying thinpowers-database. Got a similar problem.

Working on a unit test for plugin/swissmedic.rb which calls the update method too. Using stub/oddbapp to avoid mocking registrations, etc.

Now I get the following error

  1) Error:
ODDB::SwissmedicPluginTestXLSX#test_diff_2014_july_to_i_problem:
NoMethodError: undefined method `oid' for #<NoMethodError: undefined method `collect' for 1:Fixnum>
    /opt/src/oddb.org/src/util/oddbapp.rb:1465:in `create_unknown_galenic_group'
    /opt/src/oddb.org/src/util/oddbapp.rb:105:in `init'
    /opt/src/oddb.org/src/util/oddbapp.rb:101:in `initialize'
    /opt/src/oddb.org/src/util/oddbapp.rb:1500:in `new'
    /opt/src/oddb.org/src/util/oddbapp.rb:1500:in `block in initialize'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/odba-1.1.0/lib/odba/cache.rb:299:in `call'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/odba-1.1.0/lib/odba/cache.rb:299:in `block in fetch_named'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/odba-1.1.0/lib/odba/cache.rb:296:in `fetch_named'
    /opt/src/oddb.org/src/util/oddbapp.rb:1499:in `initialize'
    test/test_plugin/swissmedic.rb:105:in `new'
    test/test_plugin/swissmedic.rb:105:in `setup'

We must try a way to export/import the galenic_groups/forms to avoid this problem. Maybe for the tests we can just improve the stub. Worked half an hour to solve this problem and now I have exactly the right error I was looking for, namely:

  1) Error:
ODDB::SwissmedicPluginTestXLSX#test_diff_2014_july_to_i_problem:
NoMethodError: undefined method `first' for #<RubyXL::Row:0x00000006e4dbe8>
    /opt/src/oddb.org/src/plugin/swissmedic.rb:428:in `rows_diff'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/swissmedic-diff-0.1.8/lib/swissmedic-diff.rb:136:in `block in diff'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/swissmedic-diff-0.1.8/lib/swissmedic-diff.rb:312:in `block in each_valid_row'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rubyXL-3.1.2/lib/rubyXL/worksheet.rb:23:in `block in each'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rubyXL-3.1.2/lib/rubyXL/worksheet.rb:23:in `each'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rubyXL-3.1.2/lib/rubyXL/worksheet.rb:23:in `each'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/swissmedic-diff-0.1.8/lib/swissmedic-diff.rb:300:in `each_valid_row'
    /home/niklaus/.rvm/gems/ruby-2.1.2/gems/swissmedic-diff-0.1.8/lib/swissmedic-diff.rb:110:in `diff'
    /opt/src/oddb.org/src/plugin/swissmedic.rb:63:in `update'
    test/test_plugin/swissmedic.rb:100:in `test_diff_2014_july_to_i_problem'

As now my time to reproduce the error dropped from > 1h to 2,5 seconds, it should be a lot easier to fix the problem!

Fixed the problem with a monkey patch for the method first. Restored the latest.xml/xlsx files under data/xml and data/xls. Restarted the import_daily with the reimpoted database of thinpower of yesterday. Seems to be okay. Already received mails for FI, PI, Swissmedic XLS, Swissmedic Pharma, Co-Marketing, Patents, Lppv.

Pushed commits

Fix locally running yus and apache2 with mod_ruby

Will compile ruby 1.8.7 from source ans install it into /usr/local/ruby-1.8.7 with the following commands

autoconf
./configure --prefix=/usr/local/ruby-1.8.7
make
sudo make install

After thinking a bit longer I decided to use another way. cd /usr/portage && git checkout 8030f2b37512a03af3c07777b2f2c469d745d053 (as of 1 march) before Ruby 1.8.7 got dismissed. Followed by emerge --unmerge ruby rubygems; eselect ruby set ruby18; emerge ruby rubygem; /usr/bin/gem18 install sbsm yus. Restarted Apache and yus and oddb-ci2 is working well again.

Fix running export_csv

Got the following errror email

Error: NameError
Message: uninitialized constant Zip::ZipOutputStream
Backtrace:
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:52:in `compress'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/csv_export.rb:65:in `_export_drugs'
/var/www/oddb.org/src/plugin/csv_export.rb:30:in `export_drugs'
/var/www/oddb.org/src/util/exporter.rb:103:in `block in export_csv'
/var/www/oddb.org/src/util/exporter.rb:424:in `call'
/var/www/oddb.org/src/util/exporter.rb:424:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:102:in `export_csv'
jobs/export_csv:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/export_csv:12:in `<module:Util>'
jobs/export_csv:11:in `<module:ODDB>'
jobs/export_csv:10:in `<main>'

Found two occurrences of Zip::ZipOutputStream and replaced them with their correct name Zip::OutputStream. Running export_csv again. Completed without error. Running export_daily. Completed without problems. Pushed commit Fix problem after upgrading to rubyzip 1.1.6

  • Add export_galenic

Created a helper script to export the manually created galenic groups and forms. See commit Added export_galenic, Attach:galenic_groups.yaml.txt and Attach:galenic_forms.yaml.txt

view · edit · sidebar · attach · print · history
Page last modified on July 08, 2014, at 08:09 PM