view · edit · sidebar · attach · print · history

Index>

20160224-fix-import-daily

Summary

  • Fix import-daily to parse correctly new new items

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)

Fix import-daily to parse correctly new new items

Found the following error in my local import_daily 

Plugin: ODDB::BsvXmlPlugin
Error: NoMethodError
Message: undefined method `values' for "name":String
@report: {:name_base=>"Bexin", :name_descr=>"Sirup 25 mg/10ml ", :swissmedic_no5_bag=>"39639", :deductible=>:deductible_g, :generic_type=>:unknown, :swissmedic_no5_oddb=>"39639", :swissmedic_no8_bag=>"39639025"}
Backtrace:
/var/www/oddb.org/src/model/registration.rb:241:in `block in packages'
/var/www/oddb.org/src/model/registration.rb:240:in `each'
/var/www/oddb.org/src/model/registration.rb:240:in `inject'
/var/www/oddb.org/src/model/registration.rb:240:in `packages'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/bsv_xml.rb:406:in `tag_end'
/usr/local/lib/ruby/1.9.1/rexml/parsers/streamparser.rb:26:in `parse'
/usr/local/lib/ruby/1.9.1/rexml/document.rb:205:in `parse_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:915:in `update_preparations'
/var/www/oddb.org/src/plugin/bsv_xml.rb:601:in `block (2 levels) in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry.rb:500:in `get_input_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:601:in `block in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:139:in `block in foreach'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:138:in `foreach'
/var/www/oddb.org/src/plugin/bsv_xml.rb:596:in `_update'
/var/www/oddb.org/src/plugin/bsv_xml.rb:591:in `update'
/var/www/oddb.org/src/util/updater.rb:285:in `block in update_bsv'
/var/www/oddb.org/src/util/updater.rb:539:in `call'
/var/www/oddb.org/src/util/updater.rb:539:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:283:in `update_bsv'
/var/www/oddb.org/src/util/updater.rb:205:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

My problems stemmed from the fact that I naively assumed that the size of the UTF-8 encoded HTML size is equal to the file size.

Having even more code common for analysing FI and PI.

Had the same problem with the UTF-8 size when comparing the xml files in data/details.

Added 56079_pi_de.xml to test/data/xml/Aips_test.xml. Not test/test_plugin/text_info.rb begins to really check what is going on. Updating the statistics does not yet work correctly.

Comparing the content of the XML files seems not to work for some FIs with problematic names. Mail for FI arrives after 20 minutes. PI arrived even faster. Suspecting an error. Removing all files containing Tramal or 43788 in their names under data to force some reparsing.

Needed to improve stubbing to allow creating FI/PIs. Removing the details_dir before each run, allows me to check the statistics.

Now the counters seem to be okay. I am quite satisfied that finally test/test_plugin/text_info.rb does stress plugin/textinfo.rb in a correct way. Must fix some errors in test/test_plugin/text_info_swissmedicinfo.rb

Pushed commit Removed obsolete textinfo_news. Fix import_daily to read newest only

But something is still very slow. Therefore trying to profile the whole import using https://github.com/tmm1/perftools.rb. Added the lines to jobs/import_daily 

require 'perftools'
PerfTools::CpuProfiler.start("/tmp/import_daily_profile")
<..>
PerfTools::CpuProfiler.stop

Added gem 'perftools.rb' to the Gemfile. After running bundle install I am able to bundle exec jobs/import_daily. Looking the results is done using sudo -u apache bundle exec pprof.rb --text /tmp/import_daily_profile

view · edit · sidebar · attach · print · history
Page last modified on February 25, 2016, at 08:36 AM