Index>

20150624-refdatabase

Summary

• Adapt to changed URI of refdata in oddb2xml
• Adapt to changed URI of refdata in oddb.org

Commits

• https://github.com/ngiger/oddb2xml/commit/43e06c641d245c787588b95b60a46867cbf38302 Download refdata from new URL
• https://github.com/ngiger/oddb2xml/commit/6111b67bae3655fdbb80806f23ace9f8abb0aed4 Use TZ=UTC to fix error on travis-ci
• https://github.com/ngiger/oddb2xml/commit/784ad134c69a2441ca3d3b80da02b70adcd83ddc Try to avoid error Net::NTLM::VERSION::MAJOR

Index

• Adapt to changed URI of refdata in oddb2xml
• Adapt to changed URI of refdata in oddb.org

Keep in Mind for work to do

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
• The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
• creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
• Import via data/medreg_companies.yaml
• Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
• Fix galenic_forms when parsing swissmedic.xlsx
• Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
• Get updated ATC-codes from EPha for oddb.org, too.
• Display new fields (LABEL, MORE_INFO, CORRESP) for compositions in oddb.org.
• Use refdatabase for oddb.org, too.

Adapt to changed URI of refdata in oddb.org

Must port adaptations of oddb2xml to oddb.org.

Looking at ext/swissindex/src/swissindex.rb I remark that it implents support for read Pharma, NonPharma and Migel. As Migel is not part of the refdata I have to reorganize the code and move the Pharma/NonPharma to ext/refdata/src/refdata.rb and leave only Migel in ext/swissindex/src/swissindex.rb (renaming SwissindexNonpharma to SwissindexMigel). This means also renaming the daemons ch.oddb-swissindex_nonpharma -> ch.oddb-refdata_nonpharma, ch.oddb-swissindex_pharma ch.oddb-refdata_pharma and adding a new ch.oddb-swissindex_migel

Trying to make all unit tests pass on travis-ci. Unit in test/test_model fail on travis-ci. Why? Trying some changes.

Adapt to changed URI of refdata in oddb2xml

Reviewing result of test_options.rb before pushing commit. Running test_options did not generate any files when running with options -t md. There was no unit test for it. Therefore creating one and running bundle exec ruby bin/oddb2xml -t md --log --skip-download seem to work find. Therefore I think the test_options.rb script has a problem.

Same problem occurred when running with the following options:

• -x add
• -o
• -f xml
• -f dat --append
• -f dat --append -I 80
• -t md -c tar.gz

Skipping this problem for the time being.

Updated History.txt to document work done in the last three days. Pushed commit Download refdata from new URL

Fixed an error when running in a different timezone on travis.ci with commit Use TZ=UTC to fix error on travis-ci

Pushed commit Try to avoid error Net::NTLM::VERSION::MAJOR

Running oddb2xml -e consumes way too much time (around 4 hours compared to 45 minutes before). Why? Looking where we spend time:

• Downloading all files. 1 minute 10 seconds. Thats okay
• Build oddb_calc.xml takes 14 minutes
• prepare_articles takes about 4 seconds for 100 items. As we generate 166846 articles this takes about two hours and 26 minutes.
• build_articles took 26 minutes
• prepare_products took only about two minutes
• Total time was Took 11240 seconds, about 3 1/4 hours

Using ruby-prof (and patching builder.rb) to profile building 100 articles. Analysing output Attach:profile_build_graph.txt and Attach:profile_build.txt.

As it looks like sorting and selecting in the array takes most of the time, I try to use the keys (EAN) as integers and not as strings. This brought down the time to about 9500 seconds (two hours , as the speed-up mostly vanished when preprare_articles was accessing more elements. Relevant times are:

• prepare_articles starting with no articles. 1 hour 34 minutes
• build_article 42 minutes

Using FixNum for pharmacode hash, too. But I think I must rethink the way how I changed prepare_article. But not today.