Index>

20140423-correct-oddb2xml-2

Summary

• oddb2xml creates 20'000 (almost duplicated) entries.

Commits

• https://github.com/ngiger/oddb2xml/commit/b1377c28433bff352ff7f73fda8954e205b8e3e8 Added test target to rake
• https://github.com/ngiger/oddb2xml/commit/731c1315c4321dbd3f80c55055717c8e98c85e5f Added clean target to rake
• https://github.com/ngiger/oddb2xml/commit/14fca70d3ebab134c7b8fd0d41bacf89888568bc Added log option
• https://github.com/ngiger/oddb2xml/commit/db9a1a2a80002f1fec9f7515680e761a58412d1a Added spec test for duplicates when -e given
• https://github.com/ngiger/oddb2xml/commit/de542c290e7cbc70f06ad297b37822c6a47e4b2c Added logging, accelerated prepare_articles, fixed recognition of duplicates
• https://github.com/ngiger/oddb2xml/commit/6da58395cb6b2e85c76e0b8c8ccb159addc90a12 Bumped version to 1.8.0
• https://github.com/ngiger/oddb2xml/commit/c906e0a2a9dbc48f9bf0529102bfee29acde66f4 Bumped version to 1.8.0
• https://github.com/ngiger/oddb2xml/commit/149d35d38dac3b5fb88a26068567fdbaf52b39ac Fix problem with empty substances and limitation when running -e
• https://github.com/ngiger/oddb2xml/commit/40bfe384660e115f51e712c1ab855279bce1fa4b More logging and exit 2 unless substances and limitations okay for extended

Index

• oddb2xml creates 20'000 (almost duplicated) entries

Keep in Mind

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• Search nach Anwendung (indications), z.B. Konjunktivitis, does not report all occurrences in the section "Anwendung" of Fachinfo. Probably index is corrupted or not set up correctly.
• Error: Patents with could not connect to www.swissreg.ch: #<Net::HTTPInternalServerError:0x007f8a7d69bb58>
• vagrant-oddb.org: cleanup installation for yus with ruby 1.8, logrotate.conf and local vhosts for tests

---

Personal remarks since yesterday I had to reboot my cablecom modem four times as my internet connection got lost. Reported by phone today (waited almost 10 minutes to get connected to a real person).

oddb2xml creates 20'000 (almost duplicated) entries

The following errors must be fixed:

1. oddb_substances.xml and oddb_interactions.xml are empty when running oddb2xml -e
2. oddb_xml -e create 20'000 duplicates
3. oddb_xml -e takes very long (about an hour or more)

/oddb2xmlLooking around why we have duplicates I found the following stuff for Zyvoxid Filmtab in data/download:

data/download/oddb2xml_zurrose_transfer.dat:1122465312ZYVOXID Filmtabl 600 mg 10 Stk                    096114108275100A080190076805555800542
ata/download/swissindex_Pharma_DE.xml-      <ITEM DT="2013-06-22T00:00:00">
data/download/swissindex_Pharma_DE.xml-        <GTIN>7680555580054</GTIN>
data/download/swissindex_Pharma_DE.xml-        <PHAR>2465312</PHAR>
data/download/swissindex_Pharma_DE.xml-        <STATUS>A</STATUS>
data/download/swissindex_Pharma_DE.xml-        <STDATE>2002-01-16T00:00:00</STDATE>
data/download/swissindex_Pharma_DE.xml-        <LANG>DE</LANG>
data/download/swissindex_Pharma_DE.xml:        <DSCR>ZYVOXID Filmtabl 600 mg</DSCR>
data/download/swissindex_Pharma_DE.xml-        <ADDSCR>10 Stk</ADDSCR>
data/download/swissindex_Pharma_DE.xml-        <ATC>J01XX08</ATC>
data/download/swissindex_Pharma_DE.xml-        <COMP>
data/download/swissindex_Pharma_DE.xml-          <NAME>Pfizer AG</NAME>
data/download/swissindex_Pharma_DE.xml-          <GLN>7601001010604</GLN>
data/download/swissindex_Pharma_DE.xml-        </COMP>
data/download/swissindex_Pharma_DE.xml-      </ITEM>

Therefore I asking myself why we get a duplicate with STATUS I?

Added new task clean and test to ease testing with commits Added clean target to rake and Added test target to rake.

The reason is that I checked for entries in @index instead of searching in @articles. But I think prepare_articles will be the most time consuming operation. Adding a Oddb2xml.log function to log important options.

Added a spec test to check for duplicate entries of ZYVOXID. Made spec pass. Runnning full import now and waiting for completion.

Reworked prepare_articles and made it a lot faster. Is no longer the bootleneck. Now the bottleneck is build_article.

• When calling oddb2xml -a nonpharma we need about 62 seconds to emit the first 10'000 (of 43689) articles.
• When calling oddb2xml -e we need about 15 minutes seconds to emit the first 10'000 (of 161200) articles. Observing the logging we see that it often gets stuck. Maybe we have not enough memory or gcc problems. The process is using 3071M Virt and 2424M Res memory. Estimated timeof arrival is therefore 15x15 = 225 minutes or about 4 hours.

Pushed commits Added logging, accelerated prepare_articles, fixed recognition of duplicates and Bumped version to 1.8.0

Limitations and substances are still empty when running oddb2xml -e. Fixed with commits Fix problem with empty substances and limitation when running -e and More logging and exit 2 unless substances and limitations okay for extended

The last two commits also brought down the time for running oddb2xml -e to about 35 minutes.