Index>

20160222-fix-import-daily

Summary

• Fix import-daily to parse correctly new new items
• Analyse PRODNO in oddb_product.xml

Commits

Index

• Fix import-daily to parse correctly new new items
• Analyse PRODNO in oddb_product/article.xml

Keep in Mind for work to do

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
• The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
• creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
• Import via data/medreg_companies.yaml
• Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
• Fix galenic_forms when parsing swissmedic.xlsx
• Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
• Get updated ATC-codes from EPha for oddb.org, too.
• Use refdatabase for oddb.org, too.
• Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
• When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
• Order of entering search type and value should not matter. Both should show long URL with search
• Remove parser for minifi (but keep the minifi)

Fix import-daily to parse correctly new new items

Must fix a local error where looking at some patinfo created a nil access. Running import_daily again.

Why some files don't get created? Analysing

2016-02-22 19:10:41 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1140:in `parse_fachinfo': parse_fachinfo 1140:  no html_name for fachinfo for #<struct Struct::SwissmedicMetaInfo iksnr="14770", authNrs=["14769", "14770", "19932", "56512"], atcCode=nil, title="Kétalgine®", authHolder=nil, substances=nil, type="fi", lang="fr", informationUpdate=nil, refdata=nil, xml_file="/var/www/oddb.org/data/details/14770_fi_fr.xml">

grep -c Kétalgine data/details/* | grep -v :0
data/details/14769_fi_fr.xml:1
data/details/14770_pi_fr.xml:1

Added some more debugging output. Verified that not all FI get reparsed (but many are, because we now use the file size as indicator whether it was changed). Restarted import_daily.

Wondering why I don't find an entry for IKSNR 65856 in Packungen.XLSX where in swissmedicinfo.ch we can find DIBASE 10'000.

Now I successfully detect the new/updated FI/PI, but I don't get the correct file to parse. Okay. I found the culprit. Must get the first authNr from the metainfo, not the current one. No I must sort them to consistent. E.g. found <authnrs>50710, 54157, 53591, 50709</authnrs>

The new text from swissmedicinfo.ch do not appear immediately in the AipsDownload. Therefore must create a similar XML-file in this case.

Somehow also no Changelog information gets added. Why? Was only considered if using reparse and not with newest. Similar problem for creating new registrations.

Analyse PRODNO in oddb_product/article.xml

When running with the -e option we have 13878 (8513 uniq ones from 002771 to 659731)occurences of oddb_article.xml and 15698 (9810 uniq ones from 002771 up to 660521) in oddb_product.xml.

Creating a uniq constraint for the XSD was easy. Just added at the end of the definition of <xs:element name="PRODUCT">

    <xs:unique name="unique-prodno">
        <xs:selector xpath="ns1:PRD"/>
        <xs:field xpath="ns1:PRODNO"/>
    </xs:unique>

And now a call to xslvalidate throws an error like

 xsdvalidate oddb2xml.xsd oddb_product.xml | head
oddb_product.xml NOT valid.
/opt/src/oddb2xml/oddb_product.xml:32251:3: error: cvc-identity-constraint.4.1: Duplicate key '2781' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32264:3: error: cvc-identity-constraint.4.1: Duplicate key '2791' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32303:3: error: cvc-identity-constraint.4.1: Duplicate key '2941' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32329:3: error: cvc-identity-constraint.4.1: Duplicate key '3131' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32381:3: error: cvc-identity-constraint.4.1: Duplicate key '3381' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32626:3: error: cvc-identity-constraint.4.1: Duplicate key '6271' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32646:3: error: cvc-identity-constraint.4.1: Duplicate key '6281' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32693:3: error: cvc-identity-constraint.4.1: Duplicate key '6411' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'
/opt/src/oddb2xml/oddb_product.xml:32738:3: error: cvc-identity-constraint.4.1: Duplicate key '6701' for unique constraint 'unique-prodno@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration'