< Niklaus.20230103-ruby-320-ydim-oekk | Index | Niklaus.20220330-ruby-311-on-thinpower >>

20220411-textinfo-problems

Summary

• Problem importing some textinfos (pat- and fachinfo)
• Problem with import_bsv
• Keep in Mind
• We should investigate and (manually or whatever) fix all these warnings like ODBA::Stub was unable to replace #701017 from ODDB::Package:#230422. raise OdbaError instead of catching the exceptions

Commits

• ix some errors. Revert Remove some lines which came from up
• Recover from OdbaError when deleting invalid pointers
• Remove some lines which came from update_conflict of pharmacode
• Handle pac or sl_entry is nil correctly
• Fix import if sl_entry is a text chapter

Index:

• Port oddb.org to Ruby 3.1.1 on thinpower
• Problem with import_bsv

Problem with import_bsv

See also https://github.com/zdavatz/oddb.org/issues/158

It is not clear to me, why or whether using Ruby 3.1.1 is reponsible that I had to fix the following problem in the bsv_xml.rb

• Handle case where sl_entry was a text chapter
• Could not handle some cases the sl_entry or pac was nil
• Had to recover from OdbaError when deleting invalid pointers )eg. bsv_xml.rb creatd an sl_entry for package which did not yet exist
• The 4 lines (around nr 300) after ## don't take the Swissmedic-Category unless it's missing in the DB are responsible for updating the sl_entry. They happened to trigger a OdbaError, when the sl_entry itself was a OdbaError, this was the one of the difficulties to distinguish.

To test it I used the following steps on oddb-ci2:

• drop database oddb_ch;
• Load db_dump from thinpower as per April 30
• Used data/xml/XMLPublications-2022.04.13.zip data/xml/XMLPublications-latest.zip
• Ran sudo -u apache bundle-311 exec ruby-311 jobs/import_bsv (or import_bsv_only)

Proposing the following manual fixes via bin/admin ad visiting https://ch.oddb.org/de/gcc/search/zone/drugs/search_query/53075/search_type/st_registration? produced also a OdbaError.

# showing the problem
ch.oddb> registration('53075').sequence('01').packages.keys
-> ["010", "029"]
ch.oddb> registration('53075').sequence('01').package('010').sl_entry
-> ODBA::OdbaError
ch.oddb> registration('53075').sequence('01').package('029').sl_entry
-> ODBA::OdbaError
# next commands to fix the problem
registration('53075').sequence('01').package('010').sl_entry =nil
registration('53075').sequence('01').package('010').odba_store
registration('53075').sequence('01').package('029').sl_entry =nil
registration('53075').sequence('01').package('029').odba_store

On thinpower starting ch.oddb org showed the following error which was avoided by commenting out line 1474 of src/util/oddbapp.rb containing @system.odba_store. Was this the reason that saving the rss-feeds fails?

Port oddb.org to Ruby 3.1.1 on thinpower

Running import_daily and sudo -u apache bundle-311 exec ruby-311 jobs/update_textinfo_swissmedicinfo --target=fi --reparse 66649 failed both with an error like this

Plugin: ODDB::TextInfoPlugin

Error: NoMethodError
Message: undefined method `text' for #<ODDB::PatinfoDocument:0x00007f2cb91dcf80 @iksnrs=#<ODBA::Stub:3366440#39162757 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @name=#<ODBA::Stub:3366460#39162758 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @company=#<ODBA::Stub:3366480#39162759 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @galenic_form=nil, @effects=#<ODBA::Stub:3366500#39162760 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @amendments=#<ODBA::Stub:3366520#39162761 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @contra_indications=#<ODBA::Stub:3366540#39162762 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @precautions=#<ODBA::Stub:3366560#39162763 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @pregnancy=#<ODBA::Stub:3366580#39162764 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @usage=#<ODBA::Stub:3366600#39162765 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @unwanted_effects=#<ODBA::Stub:3366620#39162766 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @general_advice=#<ODBA::Stub:3366640#39162767 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @composition=#<ODBA::Stub:3366660#39162768 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @packages=#<ODBA::Stub:3366680#39162769 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @distribution=nil, @fabrication=nil, @date=#<ODBA::Stub:3366700#39162770 @odba_class=ODDB::Text::Chapter @odba_container=3366360#38363236>, @odba_id=38363236, @odba_persistent=true, @odba_observers=[], @revision=2018-06-01 07:47:31.097304108 +0200, @oid=38363236, @odba_prefetch=nil>

                                odba_instance.send(meth_symbol, *args, &block)
                                             ^^^^^
Did you mean?  test
Backtrace:
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
(eval):1:in `store_fachinfo'
/var/www/oddb.org/src/plugin/text_info.rb:147:in `eval'
/var/www/oddb.org/src/plugin/text_info.rb:147:in `store_fachinfo'
/var/www/oddb.org/src/plugin/text_info.rb:265:in `update_fachinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:1291:in `parse_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1494:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1493:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1493:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:537:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:493:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:531:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:329:in `update_textinfo_swissmedicinfo'
jobs/update_textinfo_swissmedicinfo:39:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:41:in `run'
jobs/update_textinfo_swissmedicinfo:12:in `<module:Util>'
jobs/update_textinfo_swissmedicinfo:11:in `<module:ODDB>'
jobs/update_textinfo_swissmedicinfo:10:in `<main>'

66649 is Dupixent. The corresponding patinfo https://ch.oddb.org/de/gcc/patinfo/reg/66649/seq/01/pack/001 the name as Dupixent@ and has correct diff. The fachinfo https://ch.oddb.org/de/gcc/fachinfo/reg/66649 gives the name as Tréprostinil OrPha which is wrong. Also no diffs for the textinfo are given!

Using bin/admin I found several inconsitencies

 $found = registrations.find_all{|iksnr, reg| reg.fachinfo && reg.fachinfo.name && reg.fachinfo.name.size > 120}
ch.oddb> $found.size
-> 5
$found.collect{ |iksnr, reg| iksnr + " size: "+ reg.fachinfo.name.size.to_s }
-> ["66649 size: 12327", "55297 size: 7371", "57489 size: 7371", "62998 size: 12276", "65334 size: 14575"]
ch.oddb> $found.collect{ |iksnr, reg| iksnr + " revision: "+ reg.revision.to_s[0..10] }
-> ["66649 revision: 2021-05-12 ", "55297 revision: 2020-05-05 ", "57489 revision: 2022-04-01 ", "62998 revision: 2019-09-05 ", "65334 revision: 2019-02-14 "]

The following patch seems to fix it. Attach:Fix-textinfo-if-no-correct-text-is-available.patch Δ But reparsing takes some time and I will continue tomorrow

Now running import_daily I got the following error in patinfo

Plugin: ODDB::TextInfoPlugin

Error: NoMethodError
Message: undefined method `size' for #<ODDB::PatinfoDocument:0x00007f28b519a070 @iksnrs=#<ODBA::Stub:28421960#48248031 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @name=#<ODBA::Stub:28421980#48248032 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @company=#<ODBA::Stub:28422000#48248033 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @galenic_form=nil, @effects=#<ODBA::Stub:28422020#48248034 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @amendments=nil, @contra_indications=#<ODBA::Stub:28422040#48248035 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @precautions=nil, @pregnancy=#<ODBA::Stub:28422060#48248036 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @usage=#<ODBA::Stub:28422080#48248037 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @unwanted_effects=#<ODBA::Stub:28422100#48248038 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @general_advice=#<ODBA::Stub:28422120#48248039 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @composition=#<ODBA::Stub:28422140#48248040 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @packages=#<ODBA::Stub:28422160#48248041 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @distribution=nil, @fabrication=nil, @date=#<ODBA::Stub:28422180#48248042 @odba_class=ODDB::Text::Chapter @odba_container=28421940#48246108>, @odba_id=48246108, @odba_persistent=true, @odba_observers=[], @revision=2022-03-31 11:38:41.548597302 +0200, @oid=48246108, @odba_prefetch=nil>

                                odba_instance.send(meth_symbol, *args, &block)
                                             ^^^^^
Backtrace:
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:292:in `store_patinfo_change_diff'
/var/www/oddb.org/src/plugin/text_info.rb:322:in `store_package_patinfo'
/var/www/oddb.org/src/plugin/text_info.rb:353:in `store_patinfo_for_all_packages'
/var/www/oddb.org/src/plugin/text_info.rb:386:in `block in update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:385:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:385:in `update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:1306:in `parse_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1500:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1499:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1499:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:537:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:493:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:531:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:329:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:210:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:41:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

In the log I get some more information

2022-04-12 01:41:10 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1212:in `extract_html': IKSNR 67284: File.size(/var/www/oddb.org/data/html/pi/fr/AJOVY_ 225 mg solution injectable en seringue pr_remplie.html) is 223035
2022-04-12 01:41:10 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1275:in `parse_textinfo': parse_textinfo 1275 /var/www/oddb.org/data/html/pi/fr/AJOVY_ 225 mg solution injectable en seringue pr_remplie.html does is not the same: ["67284"]
2022-04-12 01:41:10 +0200: /var/www/oddb.org/src/plugin/text_info.rb:362:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="67284", authNrs=["67284"], atcCode=nil, title="AJOVY® 225 mg solution injectable en seringue préremplie", authHolder="Teva Pharma AG", substances="Frémanezumab", type="pi", lang="fr", informationUpdate="03.2022", refdata=nil, xml_file="/var/www/oddb.org/data/details/67284_pi_fr_AJOVY__225_mg_solution_injectable_en_seringue_pr_remplie_problem.xml", same_content_as_xml_file=true> ["fr"]

Visiting http://oddb-ci2:8012/fr/gcc/patinfo/reg/67284/seq/01/pack/001 gives me a also the same NoMethodError. But under https://ch.oddb.org/fr/gcc/show/patinfo/67284/01/001/diff/09.04.2022 I see the correct changes.

Loading DB-Dump from thinpower of april 08 to see whether I have the same problem. Importing the dump I see some errors while building the indices

COPY 20555
ERROR:  duplicate key value violates unique constraint "collection_pkey"
DETAIL:  Key (odba_id, key)=(13255752, 04086900) already exists.
CONTEXT:  COPY collection, line 1
COPY 50594
COPY 13119
COPY 102944
COPY 5025
COPY 5023
COPY 7981
COPY 7748
COPY 1227
COPY 5555
COPY 3751
COPY 3751
COPY 10293
COPY 10292
COPY 731
COPY 731
ERROR:  duplicate key value violates unique constraint "object_pkey"
DETAIL:  Key (odba_id)=(795273) already exists.
CONTEXT:  COPY object, line 1
ERROR:  duplicate key value violates unique constraint "object_connection_pkey"
DETAIL:  Key (origin_id, target_id)=(3, 100) already exists.
CONTEXT:  COPY object_connection, line 1
COPY 473
COPY 2312

Now I get the following error

admin@ywesee.com

09:11 (vor 1 Stunde)

an mich
Plugin: ODDB::TextInfoPlugin

Error: NoMethodError
Message: undefined method `odba_store' for nil:NilClass

      @descriptions.odba_store
                   ^^^^^^^^^^^
Did you mean?  odba_instance
Backtrace:
/var/www/oddb.org/src/model/patinfo.rb:53:in `odba_store'
/var/www/oddb.org/src/plugin/text_info.rb:329:in `store_package_patinfo'
/var/www/oddb.org/src/plugin/text_info.rb:353:in `store_patinfo_for_all_packages'
/var/www/oddb.org/src/plugin/text_info.rb:386:in `block in update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:385:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:385:in `update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:1306:in `parse_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1500:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1499:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1499:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:537:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:493:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:531:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:329:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:210:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:41:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

Log says

<...>
2022-04-12 09:11:40 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1275:in `parse_textinfo': parse_textinfo 1275 /var/www/oddb.org/data/html/pi/fr/AJOVY_ 225 mg solution injectable en stylo pr_rempli.html does is not the same: ["67843"]
2022-04-12 09:11:40 +0200: /var/www/oddb.org/src/plugin/text_info.rb:362:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="67843", authNrs=["67843"], atcCode=nil, title="AJOVY® 225 mg solution injectable en stylo prérempli", authHolder="Teva Pharma AG", substances="Frémanezumab", type="pi", lang="fr", informationUpdate="03.2022", refdata=nil, xml_file="/var/www/oddb.org/data/details/67843_pi_fr_AJOVY__225_mg_solution_injectable_en_stylo_pr_rempli_problem.xml", same_content_as_xml_file=true> ["fr"]
2022-04-12 09:11:40 +0200: /var/www/oddb.org/src/plugin/text_info.rb:298:in `store_patinfo_change_diff': store_patinfo_change_diff: fr 48248466 eql? false size 0 -> 1
2022-04-12 09:11:41 +0200: /var/www/oddb.org/src/plugin/text_info.rb:347:in `store_package_patinfo': called odba_store 67843/01/001: fr AJOVY® 225 mg solution injectable en stylo prérempli change_diff
2022-04-12 09:11:41 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1212:in `extract_html': IKSNR 68280: File.size(/var/www/oddb.org/data/html/pi/de/Plegridy_ Fertigspritze.html) is 379825
2022-04-12 09:11:41 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1275:in `parse_textinfo': parse_textinfo 1275 /var/www/oddb.org/data/html/pi/de/Plegridy_ Fertigspritze.html does is not the same: ["68280"]

This is weird, because the code fragment in text_info.rb says

        package.patinfo = @app.create_patinfo
        package.patinfo.odba_store

Looks like in src/model/patinfo we must not call odba_store for @descriptions if they are nil.

Looking at the history I see that I had introduced this erro february. See line 328 in https://github.com/zdavatz/oddb.org/commit/aa2144653e98ff8acd73328abc29b1a481a8b49f#diff-c0279c0a948325425a11cc056234f13529df0feae9beef0a59378e4445731c02R323

Was easy to fix. Waiting for import_daily to finish again, but it looks better as I've seen already 6 occurrences of create patinfo. Eg. 2022-04-12 13:30:17 +0200: /var/www/oddb.org/src/plugin/text_info.rb:346:in `store_package_patinfo': called odba_store 68280/00/000: fr Plegridy™ Seringue préremplie created patinfo

Okay.Import_daily finished without any problems.