view · edit · sidebar · attach · print · history

20130821-wrong-fachinfo

<< | Index | >>


Summary

  • Run check and update only for IKSNR found in Packungen.xls of swissmedic

Commits

Index


Run check and update only for IKSNR found in Packungen.xls of swissmedic

  • Skip check if registration.inactive?
  • Added helper function read_packages to read in the iksnr, seq, name_base from Packungen.xls to double check errors.
  • Adding a helper hash @error_reasons to facilitate reporting (counting and categorising)
    • Flagged_as_inactive = 'flagged as nactive? in ODDB but found in packages.xls'
    • Mismatch_name_2_xls = 'name_base in ODDB differs from packages.xls'
    • Iksnr_only_oddb = 'iksrn found in ODDB.db but in packages.xls'
    • Fi_iksnrs_mismatched_to_aips_xml = 'fi-iksnrs should match ID froms Aips.xml'
    • Pi_iksnrs_mismatched_to_aips_xml = 'pi-iksnrs should match ID froms Aips.xml'
    • Mismatch_reg_name_to_fi_name = 'name_base in registration should match name_base in fachinfo'
    • Mismatch_reg_name_to_pi_name = 'name_base in registration should match name_base in patinfo'

Found more than 100 iksnr in ODDB.ch which are not in Packungen.xls. E.g. 45786 Legendal

Had to run the check various times to fix typos in Ruby code. Suddenly I got

/usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml/document.rb:54:in `read_memory': failed to allocate memory (NoMemoryError)
        from /usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml/document.rb:54:in `parse'
        from /usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml.rb:33:in `XML'
        from /var/www/oddb.org/src/plugin/text_info.rb:1035:in `swissmedicinfo_xml'
        from /var/www/oddb.org/src/plugin/text_info.rb:1073:in `check_swissmedicno_fi_pi'
        from /var/www/oddb.org/src/util/updater.rb:512:in `block in update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:488:in `call'
        from /var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
        from /var/www/oddb.org/src/util/updater.rb:506:in `update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:309:in `check_swissmedicno_fi_pi'
        from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:15:in `block in <module:Util>'
        from /var/www/oddb.org/src/util/job.rb:40:in `call'
        from /var/www/oddb.org/src/util/job.rb:40:in `run'
        from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:12:in `<module:Util>'
        from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:11:in `<module:ODDB>'
        from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:10:in `<main>'

Looking at various ruby process via /proc/<id>/status I found

cat /proc/32171/status
Name:   import_daily
State:  S (sleeping)
Tgid:   32171
Pid:    32171
PPid:   32170
TracerPid:      0
Uid:    81      81      81      81
Gid:    81      81      81      81
FDSize: 64
Groups: 81 998 
VmPeak:  5141512 kB
VmSize:  5141508 kB

That is more than 5 GB for VM!

The report mail begins to look good. Some minor corrections and run it again. And the check needs only about 4,5 minutes. Quite acceptable.

Now running update, then a check again.

Looked at

  • http://oddb-ci.dyndns.org/de/gcc/fachinfo/reg/00284. Disques vertebraux Sérocytol. This is an older import as it has a lot wrong italics.
  • 60605: Novo-Helisen® Depot enthält in AIPS.xml <authNrs>60590, 60595, 60597, 60611</authNrs>, aber in der Fachinfo findet (auf Swissmedicinfo)

60590, 60591, 60592, 60593, 60594, 60595, 60597, 60598, 60599, 60600, 60601, 60602, 60603, 60604, 60605, 60606, 60607, 60608, 60609, 60610, 60611

When running the update_job I get an error when running the invoice-update (Why should I run it here?) Stack-trace is

Plugin: ODDB::TextInfoPlugin
Error: IndexError
Message: Index 0 is out of range
Backtrace:
/usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:37:in `[]'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:37:in `block in column_info'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:25:in `upto'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:25:in `column_info'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/statement.rb:83:in `column_info'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:158:in `column_names'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:27:in `initialize'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:81:in `new'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:81:in `execute'
/usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:128:in `select_all'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:39:in `block in method_missing'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:29:in `next_connection'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:38:in `method_missing'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:508:in `restore_collection'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:236:in `fetch_collection'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:65:in `block in bulk_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `each'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `bulk_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:56:in `bulk_fetch'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:260:in `fetch_collection'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:640:in `restore_object'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:605:in `load_object'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:226:in `block in fetch'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:225:in `fetch'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:49:in `odba_receiver'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/util/oddbapp.rb:216:in `clean_invoices'
/var/www/oddb.org/src/util/oddbapp.rb:1440:in `clean'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:140:in `block (3 levels) in run_cleaner'
<internal:prelude>:10:in `synchronize'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:139:in `block (2 levels) in run_cleaner'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:137:in `loop'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:137:in `block in run_cleaner'

When running the update job. Received the following error, but updating continued!

Plugin: ODDB::TextInfoPlugin
Error: DBI::ProgrammingError
Message: ERROR:  prepared statement "ruby-dbi:Pg:700110282075801377079174.9888122" already exists

The at the end (*) the job terminated with

update_patinfo Cerumenol® iksnrs_from_xml ["42694"] empty false
(druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:736:in `rescue in block in open': druby://oddb.niklaus.org:56865 - #<Errno::ETIMEDOUT: Connection timed out - connect(2)> (DRb::DRbConnError)
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:730:in `block in open'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `each'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `open'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1191:in `initialize'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `new'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `open'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1087:in `block in method_missing'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1105:in `with_friend'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1086:in `method_missing'
        from (druby://localhost:10000) /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/odba.rb:38:in `unpeer'
        from (druby://localhost:10000) /var/www/oddb.org/src/util/oddbapp.rb:1610:in `unpeer_cache'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1508:in `perform'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `loop'
        from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
        from /var/www/oddb.org/src/util/job.rb:50:in `run'
        from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:12:in `<module:Util>'
        from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:11:in `<module:ODDB>'
        from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:10:in `<main>'
update_patinfo.pointer1 42694 
store_patinfo existing  -> ptr false languages [:fr] reg.iksnr 42694
update_patinfo.pointer2 42694 :!patinfo,1911.
update_patinfo Cerumenol® iksnr 42694 update
import_info: {:fi=>"fachinfo", :pi=>"patinfo"} names {:de=>{:fi=>[nil], :pi=>["Cerumenol®"]}, :fr=>{:fi=>[nil], :pi=>["Cerumenol®"]}} iksnrs []
import_swissmedicinfo_by_iksnrs iksnr "43216" {}
extract_matched_name 43216 fachinfo as 'fi' lang 'de' path is //medicalInformation[@type='fi' and @lang='de']/authNrs returns Xanax®/- retard/- SL
extract_matched_name 43216 patinfo as 'pi' lang 'de' path is //medicalInformation[@type='pi' and @lang='de']/authNrs returns Xanax®/- retard/- SL

Found only 173 of expect "import_swissmedicinfo_by_iksnrs iksnr" (first wer 00035, 00274, last was 43216 Xanax 0,25 mg,). Next would have been 43225

To improve logging added helper function puts_sync which add a timestamp and $stdout.flush to each puts statement to see more clearly where we are when the drb-error comes.

Also with a sleep of 4 minutes between check and re-import we are able to import about 545 iksnrs. But still get the drb connection error. Remove the sleep to commit. Committed: Improved checking FI/PI against Aips.xml and Packungen.xls

When running update-job without a running oddbd, the job hangs for more than 13 minutes at

2013-08-21 13:42:11 +0200: import_swissmedicinfo_by_iksnrs iksnr "18056" {}
2013-08-21 13:42:11 +0200: extract_matched_name 18056 fachinfo as 'fi' lang 'de' path is //medicalInformation[@type='fi' and @lang='de']/authNrs returns Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung
2013-08-21 13:42:11 +0200: extract_matched_name 18056 fachinfo as 'fi' lang 'fr' path is //medicalInformation[@type='fi' and @lang='fr']/authNrs returns Euphyllin® retard N Comprimés filmés, Euphyllin® N Solution injectable
2013-08-21 13:42:11 +0200: import_info: {:fi=>"fachinfo", :pi=>"patinfo"} names {:de=>{:fi=>["Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung"], :pi=>[nil]}, :fr=>{:fi=>["Euphyllin® retard N Comprimés filmés, Euphyllin® N Solution injectable"], :pi=>[nil]}}
2013-08-21 13:42:11 +0200: parse_and_update {:de=>"Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung"} fachinfo
2013-08-21 13:42:11 +0200: parse_and_update: calls parse_fachinfo dist /var/www/oddb.org/data/html/fachinfo/de/Euphyllin_ retard N Filmtabletten, Euphyllin_ N Injektionsl_sung_swissmedicinfo.html iksnrs_from_xml ["18056", "36694"] Euphyllin_ retard N Filmtabletten, Euphyllin_ N Injektionsl_sung_swissmedicinfo.html, name Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung de title Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung
2013-08-21 13:42:12 +0200: update_fachinfo Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung iksnr ["18056", "36694"]
2013-08-21 13:42:12 +0200: update_fachinfo Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung iksnr 18056 store_fachinfo {}

Uses about 3.2 GB of virtual memory (Virtual machine has 8 GB allocated). This should not yet be a problem.

Stopping the job. Running an import of all mentions iksrns (833). Observing memory usage while importing. Stays at 1540 MB for quite some time. After some 600 imported iksnr mounted to 1756 MB. Import completed successfully. After sending the email the job is busy for more than 7 minutes. Memory usage mounts to 1824MB, CPU-load is at 100 (only one thread), postgresql is not busy.

Searched for
Stored 312 Fachinfos
Ignored 0 Pseudo-Fachinfos
Ignored 0 up-to-date Fachinfo-Texts
Stored 328 Patinfos
Ignored 0 up-to-date Patinfo-Texts
Checked 0 companies
Unknown Iks-Numbers: 3
Synalar®-N (29300)
Bicarbonate de sodium B. Braun (33709)
STALORAL Pollen® (69870)

The job did not complete because of a missing return true. After adding it, a simple import job sent the email, but also had the error "#<Errno::ECONNREFUSED: Connection refused - connect(2)> (DRb::DRbConnError)". The error comes from job.rb, which after removing the log/job.pid, wants to call "system.unpeer_cache ODBA.cache unless opts[:readonly]". In my opinion we should add a rescue Errno::ECONNREFUSED clause to system.unpeer_cache as there is also on when calling system.peer_cache

Improved check with don't reimport errors of type 'iksrn found in ODDB.db but not in packages.xls'. Clarified some error messages. Running check-job again. Looks better. Asked Zeno whether it is okay.

Run update-job again. Imports only 486 iksnrs, as clearly missing not. Observing memory usage. Is at 1928 MB while checking. Goes up to 3178 MB when running import, at the end of import is at 3592 MB. Postgres

Will continue next monday.

After telephone with Zeno, pushed commit to fix a typo Fix typo sw -> authNrs

view · edit · sidebar · attach · print · history
Page last modified on August 21, 2013, at 06:20 PM