<< | Index | >>
Found more than 100 iksnr in ODDB.ch which are not in Packungen.xls. E.g. 45786 Legendal
Had to run the check various times to fix typos in Ruby code. Suddenly I got
/usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml/document.rb:54:in `read_memory': failed to allocate memory (NoMemoryError)
from /usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml/document.rb:54:in `parse'
from /usr/local/lib64/ruby/gems/1.9.1/gems/nokogiri-1.5.9/lib/nokogiri/xml.rb:33:in `XML'
from /var/www/oddb.org/src/plugin/text_info.rb:1035:in `swissmedicinfo_xml'
from /var/www/oddb.org/src/plugin/text_info.rb:1073:in `check_swissmedicno_fi_pi'
from /var/www/oddb.org/src/util/updater.rb:512:in `block in update_notify_simple'
from /var/www/oddb.org/src/util/updater.rb:488:in `call'
from /var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
from /var/www/oddb.org/src/util/updater.rb:506:in `update_notify_simple'
from /var/www/oddb.org/src/util/updater.rb:309:in `check_swissmedicno_fi_pi'
from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:15:in `block in <module:Util>'
from /var/www/oddb.org/src/util/job.rb:40:in `call'
from /var/www/oddb.org/src/util/job.rb:40:in `run'
from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:12:in `<module:Util>'
from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:11:in `<module:ODDB>'
from /var/www/oddb.org/jobs/check_swissmedicno_fi_pi:10:in `<main>'
Looking at various ruby process via /proc/<id>/status I found
cat /proc/32171/status Name: import_daily State: S (sleeping) Tgid: 32171 Pid: 32171 PPid: 32170 TracerPid: 0 Uid: 81 81 81 81 Gid: 81 81 81 81 FDSize: 64 Groups: 81 998 VmPeak: 5141512 kB VmSize: 5141508 kB
That is more than 5 GB for VM!
The report mail begins to look good. Some minor corrections and run it again. And the check needs only about 4,5 minutes. Quite acceptable.
Now running update, then a check again.
Looked at
60590, 60591, 60592, 60593, 60594, 60595, 60597, 60598, 60599, 60600, 60601, 60602, 60603, 60604, 60605, 60606, 60607, 60608, 60609, 60610, 60611
When running the update_job I get an error when running the invoice-update (Why should I run it here?) Stack-trace is
Plugin: ODDB::TextInfoPlugin Error: IndexError Message: Index 0 is out of range Backtrace: /usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:37:in `[]' /usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:37:in `block in column_info' /usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:25:in `upto' /usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/tuples.rb:25:in `column_info' /usr/local/lib64/ruby/gems/1.9.1/gems/dbd-pg-0.3.9/lib/dbd/pg/statement.rb:83:in `column_info' /usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:158:in `column_names' /usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:27:in `initialize' /usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:81:in `new' /usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:81:in `execute' /usr/local/lib64/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/database.rb:128:in `select_all' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:39:in `block in method_missing' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:29:in `next_connection' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:38:in `method_missing' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:508:in `restore_collection' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:236:in `fetch_collection' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:65:in `block in bulk_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `each' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `bulk_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:56:in `bulk_fetch' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:260:in `fetch_collection' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:640:in `restore_object' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:605:in `load_object' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:226:in `block in fetch' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:225:in `fetch' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:49:in `odba_receiver' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/util/oddbapp.rb:216:in `clean_invoices' /var/www/oddb.org/src/util/oddbapp.rb:1440:in `clean' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:140:in `block (3 levels) in run_cleaner' <internal:prelude>:10:in `synchronize' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:139:in `block (2 levels) in run_cleaner' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:137:in `loop' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/drbserver.rb:137:in `block in run_cleaner'
When running the update job. Received the following error, but updating continued!
Plugin: ODDB::TextInfoPlugin Error: DBI::ProgrammingError Message: ERROR: prepared statement "ruby-dbi:Pg:700110282075801377079174.9888122" already exists
The at the end (*) the job terminated with
update_patinfo Cerumenol® iksnrs_from_xml ["42694"] empty false
(druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:736:in `rescue in block in open': druby://oddb.niklaus.org:56865 - #<Errno::ETIMEDOUT: Connection timed out - connect(2)> (DRb::DRbConnError)
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:730:in `block in open'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `each'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `open'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1191:in `initialize'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `new'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `open'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1087:in `block in method_missing'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1105:in `with_friend'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1086:in `method_missing'
from (druby://localhost:10000) /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/odba.rb:38:in `unpeer'
from (druby://localhost:10000) /var/www/oddb.org/src/util/oddbapp.rb:1610:in `unpeer_cache'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1508:in `perform'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `loop'
from (druby://localhost:10000) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
from /var/www/oddb.org/src/util/job.rb:50:in `run'
from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:12:in `<module:Util>'
from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:11:in `<module:ODDB>'
from /var/www/oddb.org/jobs/update_swissmedicno_fi_pi:10:in `<main>'
update_patinfo.pointer1 42694
store_patinfo existing -> ptr false languages [:fr] reg.iksnr 42694
update_patinfo.pointer2 42694 :!patinfo,1911.
update_patinfo Cerumenol® iksnr 42694 update
import_info: {:fi=>"fachinfo", :pi=>"patinfo"} names {:de=>{:fi=>[nil], :pi=>["Cerumenol®"]}, :fr=>{:fi=>[nil], :pi=>["Cerumenol®"]}} iksnrs []
import_swissmedicinfo_by_iksnrs iksnr "43216" {}
extract_matched_name 43216 fachinfo as 'fi' lang 'de' path is //medicalInformation[@type='fi' and @lang='de']/authNrs returns Xanax®/- retard/- SL
extract_matched_name 43216 patinfo as 'pi' lang 'de' path is //medicalInformation[@type='pi' and @lang='de']/authNrs returns Xanax®/- retard/- SL
Found only 173 of expect "import_swissmedicinfo_by_iksnrs iksnr" (first wer 00035, 00274, last was 43216 Xanax 0,25 mg,). Next would have been 43225
To improve logging added helper function puts_sync which add a timestamp and $stdout.flush to each puts statement to see more clearly where we are when the drb-error comes.
Also with a sleep of 4 minutes between check and re-import we are able to import about 545 iksnrs. But still get the drb connection error. Remove the sleep to commit. Committed: Improved checking FI/PI against Aips.xml and Packungen.xls
When running update-job without a running oddbd, the job hangs for more than 13 minutes at
2013-08-21 13:42:11 +0200: import_swissmedicinfo_by_iksnrs iksnr "18056" {}
2013-08-21 13:42:11 +0200: extract_matched_name 18056 fachinfo as 'fi' lang 'de' path is //medicalInformation[@type='fi' and @lang='de']/authNrs returns Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung
2013-08-21 13:42:11 +0200: extract_matched_name 18056 fachinfo as 'fi' lang 'fr' path is //medicalInformation[@type='fi' and @lang='fr']/authNrs returns Euphyllin® retard N Comprimés filmés, Euphyllin® N Solution injectable
2013-08-21 13:42:11 +0200: import_info: {:fi=>"fachinfo", :pi=>"patinfo"} names {:de=>{:fi=>["Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung"], :pi=>[nil]}, :fr=>{:fi=>["Euphyllin® retard N Comprimés filmés, Euphyllin® N Solution injectable"], :pi=>[nil]}}
2013-08-21 13:42:11 +0200: parse_and_update {:de=>"Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung"} fachinfo
2013-08-21 13:42:11 +0200: parse_and_update: calls parse_fachinfo dist /var/www/oddb.org/data/html/fachinfo/de/Euphyllin_ retard N Filmtabletten, Euphyllin_ N Injektionsl_sung_swissmedicinfo.html iksnrs_from_xml ["18056", "36694"] Euphyllin_ retard N Filmtabletten, Euphyllin_ N Injektionsl_sung_swissmedicinfo.html, name Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung de title Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung
2013-08-21 13:42:12 +0200: update_fachinfo Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung iksnr ["18056", "36694"]
2013-08-21 13:42:12 +0200: update_fachinfo Euphyllin® retard N Filmtabletten, Euphyllin® N Injektionslösung iksnr 18056 store_fachinfo {}
Uses about 3.2 GB of virtual memory (Virtual machine has 8 GB allocated). This should not yet be a problem.
Stopping the job. Running an import of all mentions iksrns (833). Observing memory usage while importing. Stays at 1540 MB for quite some time. After some 600 imported iksnr mounted to 1756 MB. Import completed successfully. After sending the email the job is busy for more than 7 minutes. Memory usage mounts to 1824MB, CPU-load is at 100 (only one thread), postgresql is not busy.
Searched for Stored 312 Fachinfos Ignored 0 Pseudo-Fachinfos Ignored 0 up-to-date Fachinfo-Texts Stored 328 Patinfos Ignored 0 up-to-date Patinfo-Texts Checked 0 companies Unknown Iks-Numbers: 3 Synalar®-N (29300) Bicarbonate de sodium B. Braun (33709) STALORAL Pollen® (69870)
The job did not complete because of a missing return true. After adding it, a simple import job sent the email, but also had the error "#<Errno::ECONNREFUSED: Connection refused - connect(2)> (DRb::DRbConnError)". The error comes from job.rb, which after removing the log/job.pid, wants to call "system.unpeer_cache ODBA.cache unless opts[:readonly]". In my opinion we should add a rescue Errno::ECONNREFUSED clause to system.unpeer_cache as there is also on when calling system.peer_cache
Improved check with don't reimport errors of type 'iksrn found in ODDB.db but not in packages.xls'. Clarified some error messages. Running check-job again. Looks better. Asked Zeno whether it is okay.
Run update-job again. Imports only 486 iksnrs, as clearly missing not. Observing memory usage. Is at 1928 MB while checking. Goes up to 3178 MB when running import, at the end of import is at 3592 MB. Postgres
Will continue next monday.
After telephone with Zeno, pushed commit to fix a typo Fix typo sw -> authNrs