< Niklaus.20210105-ruby-30 | Index | Niklaus.20200603-ruby-271 >>
bundle-217 uninstall date; bundle-271 install date
After the commit (#109) Work around problem loading parslet_compositions running import_daily fails, because way too much memory is required.
2020-06-22 12:17:37 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:582:in `get_latest_file': cmd @latest_packungen = '/var/www/oddb.org/data/xls/Packungen-latest.xlsx'
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:606:in `get_latest_file': skip writing /var/www/oddb.org/data/xls/Packungen-2020.06.22.xlsx as /var/www/oddb.org/data/xls/Packungen-latest.xlsx is 2925536 bytes. Returning latest_name /var/www/oddb.org/data/xls/Packungen-latest.xlsx
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:290:in `update': Loaded /var/www/oddb.org/src/plugin/parslet_compositions.rb
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:308:in `update': opts {} @update_comps update file2open nil Latest /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2925536 bytes
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:217:in `show_memory_useage': Using 23163 MB of memory. Limit is 16384. Swissmedic_do_tracing true
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:387:in `update': @update_comps check_all_packages
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:391:in `update': file2open nothing to do for opts {}
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:394:in `update': done. 0 export_registrations @update_comps was with no change information
2020-06-22 12:17:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:228:in `block in trace_memory_useage': Aborting as using 23163 MB of memory > than limit of 16384
Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Error: swissmedic - 06/2020 size 710 with 0 attachments. #<Mail::Body:0x00007f44b78cc8c0 @boundary=nil, @preamble=nil, @epilogue=nil, @charset="UTF-8", @part_sort_order=["text/plain", "text/enriched", "text/html", "multipart/alternative"], @parts=[], @raw_source="Plugin: ODDB::SwissmedicPlugin\r\n \r\nError: SystemExit\r\nMessage: SystemExit\r\nBacktrace:\r\n/var/www/oddb.org/src/plugin/swissmedic.rb:396:in `join'\r\n/var/www/oddb.org/src/plugin/swissmedic.rb:396:in `map'\r\n/var/www/oddb.org/src/plugin/swissmedic.rb:396:in `update'\r\n/var/www/oddb.org/src/util/updater.rb:406:in `block in update_swissmedic'\r\n/var/www/oddb.org/src/util/updater.rb:508:in `wrap_update'\r\n/var/www/oddb.org/src/util/updater.rb:404:in `update_swissmedic'\r\n/var/www/oddb.org/src/util/updater.rb:212:in `run'\r\njobs/import_daily:13:in `block in <module:Util>'\r\n/var/www/oddb.org/src/util/job.rb:41:in `run'\r\njobs/import_daily:12:in `<module:Util>'\r\njobs/import_daily:11:in `<module:ODDB>'\r\njobs/import_daily:10:in `<main>'", @ascii_only=true, @encoding="7bit">
/var/www/oddb.org/log/job.pid is deleted
Trying to fix this problem, be moving the loading to the original line and using a global variable.
The https://ch.oddb.org/de/gcc/migel_search/migel_product/010102022 list "4641084 AMEDA Adapterset für Einhandmilchpumpe", which contains an Umlaut. Looking via oddb.org/bin/admin (on thinpower and oddb-ci2) I find44
ch.oddb> search_migel_items('4641084', 'de').first.ean_code
-> 7612479000158
ch.oddb> search_migel_items('4641084', 'de').first.pharmacode
-> 4641084
ch.oddb> search_migel_items('4641084', 'de').first.name.de
-> AMEDA Adapterset für Einhandmilchpumpe
ch.oddb> search_migel_items('4641084', 'de').first.name.de.encoding
-> ASCII-8BIT
ch.oddb> CGI.unescape(search_migel_items('4641084', 'de').first.name.de)
-> AMEDA Adapterset für Einhandmilchpumpe
ch.oddb> search_migel_items('4641084', 'de').first.name.de.force_encoding('UTF-8')
-> AMEDA Adapterset für Einhandmilchpumpe
ch.oddb> search_migel_items('4641084', 'de').first.name.de.force_encoding('UTF-8').encoding
-> UTF-8
This is confirmed via migel/bin/admin (also on thinpower and oddb-ci2)
migel> products.values.find{ |x| x.ean_code.to_i == 7612479000158}
-> AMEDA Adapterset für Einhandmilchpumpe
migel> products.values.find{ |x| x.ean_code.to_i == 7612479000158}.name.de.encoding
-> ASCII-8BIT
migel> products.values.find{ |x| x.ean_code.to_i == 7612479000158}.name.de
-> AMEDA Adapterset für Einhandmilchpumpe
Now I must look at why importing does not set the correct URL. Zeno told me that /var/www/oddb.org/etc/oddb.yml sets the URL to parse
like this migel_base_url:
This URL could be reached via migel_nonpharma in src/util/updater.rb, but is never called. The pharmacode cited above is no longer found in the XMLRefDataNonPharma.xml.
In 2012 Mass added with commit https://github.com/zdavatz/oddb.org/commit/349dc3ca68e552aefef7b9e2920f2b54af9b9fe9 jobs/export_oddb2_csv and jobs/export_oddb_csv. Zeno removed the jobs/update_migel from oddb.org with commit https://github.com/zdavatz/oddb.org/commit/a36c2790f8a15f45232ccc65a90a19e065500bed.
Added to src/util/oddbapp.rb the following snippet to test fetching a single pharmacode
def refdata_migel_update(pharmacode)
puts "refdata_migel_update #{pharmacode}"
handler = ODDB::SwissindexMigelPlugin.new(self)
handler.search_item(pharmacode)
end
Setting I could see that we parsed then got the error @@HTTP 404.
Remarked that the services /service/ch.oddb-swissindex_pharma/run, /service/ch.oddb-swissreg/run where still using an old ruby version. Also /service/migeld/run still had a bundle-271 as command, which made it NOT use the new date gem. Fixed these problems.
Now the prices for some migel items did not show.
Remarked that on oddb-ci2 the dojo.js is not loaded when visiting the interactions. This must be fixed.
Pushed https://github.com/ngiger/sbsm/commit/ff902dc17ddac5d2b43cdd60a4a473cb203eaa9f , as logo, css and javascript files were not loaded.
Released SBSM 1.6.0, Htmlgrid 1.1.7. Updated oddb.org to use these newer gems. Running unit tests and spec tests before installing new version on thinpower.
After restarting the ch.oddb-export the bsv_followers worked again.
Must release yus-1.0.6.gem. Updating yus on thinpower.
rm /usr/bin/pg_config ln -s /usr/local/pg-10_1/bin/pg_config /usr/bin/pg_config gem-271 install yus
Setting exec in /service/yus/run to exec sudo -u apache /usr/local/bin/ruby-271 /usr/local/ruby-2.7.1/bin/yusd
Changed localhost -> 127.0.0.1 in /etc/yus/yus.yml
Cannot login into ch.oddb.org via desitin, as I get the error
DRb::DRbConnError at /de/gcc drbssl://localhost:9997 - #<OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello> Ruby /usr/local/ruby-2.5.0/lib/ruby/2.5.0/drb/drb.rb: in rescue in block in open, line 744 Web POST ch.oddb.org/de/gcc
Setting exec in /service/migeld/run to exec sudo -u apache ruby-271 bin/migeld
cd /var/www/migel # as root bundle-271 config set system 'true' bundle-271 install --system # ignoring warning # as ywesee cd /var/www/oddb.org git log https://github.com/ngiger/oddb.org.git # as root bundle-271 config set system 'true' bundle-271 install --system # ignoring warning
Added line ywesee ALL=(apache) /usr/local/ruby-2.7.1/bin/* to /etc/sudoers
Must install new date gem
cd /home/ywesee git clone https://github.com/ngiger/date.git git checkout origin/marshal-load-failure-20 -b marshal-load-failure-20 bundle-271 config set with 'development' rake-271 compile # as root gem-271 install /home/ywesee/date/pkg/date-3.0.1.gem svc -h /service/migel # Now https://ch.oddb.org/de/gcc/home_migel/ can be visited svc -h /service/ch.oddb svc -h /service/yus # still unable to login as desitin
Updated all /service/ch*/run scripts to use ruby-271. Checked all running ruby processs. Only currency and ydim still run with ruby 2.5.0. Both are not use by ch.oddb.org
Had to regenerate the Certificate for yus which dated from 2006. Executed as root
cd /etc/yus/data root@thinpower:/etc/yus/data# openssl req -new -x509 -key ./yus.key -out ./yus.crt You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:CH State or Province Name (full name) [Some-State]:ZH Locality Name (eg, city) []:Zuerich Organization Name (eg, company) [Internet Widgits Pty Ltd]:ywesee Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []: Email Address []: svc -h /service/yus
Now I could login as desitin.