20110511-swissindex_plugin-ch_oddb-debug-atc-zuweisen-de_oddb
<< Masa.20110512-debug-update-function-de_oddb | 2011 | Masa.20110510-swissindex_plugin-ch_oddb-debug-atc-zuweisen-de_oddb >>
- Check swissindex report process
- Check how to set up daemontools
- Output ATC less sequence list (de.oddb)
- Bug: Strange company name
- Goal/Estimate/Evaluation
- SwissindexPlugin / 100% / 100%
- Milestones
commitsetup on server (daemontools)run on server- testcases
suspend - de.oddb atc less list
- Summary
:Commits
- Added swissindex plugin and the DRb servers (oddb.org)
- Added a new bin/admin command, atcless (de.oddb)
Check swissindex report process
Check the report process
- src/plugin/swissindex.rb
class SwissindexPharmaPlugin < SwissindexPlugin
SWISSINDEX_PHARMA_SERVER = DRbObject.new(nil, ODDB::Swissindex::SwissindexPharma::URI)
def update_package_trade_status(logging = false)
Logging.flag = logging
log_dir = File.expand_path('../../log/oddb/debug', File.dirname(__FILE__))
log_file = File.join(log_dir, 'update_package_trade_status.log')
Logging.start(log_file) do |log|
log.print "update_package_trade_status.log\n"
end
@out_of_trade_true_list = []
@out_of_trade_false_list = []
@update_pharmacode_list = []
@delete_pharmacode_list = []
count = 1
start_time = Time.now
@total_packages = @app.packages.length
@app.each_package do |pack|
SWISSINDEX_PHARMA_SERVER.session do |swissindex|
if item = swissindex.search_item(pack.barcode.to_s)
# Process 1
# Check swissindex by eancode and then check if the package is out of trade (true) in ch.oddb,
# if so the package becomes in trade (false)
if pack.out_of_trade
@out_of_trade_false_list << pack
end
# Process 2
# if the package does not have a pharmacode and there is a pharmacode found in swissindex,
# then put the pharmacode into ch.oddb
# We may have to cross-check the pharmacodes in the future
if !pack.pharmacode and pharmacode = item[:phar]
@update_pharmacode_list << [pack, pharmacode]
end
else
# Process 3
# if there is no eancode in swissindex and the package is in trade in ch.oddb,
# then the package becomes out of trade (true) in ch.oddb
unless pack.out_of_trade
@out_of_trade_true_list << pack
end
# Process 4
# if there is no eancode in swissindex then delete the according pharmacode in ch.oddb
if pharmacode = pack.pharmacode
@delete_pharmacode_list << [pack, pharmacode]
end
end
sleep(0.05)
end
Logging.append(log_file) do |log|
log.print pack.barcode, "\t"
end
Logging.append_estimate_time(log_file, count, @total_packages)
count += 1
if @out_of_trade_false_list.length > 1 and
out_of_trade_true_list.length > 1 and
@update_pharmacode_list.length > 1 and
@delete_pharmacode_list.length > 1
break
end
end
...
Result
Checked 23776 packages Updated in trade (out_of_trade:false): 13 packages Updated out of trade (out_of_trade:true) : 11 packages Updated pharmacode: 2 packages Deleted pharmacode: 20 packages Updated in trade (out_of_trade:false): 13 packages Check swissindex by eancode and then check if the package is out of trade (true) in ch.oddb, if so the package becomes in trade (false) 7680456901248: http://ch.oddb.org/de/gcc/resolve/pointer/:!registration,45690!sequence,04!package,124. ... Updated out of trade (out_of_trade:true) : 11 packages If there is no eancode in swissindex and the package is in trade in ch.oddb, then the package becomes out of trade (true) in ch.oddb 7680456911445: http://ch.oddb.org/de/gcc/resolve/pointer/:!registration,45691!sequence,04!package,144. ... Updated pharmacode: 2 packages If the package does not have a pharmacode and there is a pharmacode found in swissindex, then put the pharmacode into ch.oddb 7680252720210: http://ch.oddb.org/de/gcc/resolve/pointer/:!registration,25272!sequence,01!package,021. ... Deleted pharmacode: 20 packages If there is no eancode in swissindex then delete the according pharmacode in ch.oddb 7680361850235: http://ch.oddb.org/de/gcc/resolve/pointer/:!registration,36185!sequence,01!package,023. ...
Check nonpharma report
/home/masa/ywesee/oddb.org/data/csv/swissindex_migel.csv
Note
- Both look no problem
Summary Swissindex Pharma algorithm
src/plugin/swissindex.rb#update_package_trade_status
@app.each_package do |pack|
SWISSINDEX_PHARMA_SERVER.session do |swissindex|
if item = swissindex.search_item(pack.barcode.to_s)
# Process 1
# Check swissindex by eancode and then check if the package is out of trade (true) in ch.oddb,
# if so the package becomes in trade (false)
if pack.out_of_trade
@out_of_trade_false_list << pack
end
# Process 2
# if the package does not have a pharmacode and there is a pharmacode found in swissindex,
# then put the pharmacode into ch.oddb
# We may have to cross-check the pharmacodes in the future
if !pack.pharmacode and pharmacode = item[:phar]
@update_pharmacode_list << [pack, pharmacode]
end
else
# Process 3
# if there is no eancode in swissindex and the package is in trade in ch.oddb,
# then the package becomes out of trade (true) in ch.oddb
unless pack.out_of_trade
@out_of_trade_true_list << pack
end
# Process 4
# if there is no eancode in swissindex then delete the according pharmacode in ch.oddb
if pharmacode = pack.pharmacode
@delete_pharmacode_list << [pack, pharmacode]
end
end
sleep(0.05)
...
- Process 1: out of trade in ch.oddb (out_of_trade == true) and an eancode in swissindex => in trade (out_of_trade == false)
- Process 3: in trade in ch.oddb (out_of_trade == false) and no eancode in swissindex => out of trade (out_of_trade == true)
- Process 2: pharmacodes are different between ch.oddb and swissinex => update the pharmacode
- Process 4: a pharmacode in ch.oddb and no eancode in swissindex => delete pharmacode
Check how to set up daemontools
1. make a directory which has an actual script to execute the server script
masa@masa ~/ywesee/oddb.org/svc $ mkdir swissindex_nonpharma
2. make an actual run script
masa@masa ~/ywesee/oddb.org/svc/swissindex_nonpharma $ cat run #!/bin/sh exec 2>&1 ulimit -v 10240000 exec /usr/bin/ruby -rubygems /home/masa/ywesee/oddb.org/ext/swissindex/bin/swissindex_nonpharmad masa@masa ~/ywesee/oddb.org/svc/swissindex_nonpharma $ ls -al -rwxr-xr-x 1 masa masa 226 11. Mai 11:37 run
3. make a symbolic link on /service directory
masa@masa /service $ sudo ln -s /home/masa/ywesee/oddb.org/svc/swissindex_nonpharma swissindex_nonpharma
4. start the daemon through daemontools commands
masa@masa /service $ sudo svc -u swissindex_nonpharma masa@masa /service $ sudo svstat swissindex_nonpharma swissindex_nonpharma: up (pid 10228) 424 seconds masa@masa /service $ sudo svc -h swissindex_nonpharma masa@masa /service $ sudo svstat swissindex_nonpharma swissindex_nonpharma: up (pid 10805) 2 seconds
5. check if the client actually run
masa@masa ~/ywesee/oddb.org $ bin/admin ch.oddb> Updater.new(self).migel_nonpharma '/home/masa/work/pharmacode_list.dat'
Setup daemontools for swissindex
Install savon (0.8.6)
$ bin/swissindex_nonpharmad
/usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- savon (LoadError)
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /var/www/oddb.org/ext/swissindex/src/swissindex.rb:6
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from bin/swissindex_nonpharmad:7
Error
ywesee@thinpower /var/www/oddb.org/ext/swissindex $ bin/swissindex_nonpharmad
/usr/lib64/ruby/gems/1.8/gems/rack-1.2.2/lib/rack/utils.rb:138:in `union': can't convert Array into String (TypeError)
from /usr/lib64/ruby/gems/1.8/gems/rack-1.2.2/lib/rack/utils.rb:138
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /usr/lib64/ruby/gems/1.8/gems/httpi-0.9.3/lib/httpi/request.rb:3
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /usr/lib64/ruby/gems/1.8/gems/savon-0.8.6/lib/savon/client.rb:1
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from /usr/lib64/ruby/gems/1.8/gems/savon-0.8.6/lib/savon.rb:3
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from /var/www/oddb.org/ext/swissindex/src/swissindex.rb:6
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from bin/swissindex_nonpharmad:7
The following lib versions is necessary to execute the current swissindex plugin
- rack 1.0.0
- httpi 0.9.0
- savon 0.8.6
Note
- These libraries are not up-to-date.
In the future, the swissindex plugin should be updated for the latest savon- The policy to update is some 'necessity', not the version dependency.
But I cannot install rack 1.0.0 on server
# gem install rack --version "=1.0.0" ERROR: could not find gem rack locally or in a repository
Make a gem manually from rake.gemspec in my local PC
masa@masa /usr/lib64/ruby/gems/1.8/gems/rack-1.0.0 $ sudo gem build rack.gemspec
and send it to server via scp and install it
Output ATC less sequence list (de.oddb)
Note (hierarchy of Drugs data (class) structure in de.oddb)
Atc *---------+
|1
Product *--1 Sequence *--1 Package
Experiment (lib/oddb/util/server.rb#test)
def test
open("/home/masa/work/test1.dat","w") do |out|
open("/home/masa/work/test2.dat","w") do |out2|
out.print "total ATCs: ", ODDB::Drugs::Atc.all.length, "\n"
out.print "Total products: ", ODDB::Drugs::Product.all.length, "\n"
all_sequences = 0
all_packages = 0
no_atc_sequences = 0
count = 1
ODDB::Drugs::Product.all.each do |product|
all_sequences += product.sequences.length
product.sequences.each do |sequence|
all_packages += sequence.packages.length
unless sequence.atc
no_atc_sequences += 1
# out2.print count, "\t", sequence.cascading_name('de'), "\n"
count += 1
end
end
end
out.print "Total sequences: ", all_sequences, "\n"
out.print "Total sequences without ATC: ", no_atc_sequences, "\n"
out.print "Total packages: ", all_packages, "\n"
end
end
end
Run
- de.oddb.org/bin/oddbd
- bin/admin
masa@masa ~/ywesee/de.oddb.org $ bin/admin de.oddb> test
Result
total ATCs: 5693 Total products: 8410 Total sequences: 18105 Total sequences without ATC: 2859 Total packages: 42885
Updated
def atcless
out_dir = File.expand_path('../../../log', File.dirname(__FILE__))
FileUtils.mkdir_p out_dir
out_file = File.join(out_dir, 'atcless')
open(out_file,"w") do |out|
out.print "Total ATCs: ", ODDB::Drugs::Atc.all.length, "\n"
out.print "Total products: ", ODDB::Drugs::Product.all.length, "\n"
all_sequences = 0
all_packages = 0
atcless_sequences = []
ODDB::Drugs::Product.all.each do |product|
all_sequences += product.sequences.length
product.sequences.each do |sequence|
all_packages += sequence.packages.length
unless sequence.atc
if sequence.fachinfo.to_s == ""
atcless_sequences << [sequence.cascading_name('de'), '']
else
atcless_sequences << [sequence.cascading_name('de'), 'FI']
end
end
end
end
out.print "Total sequences: ", all_sequences, "\n"
out.print "Total sequences without ATC: ", atcless_sequences.uniq.length, "\n"
out.print "Total packages: ", all_packages, "\n"
out.print "\n"
out.print "ATC less sequence list\n"
atcless_sequences.uniq.sort.each_with_index do |seq, i|
out.print i+1, "\t", seq[1], "\t", seq[0], "\n"
end
end
end
Run
masa@masa ~/ywesee/de.oddb.org $ bin/admin de.oddb> atcless
Result
masa@masa ~/ywesee/de.oddb.org $ head log/atcless -n 20 Total ATCs: 5693 Total products: 8410 Total sequences: 18105 Total sequences without ATC: 1779 Total packages: 42885 ATC less sequence list 1 5 Fu Cell 2 A Risedronsaeureratihpo 3 FI ACC Hexal AG 4 ASTHMA BRONCHIALE STAUFEN 5 Abseamed 6 Acc Akut Z Hustenloese 7 Acemetacin Sandoz 60mg 8 Acetylcystein Siga Mg 9 Acic Hexal 10 Aciclo Basics Mg 11 FI Aciclovir Lindo 12 Aciclovir Mayne 13 Acivir Actavis Mg ...
Commit
Bug: Strange company name
Problem
- Some company names are strange
- Even if it is corrected on browser, the change is not reflected immediately to the cache data
- After rebooting de.oddb, the result changes
Example
