20101019-debug-import_gkv

<< Masa.20101022-debug-export_chde_xls update-xls2odat | 2010 | Masa.20101018-update-xls2odat >>

1. Set up yus on production server
2. Document of yusd setting
3. Set yus privileges on production server
4. Look at data when the error occurs

Goal

• Debug ODDB::Import::Gkv#import / 60 %

Milestones

1. Set up yus on production server 10:15
   • Read README of de.oddb.org 10:15
2. Write a document for yusd setting 11:30
3. Write README of yus 11:30
4. Add the privileges of yus on production server 12:00
5. Continue to debug import_gkv
   • Confirm the split error again 13:30
   • Look at the data when the error occurs 15:30
6. report the product missing the name and create a link to the missing product
7. look for the way to delete the product suspend
8. Organize the information of yus README suspend

Summary

• http://masay.ch/2010/10/19/debug-import_gkv/

Commits

• Added a comment to yus README regarding password
• Add report when a product has no name in import_gkv

ToDo Tomorrow

Keep in Mind

• emerge --sync
• Delete all SL-Entries before importing the new flags.
• Dimdi-Import-Error.
• 20100922-trace-limitation-data-flow-in-bsvXml?action=edit
  • 20100922 Options for LimitationNiveau
• 20100908 rpdf2txt GPLv3
• block sample http://q.hatena.ne.jp/1217573703 (japanese)
• check ydpm priorities

Attached Files

Set up yus on production server

Procedure

1. rsa key files (masa_rsa) by ssh-keygen command
2. certificate file (masa_rsa.crt) by openssl command (openssl req -new -x509 -key id_rsa -out id_rsa.crt)
3. sha256 string by the following script sha256.rb
4. create yus.yml (see sample)
5. put yus.yml on /etc/yus directory
6. put masa_rsa, masa_rsa.crt on /etc/yus/data directory
7. restart yusd

sha256.rb

require 'digest/sha2'
print Digest::SHA256.hexdigest('password'),"\n"

yus.yml sample

root_name: mhatakeyama@ywesee.com
root_pass: a2e287173725b52a92ee79532009d2ffee054bd222222222211111111111
log_level: DEBUG
ssl_key: /etc/yus/data/masa_rsa
ssl_cert: /etc/yus/data/masa_rsa.crt

Restart yusd on production server

su
svc -d /service/yus
svc -u /service/yus
svstat /service/yus

Result

• yusd looks not running

# yus_show mhatakeyama@ywesee.com
/usr/lib64/ruby/1.8/drb/drb.rb:736:in `open': drbssl://localhost:9997 - #<Errno::ECONNREFUSED: Connection refused - connect(2)> (DRb::DRbConnError)
        from /usr/lib64/ruby/1.8/drb/drb.rb:729:in `each'
        from /usr/lib64/ruby/1.8/drb/drb.rb:729:in `open'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1189:in `initialize'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1169:in `new'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1169:in `open'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1085:in `method_missing'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1103:in `with_friend'
        from /usr/lib64/ruby/1.8/drb/drb.rb:1084:in `method_missing'
        from /usr/bin/yus_show:54

Notes

• It is confirmed that the original setting of yus runs certainly with the same restart procedure above
• The password for yusd is required on my local machine but is NOT required on Davatz-san's local machine

Document of yusd setting

yus.yml sample

root_name: mhatakeyama@ywesee.com
root_pass: a2e287173725b52a92ee79532009d2ffee054bd222222222211111111111
log_level: DEBUG
ssl_key: /etc/yus/data/id_rsa.key
ssl_cert: /etc/yus/data/id_rsa.crt

Summary

• root_pass is corresponding to root_name user.
• This password is used when the client application is called (yus_show, yus_adduser, etc.)
• This password string should be generated by the ruby script above (sha256.rb)
• ssl_key file should be generated by ssh-keygen command (ssh-keygen -t rsa) WITHOUT a password
• If you set a pass phrase, the pass phrase is required when yusd starts to run
• ssl_cert file should be generated by the openssl command (openssl req -new -x509 -key id_rsa.key -out id_rsa.crt)
• These ssl_key and ssl_cert files are used when yusd starts to run, which are independent from root_pass

Restart yusd on production server

1. su
2. svc -d /service/yus (stop)
3. svc -u /service/yus (start)
4. svstat /service/yus (check)

Commit

• Added a comment to README regarding password

Set yus privileges on production server

 $ yus_grant mhatakeyama@ywesee.com login org.oddb.de.PowerUser
 $ yus_grant mhatakeyama@ywesee.com login org.oddb.de.Admin
 $ yus_grant mhatakeyama@ywesee.com view org.oddb.de
 $ yus_show mhatakeyama@ywesee.com
 $ yus_show mhatakeyama@ywesee.com
 Password for mhatakeyama@ywesee.com: 
 ["mhatakeyama@ywesee.com",
 ["credit", ],
 ["login", ],
 ["reset_password", []],
 ["set_password", "mhatakeyama@ywesee.com"?],
 ["view", ],
 ["RootUser"]]

Look at data when the error occurs

Experiment lib/oddb/import/gkv.rb

  def postprocess
    Drugs::Package.search_by_code(:type => 'zuzahlungsbefreit',
                                  :value => 'true',
                                  :country => 'DE').each { |package|
      pzn = package.code(:cid).value
      unless(@confirmed_pzns.include?(pzn))
        @deleted += 1
        package.code(:zuzahlungsbefreit).value = false
        save package
      end
    } unless(@confirmed_pzns.empty?)
print "Drugs::Product.all.length=", Drugs::Product.all.length, "\n"
  Drugs::Product.all.length
    Drugs::Product.all { |product|
      unless(product.company)
if product.name.de == nil
  pp product
end

Result

Drugs::Product.all.length=7990
#<ODDB::Drugs::Product:0x7fad3c0913e8
 @name=
  #<ODDB::Util::Multilingual:0x7fad3c08f200
   @canonical={:de=>nil},
   @synonyms=[]>,
 @odba_id=3480899,
 @odba_observers=[],
 @odba_persistent=true,
 @sequences=
  #<ODBA::Stub:70191006647340#3480900 @odba_class=Array @odba_container=70191006648820#3480899>>

Notes

• :de=>nil
• How do I look at the product detail on the web?
• How is the product name updated?

Experiment

  def postprocess
    Drugs::Package.search_by_code(:type => 'zuzahlungsbefreit',
                                  :value => 'true',
                                  :country => 'DE').each { |package|
      pzn = package.code(:cid).value
      unless(@confirmed_pzns.include?(pzn))
        @deleted += 1
        package.code(:zuzahlungsbefreit).value = false
        save package
      end
    } unless(@confirmed_pzns.empty?)
# masa
print "Drugs::Product.all.length=", Drugs::Product.all.length, "\n"
count = 0
  Drugs::Product.all.length
    Drugs::Product.all { |product|
      unless(product.company)
if product.name.de == nil
  count += 1
else
        keys = product.name.de.split
        key = keys.pop
        if(key == 'Comp')
          key = keys.pop
        end
        company = Business::Company.find_by_name(key)
        if(company.nil?)
          companies = Business::Company.search_by_name(key)
          if(companies.size == 1)
            company = companies.pop
          end
        end
        if(company)
          @assigned_companies += 1
          product.company = company
          save product
        end
end
      end
    }
print "product.name.de missing = ", count, "\n"

Result

Drugs::Product.all.length=7990
product.name.de missing = 1

Notes

• The number of the products which misses the name equals only 1
• In de.oddb.org, there is a hierarchy regarding drugs, product-sequence-package
• Each product has 'uid' uniquely
• Each package has 'pzn' uniquely
• We can search a product like http://de.oddb.org/de/drugs/product/uid/348089
• We can search a package like http://de.oddb.org/de/drugs/package/pzn/6588939

Question

• uid is corresponding to odba_id or not?
• How do I delete a product data?

Experiment lib/oddb/import/gkv.rb

  def postprocess
p "getin Gkv#postprocess"
    Drugs::Package.search_by_code(:type => 'zuzahlungsbefreit',
                                  :value => 'true',
                                  :country => 'DE').each { |package|
      pzn = package.code(:cid).value
      unless(@confirmed_pzns.include?(pzn))
        @deleted += 1
        package.code(:zuzahlungsbefreit).value = false
        save package
      end
    } unless(@confirmed_pzns.empty?)
# masa
print "Drugs::Product.all.length=", Drugs::Product.all.length, "\n"
count = 0
product_count = 0
  Drugs::Product.all.length
    Drugs::Product.all { |product|
product_count += 1
if product_count == 4236 or product_count == 1000
  print "product_count: ", product_count, "\n"
  pp product
end

Run import_gkv

masa@masa ~/ywesee/de.oddb.org $ rm -rf var/
masa@masa ~/ywesee/de.oddb.org $ jobs/import_gkv

Result

product_count: 1000
#<ODDB::Drugs::Product:0x7fa2ddf8b510
 @company=
  #<ODDB::Business::Company:0x7fa2ddf8b178
   @odba_class=nil,
   @odba_container=nil,
   @odba_id=214510,
   @receiver=nil,
   @receiver_loaded=nil>,
 @name=
  #<ODDB::Util::Multilingual:0x7fa2ddf8b0d8
   @canonical={:de=>"Laticort"},
   @synonyms=[]>,
 @odba_id=31046,
 @odba_observers=[],
 @odba_persistent=true,
 @sequences=
  #<ODBA::Stub:70168742746620#31047 @odba_class= @odba_container=70168742746760#31046>>
product_count: 4236
#<ODDB::Drugs::Product:0x7fa2ddf31650
 @company=
  #<ODDB::Business::Company:0x7fa2ddf30d90
   @odba_class=nil,
   @odba_container=nil,
   @odba_id=213625,
   @receiver=nil,
   @receiver_loaded=nil>,
 @name=
  #<ODDB::Util::Multilingual:0x7fa2ddf2f850
   @canonical={:de=>"Glimepirid Biomo"},
   @synonyms=
    ["Glimepirid Biomo 4mg Tabl",
     "Glimepirid Biomo 6mg Tabl",
     "Glimepirid Biomo Mg Tabl"]>,
 @odba_id=81057,
 @odba_observers=[],
 @odba_persistent=true,
 @sequences=
  #<ODBA::Stub:70168742562260#81058 @odba_class=Array @odba_container=70168742562600#81057>>

Conclusion

• @odba_id and uid are the same
• We can access by using the @odba_id (uid) on the web as follows: http://de.oddb.org/de/drugs/product/uid/31046

Commit

• Add report when a product has no name in import_gkv.

Report mail sample

Tue Oct 19 16:45:49 2010: de.oddb.org ODDB::Import::Gkv#import
Imported     0 Zubef-Entries on 19.10.2010:
Visited      0 existing Zubef-Entries
Visited      0 existing Companies
Visited      0 existing Substances
Created      0 new Zubef-Entries
Created      0 new Products
Created      0 new Sequences
Created      0 new Companies
Created      0 new Substances
Assigned     0 Chemical Equivalences
Assigned     0 Companies
Created      0 Incomplete Packages:
Created      1 Product(s) without a name (missing product name):
http://de.oddb.org/de/drugs/product/uid/3480899