view · edit · sidebar · attach · print · history

Index>

20140218-Fix-Patents

Summary

  • Fix error in patents when running jobs/import_swissmedic
  • Search for ndications, z.B. Konjunktivitis, does not report all occurrences

Commits

Index

Keep in Mind
  • Search nach Anwendung (indications), z.B. Konjunktivitis, does not report all occurrences in the section "Anwendung" of Fachinfo. Probably index is corrupted or not set up correctly.
  • Error: Patents with could not connect to www.swissreg.ch: #<Net::HTTPInternalServerError:0x007f8a7d69bb58>
  • vagrant-oddb.org: cleanup installation for yus with ruby 1.8, logrotate.conf and local vhosts for tests

---

convert Sinovial(R) HighVisc 1,6% to HTML and XML

I try the gem ydocx to convert Attach:sinovial_de.txt. This example produces the following exception

/opt/src/ydocx/lib/ydocx/templates/fachinfo.rb:109:in `parse_title': undefined method `inner_text' for nil:NilClass (NoMethodError)
        from /opt/src/ydocx/lib/ydocx/templates/fachinfo.rb:130:in `parse_block'
        from /opt/src/ydocx/lib/ydocx/parser.rb:219:in `parse_paragraph'
        from /opt/src/ydocx/lib/ydocx/parser.rb:37:in `block in parse'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/nokogiri-1.6.1/lib/nokogiri/xml/node_set.rb:237:in `block in each'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/nokogiri-1.6.1/lib/nokogiri/xml/node_set.rb:236:in `upto'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/nokogiri-1.6.1/lib/nokogiri/xml/node_set.rb:236:in `each'
        from /opt/src/ydocx/lib/ydocx/parser.rb:30:in `map'
        from /opt/src/ydocx/lib/ydocx/parser.rb:30:in `parse'
        from /opt/src/ydocx/lib/ydocx/document.rb:126:in `block in read'
        from /opt/src/ydocx/lib/ydocx/parser.rb:24:in `initialize'
        from /opt/src/ydocx/lib/ydocx/document.rb:124:in `new'
        from /opt/src/ydocx/lib/ydocx/document.rb:124:in `read'
        from /opt/src/ydocx/lib/ydocx/document.rb:33:in `initialize'
        from /opt/src/ydocx/lib/ydocx/document.rb:19:in `new'
        from /opt/src/ydocx/lib/ydocx/document.rb:19:in `open'
        from /opt/src/ydocx/lib/ydocx/command.rb:99:in `run'
        from bin/docx2xml:12:in `<main>'

Will fork the gem. Add a Gemfile and a .travis.ci. Then add tests with this example.

Found out that I when passing parameter --plain the transformation works (it neither a fachinfo nor a patinfo).

Pushed the following commits:

Agreed on the following requirements with Zeno:

  • Create jobs/import_medical_products which imports *.docx files from data/medical_products. Has 1 parameter a glob style string for selecting files, e.g. '*' or 'Sinovial_DE.docx'
  • imported items must be found using the search by atcclass. We will add a pseudo atc_class with code 'medical'.
  • We must be able to find it by its EAN13 code.
  • We must update oddbapp.rb to return medical_products and be able to search it via name.
  • We assume the convention name{_DE|_FR}.docx to find out the language the document is.

Looking at the examples provided by Zeno, I see that I still need some more parsing to get nice result. E.g. I would like be able to look (like in fachinfo) registration('62630').indication. Using 'ydocx' as it is I get results like

inhalt.keys
=> ["Titel",
 "titel",
 "zusammens",
 "ind_anwmoegl",
 "dos_anw",
 "kontraind",
 "warnhinw",
 "interakt",
 "unerwwirkungen",
 "eigensch",
 "sonstige_h",
 "packungen",
 "hersteller"]

My solution worked fine for Sinovial_0.8_DE.docx, but failed miserably with Sinovial_0.8_DE.docx as the inner structure was quite different even when both files look quit similar!

Comparing the XML-output I find lot less differences. Therefore I think the correct way is to first create an xml file and extract the chapter info afterwards. With the following loop I can iterate over the parts of the XML document, which interest me

require 'nokogiri'
doc = Nokogiri::XML(open('/opt/src/ydocx/spec/data/Sinovial_DE.xml'))
doc.xpath('//chapters/chapter').each{ |x| next if x.xpath('heading').size == 0; puts "\n\n"+x.xpath('heading').text; puts x.xpath('paragraph').text}
name = doc.xpath('//chapters/paragraph/bold').text
ean13 = [] ; doc.xpath('//chapters/chapter').each{ |x| next unless x.xpath('heading').text.match(/Packungen/); x.xpath('paragraph').each{
	|p| m= p.text.match(/(\d{13})($|\s|\W)/); ean13 << m[1] if m } }; 
ean13

Search for ndications, z.B. Konjunktivitis, does not report all occurrences

Search nach Anwendung (indications), z.B. Konjunktivitis, does not report all occurrences in the section "Anwendung" of Fachinfo. Probably index is corrupted or not set up correctly. Analysing the problem with bin/admin

indications.find_all{ |ind| /Konjunktivitis/.match(ind.search_text) }.size
-> 6
indications.find_all{ |ind| /Konjunktivitis/.match(ind.search_text) }.first.search_text
-> Symptomatische Behandlung der saisonalen allergischen Konjunktivitis und Rhinokonjunktivitis
indications.find_all{ |ind| /Konjunktivitis/.match(ind.search_text) }.first.registrations.first.iksnr
-> 56248
ch.oddb> @all = indications.find_all{ |ind| /Konjunktivitis/.match(ind.search_text) }
-> Array
@all.collect{|indication| indication.registrations.first.iksnr }
-> ["56248", "52803", "53308", "52044", "56310", "47036"]
@all2 = registrations.find_all{ |reg| reg[1].fachinfo and /Konjunktivitis/.match(reg[1].fachinfo.search_text(:de)) }
-> IndikationenAnwendungsmoeglichkeiten Blepharitis Hordeolum Bakterielle Keratitis und Konjunktivitis
@all2.size
-> 266
@all2.first[1].iksnr
-> 13131
@all2.first[1].fachinfo.search_text(:de)
-> IndikationenAnwendungsmoeglichkeiten Blepharitis Hordeolum Bakterielle Keratitis und Konjunktivitis

With these few lines we demonstrated that using indications (which I suspect that it uses the index) we find only 6 occurrences of Konjunktivitis. Whereas looking in the german version of the searchtext of all registrations we find 266 occurrences.

Fix error in patents when running jobs/import_swissmedic

Added some debug output and running jobs/import_swissmedic on oddb-ci2. It goes further but has the following error

Plugin: ODDB::SwissregPlugin
Error: NameError
Message: undefined local variable or method `patents' for #<ODDB::SwissregPlugin:0x007fc8eb3db740>
Backtrace:
/var/www/oddb.org/src/plugin/swissreg.rb:61:in `update_registrations'
/var/www/oddb.org/src/plugin/swissreg.rb:55:in `block in update_news'
/var/www/oddb.org/src/plugin/swissreg.rb:52:in `each_key'
/var/www/oddb.org/src/plugin/swissreg.rb:52:in `update_news'
/var/www/oddb.org/src/util/updater.rb:505:in `update_immediate'
/var/www/oddb.org/src/util/updater.rb:418:in `update_swissreg_news'
/var/www/oddb.org/src/util/updater.rb:406:in `update_swissmedic_followers'
jobs/import_swissmedic:15:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

Error comes from debug statement. Correcting and restarting import again. Now I got the following error

Plugin: ODDB::SwissregPlugin
Error: RuntimeError
Message: could not connect to www.swissreg.ch https://www.swissreg.ch/srclient/faces/jsp/spc/sr3.jsp: #<Net::HTTPInternalServerError:0x007f13e5613668>
Backtrace:
(druby://localhost:10007) /var/www/oddb.org/src/util/http.rb:98:in `post'
(druby://localhost:10007) /var/www/oddb.org/ext/swissreg/src/session.rb:203:in `post'
(druby://localhost:10007) /var/www/oddb.org/ext/swissreg/src/session.rb:196:in `get_result_list'
(druby://localhost:10007) /var/www/oddb.org/ext/swissreg/src/swissreg.rb:12:in `search'
(druby://localhost:10007) /usr/lib64/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10007) /usr/lib64/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10007) /usr/lib64/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10007) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10007) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissreg.rb:62:in `update_registrations'
/var/www/oddb.org/src/plugin/swissreg.rb:55:in `block in update_news'
/var/www/oddb.org/src/plugin/swissreg.rb:52:in `each_key'
/var/www/oddb.org/src/plugin/swissreg.rb:52:in `update_news'
/var/www/oddb.org/src/util/updater.rb:505:in `update_immediate'
/var/www/oddb.org/src/util/updater.rb:418:in `update_swissreg_news'
/var/www/oddb.org/src/util/updater.rb:406:in `update_swissmedic_followers'
jobs/import_swissmedic:15:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

avoid error when running jobs/import_swissmedic

Before running jobs/import_swissmedic I removed manually all files containg latest in their name under data/xls/. This broke the import, as a debug msg tried to get the size of the not existing files.

Pushed commit Avoid error when latest does not exist

view · edit · sidebar · attach · print · history
Page last modified on February 18, 2014, at 06:02 PM