view · edit · sidebar · attach · print · history

20110201-debug-fach_news-oddb_org

<< | Index | >>

Confirm the current status
Merge the grabing script
Create a GitHub account
Simplify the code
Update test-cases

Goal

Debug fach patinfo news / 100%

Milestones

~~Check the current status~~ 8:30
~~Check the update_textinfo_news method works~~ 8:30
~~Merge the script of grabing Fach and Patinfo list~~ 10:20
~~Check the update_textinfo_news working~~ 10:20
~~Simplify the code~~ 15:00
~~Check the script working~~ 15:20
Update test-cases
Check how to use git hub

Summary

http://masay.ch/2011/02/01/debug-fach_news-oddb_org-2/

Commits

Debug Fach- and Patinfo News job, since the web site is updated (oddb.org)

ToDo Tomorrow
Keep in Mind

Check rdbi instead of dbi for ODBA
Encoding woes (from Davatz-san)
Feedback: This option indicates that the regular expression is parsed as 'UTF8' (from Davatz-san)
pg on Ubuntu - see http://dev.ywesee.com/wiki.php/Gem/Pg (from Davatz-san)
On Ice
emerge --sync

Confirm the current status

Change 'text_info_newssource:' (etc/oddb.yml) Note

This URL is useless for this test, because the 'names' variable is re-set below
But a correct URL is necessary for the update_textinfo_news method to run

src/plugin/text_info.rb#import_news

    def import_news agent=init_agent
      old_news = old_fachinfo_news
      updates = true_news fachinfo_news(agent), old_news
      updates.reverse!
      indices, names = updates.transpose
names = ['Abilify']
      if names
        import_name names, agent
        log_news updates + old_news
        postprocess
      end
      !updates.empty?
    end

Run

oddb.org/bin/oddbd
currency/bin/currencyd

Run update_textinfo_news

ch.oddb> Updater.new(self).update_textinfo_news

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for Abilify
Stored 0 Fachinfos
Ignored 0 Pseudo-Fachinfos
Ignored 0 up-to-date Fachinfo-Texts
Stored 0 Patinfos
Ignored 0 up-to-date Patinfo-Texts
Checked 0 companies
Unknown Iks-Numbers: 0
Fachinfos without iksnrs: 0
Session failures: 0
Download errors: 0

Parse Errors: 2
druby://localhost:10002 - #<Errno::ECONNREFUSED: Connection refused - connect(2)>
druby://localhost:10002 - #<Errno::ECONNREFUSED: Connection refused - connect(2)>

Run fiparsed

masa@masa ~/ywesee/oddb.org/ext/fiparse $ ruby bin/fiparsed

Result

/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: redefining constant Struct::MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: already initialized constant MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: redefining constant Struct::ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: already initialized constant ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: redefining constant Struct::Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: already initialized constant Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: redefining constant Struct::Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: already initialized constant Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/grammar.rb:591: warning: already initialized constant EpsilonToken
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: redefining constant Struct::DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: already initialized constant DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: redefining constant Struct::GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: already initialized constant GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: redefining constant Struct::LaLrPair
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: already initialized constant LaLrPair
/usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- rwv2/rwv2 (LoadError)
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /home/masa/ywesee/oddb.org/ext/fiparse/src/fachinfo_doc.rb:4
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /home/masa/ywesee/oddb.org/ext/fiparse/src/fiparse.rb:11
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from bin/fiparsed:9

Install rwv2

Run again

masa@masa ~/ywesee/oddb.org/ext/fiparse/bin $ ruby fiparsed 
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: redefining constant Struct::MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: already initialized constant MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: redefining constant Struct::ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: already initialized constant ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: redefining constant Struct::Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: already initialized constant Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: redefining constant Struct::Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: already initialized constant Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/grammar.rb:591: warning: already initialized constant EpsilonToken
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: redefining constant Struct::DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: already initialized constant DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: redefining constant Struct::GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: already initialized constant GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: redefining constant Struct::LaLrPair
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: already initialized constant LaLrPair
/usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require': no such file to load -- rwv2 (LoadError)
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /usr/lib64/ruby/gems/1.8/gems/rwv2-0.6.0/lib/rwv2/rwv2.rb:25
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
        from /home/masa/ywesee/oddb.org/ext/fiparse/src/fachinfo_doc.rb:4
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from /home/masa/ywesee/oddb.org/ext/fiparse/src/fiparse.rb:11
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from fiparsed:9

Experiment

/usr/lib64/ruby/gems/1.8/gems/rwv2-0.6.0/lib/rwv2/rwv2.rb

#require 'rwv2'
require 'rwv2/handlers'

VERSION = '0.6.0'

Run again

masa@masa ~/ywesee/oddb.org/ext/fiparse $ ruby bin/fiparsed 
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: redefining constant Struct::MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:73: warning: already initialized constant MethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: redefining constant Struct::ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/sourcecode_dumpable.rb:74: warning: already initialized constant ModuleMethodNamed
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: redefining constant Struct::Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/glr_parser.rb:275: warning: already initialized constant Link
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: redefining constant Struct::Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/conflict_resolution.rb:18: warning: already initialized constant Relation
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/grammar.rb:591: warning: already initialized constant EpsilonToken
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: redefining constant Struct::DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/graphviz_dot.rb:52: warning: already initialized constant DotGraph
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: redefining constant Struct::GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/directed_graph.rb:16: warning: already initialized constant GraphLink
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: redefining constant Struct::LaLrPair
/usr/lib64/ruby/site_ruby/1.8/rpdf2txt-rockit/reduce_actions_generator.rb:273: warning: already initialized constant LaLrPair
/usr/lib64/ruby/gems/1.8/gems/rwv2-0.6.0/lib/rwv2/rwv2.rb:28: warning: already initialized constant VERSION

Note

Many warning but it works

Try to run update_textinfo_news again

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for Abilify
Stored 0 Fachinfos
Ignored 0 Pseudo-Fachinfos
Ignored 2 up-to-date Fachinfo-Texts
Stored 0 Patinfos
Ignored 0 up-to-date Patinfo-Texts
Checked 0 companies
Unknown Iks-Numbers: 0
Fachinfos without iksnrs: 0
Session failures: 0
Download errors: 0
Parse Errors: 0

Note

Looks good

Experiment (delete data/html)

Delete files

masa@masa ~/ywesee/oddb.org $ rm -rf log/
masa@masa ~/ywesee/oddb.org $ rm -rf data/html/
masa@masa ~/ywesee/oddb.org $ rm -rf data/rss/

masa@masa ~/ywesee/oddb.org $ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   src/plugin/text_info.rb
#
no changes added to commit (use "git add" and/or "git commit -a")

Run again

Result

Email ch.ODDB.org Report - Error: Fach- und Patienteninfo News - 02/2011

Plugin: ODDB::TextInfoPlugin
Error: Errno::ENOENT
Message: No such file or directory - /home/masa/ywesee/oddb.org/log/fachinfo.txt
Backtrace:
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:218:in `initialize'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:218:in `open'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:218:in `log_news'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:190:in `import_news'
/home/masa/ywesee/oddb.org/src/util/updater.rb:482:in `send'
/home/masa/ywesee/oddb.org/src/util/updater.rb:482:in `update_notify_simple'
/home/masa/ywesee/oddb.org/src/util/updater.rb:462:in `call'
/home/masa/ywesee/oddb.org/src/util/updater.rb:462:in `wrap_update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:480:in `update_notify_simple'
/home/masa/ywesee/oddb.org/src/util/updater.rb:282:in `update_textinfo_news'
(eval):1:in `_admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1399:in `_admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `initialize'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `new'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'
bin/oddbd:38

Note

It seems that 'log/fachinfo.txt' is necessary for 'update_textinfo_news' to run
This is caused because there is no directory
This can be resolved by making a parent directory with 'FileUtils.mkdir_p'

Experiment

masa@masa ~/ywesee/oddb.org $ rm -rf data/html/
masa@masa ~/ywesee/oddb.org $ rm -rf data/rss/
masa@masa ~/ywesee/oddb.org $ rm -rf log
masa@masa ~/ywesee/oddb.org $ mkdir log
masa@masa ~/ywesee/oddb.org $ touch log/fachinfo.txt

masa@masa ~/ywesee/oddb.org $ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   src/plugin/text_info.rb
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       log/
no changes added to commit (use "git add" and/or "git commit -a")

Run again

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for Abilify
Stored 2 Fachinfos
Ignored 0 Pseudo-Fachinfos
Ignored 0 up-to-date Fachinfo-Texts
Stored 0 Patinfos
Ignored 0 up-to-date Patinfo-Texts

Checked 0 companies
Unknown Iks-Numbers: 0
Fachinfos without iksnrs: 0
Session failures: 0
Download errors: 0
Parse Errors: 0

Note

That is what I expected
log/fachinfo.txt is necessary for 'update_textinfo_news' to run
'log/fachinfo.txt' is used for the comparison of the new list with the old one
Also, ext/fiparse/bin/fiparsed must be running

Check updated date on browser

Note

Looks good

Merge the grabing script

Check the script of grabbing the Fach- and Patienteninformation list

getlist.rb

Result

"Abilify\302\256"
"Abilify\302\256 Injektionsl\303\266sung"
"Abseamed\302\256"
"Aceril\302\256- mite"
...

Note

It is a problem that the URL is static, in other words, it will change in next year or someday

Experiment

Change text_info_newssource: etc/oddb.yml

src/plugin/text_info.rb#fachinfo_news

    def fachinfo_news agent=init_agent
      url = ODDB.config.text_info_newssource \
        or raise 'please configure ODDB.config.text_info_newssource to proceed'
      ids = []
      page = agent.get url
=begin
      page.links.each do |link|
        if id = extract_fachinfo_id(link.href)
          ids.push [id, link.text.gsub(/;$/, '')]
        end
      end
=end
      list = page.at('div[id="blockContentInner"]/p')
      list.to_html.split("\<br\>").each do |element|
        name = element.delete("\<p\>").delete("\<\/p\>").chomp.strip
        ids << ['id', name]
      end
      ids

Run update_textinfo_news

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Note

Leave 'data' and 'log' directries as they are not deleted

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for Zyloric�, Zyban�, ...
Stored 986 Fachinfos
Ignored 75 Pseudo-Fachinfos
Ignored 44 up-to-date Fachinfo-Texts
Stored 669 Patinfos
Ignored 30 up-to-date Patinfo-Texts
Checked 0 companies

Unknown Iks-Numbers: 4
...

Fachinfos without iksnrs: 26
...
Session failures: 0

Download errors: 0

Parse Errors: 6
...

Note

It tooks about 30 minutes (Stored 986 Fachinfos, Stored 669 Patinfos)
The result looks good, but there are several parts that I should update
1. the URL should be dynamically grabbed, not static, if it changes (this must be, because the URL will change definitely next year)
2. ID data of the new list in 'fachinfo_news' method is not used (this should be, in order to simplify the code and logic)
  - There is no ID information in the HTML code of the URL
  - It is only used for the comparison between the new and old list
  - But the checking algorithm is only the check if the first element is saved in the log/fachinfo.txt or not
  - Furthermore, the log/fachinfo.txt will only increase (never deleted)
  - It is enough to use the names of the list
3. The search list should be reversed, like A-Z

(The followings are better to be considered, in my opinion)

The name list is directly created from the html code in the code above
I guess this is not good for the future because this code does not work if the html code changes
Probably we expect some errors come in such a case, but we can think a possible case where the job runs without error but the updating fails
In my opinion, we should make a logic to detect the different HTML code and URL
- because the html code (and also URL) is easy to be changed by the administrator of the site
- and moreover, if possible, we should manage them separately from the main logic in order to change them quickly in the future
- Otherwise, we have to review the code and search the cause again, and we have to update the logic everytime when the website is updated

Run again (in order to compare the time to run)

Result

It ends quickly
Nothing to come (by email and on console), since the 'names' value equals to 'nil' in 'import_news' method

Create a GitHub account

Link

https://github.com/

How to use

Simplify the code

Memo

The ID in log/fachinfo.txt is not valid anymore
I have to think the comparison process between the new and the old list

Design

Remove 'ID process'
Just compare the lists by name
Get the diff between the new and the old list (new Array - old Array)
- In this case, the element of the same name to the old one will not be updated even if it is really updated online
'true_news' methods returns the diff
'transpose' process is not necessary in 'import_news', because the array includes only names, not IDs

Experiment

src/plugin/text_info.rb#old_fachinfo_new

    def old_fachinfo_news
      begin
        File.readlines(@news_log).collect do |line|
          line.strip
        end
      #  File.readlines(@news_log).collect do |line|
      #    line.strip.split ' ', 2
      #  end
      #rescue Errno::ENOENT
      rescue
        []
      end
    end

src/plugin/text_info.rb#fachinfo_news

    def fachinfo_news agent=init_agent
      url = ODDB.config.text_info_newssource \
        or raise 'please configure ODDB.config.text_info_newssource to proceed'
      name_list = []
      page = agent.get url
      list = page.at('div[id="blockContentInner"]/p')
      list.to_html.split("\<br\>").each_with_index do |element,i|
        name = element.delete("\<p\>").delete("\<\/p\>").chomp.strip
        name_list << name
      end
      return name_list.sort
    end

src/plugin/text_info.rb#true_news

    def true_news news, old_news
      news - old_news
    end

src/plugin/text_info.rb#import_news

    def import_news agent=init_agent
      old_news = old_fachinfo_news
      news = fachinfo_news(agent)
      #updates = true_news fachinfo_news(agent), old_news
      #if names
      if update_name_list = true_news(news, old_news)
        import_name(update_name_list, agent)
        log_news news
        postprocess
        #import_name names, agent
        #log_news updates + old_news
        #postprocess
      end
      !update_name_list.empty?
      #!updates.empty?
    end

src/plugin/text_info.rb#log_news

    def log_news lines
      File.open @news_log, 'w' do |fh|
        fh.print lines.join("\n")
      end
    end

Reboot

oddb.org/bin/oddbd

Run

currency/bin/currencyd
oddb.org/bin/ext/fiparse/bin/fiparsed

Run update_textinfo_news

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Memo

Leave the current files, 'data' and 'log' directories, as they are

Result

Email ch.ODDB.org Report - Error: Fach- und Patienteninfo News - 02/2011

Plugin: ODDB::TextInfoPlugin
Error: EOFError
Message: end of file reached
Backtrace:
/usr/lib64/ruby/1.8/net/protocol.rb:133:in `sysread'
/usr/lib64/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'
/usr/lib64/ruby/1.8/timeout.rb:62:in `timeout'
/usr/lib64/ruby/1.8/timeout.rb:93:in `timeout'
/usr/lib64/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
/usr/lib64/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib64/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib64/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/lib64/ruby/1.8/net/http.rb:2009:in `read_new'
/usr/lib64/ruby/1.8/net/http.rb:1050:in `request'
/usr/lib64/ruby/gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:538:in `fetch_page'
/usr/lib64/ruby/gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:464:in `post_form'
/usr/lib64/ruby/gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:370:in `submit'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:378:in `submit_event'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:63:in `download_info'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:232:in `import_product'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:227:in `import_products'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:226:in `each'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:226:in `import_products'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:193:in `import_name'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:190:in `each'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:190:in `import_name'
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:211:in `import_news'
/home/masa/ywesee/oddb.org/src/util/updater.rb:482:in `send'
/home/masa/ywesee/oddb.org/src/util/updater.rb:482:in `update_notify_simple'
/home/masa/ywesee/oddb.org/src/util/updater.rb:462:in `call'
/home/masa/ywesee/oddb.org/src/util/updater.rb:462:in `wrap_update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:480:in `update_notify_simple'
/home/masa/ywesee/oddb.org/src/util/updater.rb:282:in `update_textinfo_news'
(eval):1:in `_admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1399:in `_admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `initialize'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `new'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1397:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'
bin/oddbd:38

Note

Error

Debug this error

Memo

I got some different errors
ch.ODDB.org Report - Error: Fach- und Patienteninfo News - 02/2011 end of file reached
ch.ODDB.org Report - Error: Fach- und Patienteninfo News - 02/2011 500 => Net::HTTPInternalServerError

Note

The rescue process is necessary for these errors
The 'update_textinfo_news' job succeeds in the case of several product names

Experiment

src/plugin/text_info.rb#import_news

    def import_news agent=init_agent
      old_news = old_fachinfo_news
      news = fachinfo_news(agent)
      #updates = true_news fachinfo_news(agent), old_news
      #updates = true_news(news, old_news)
      #if names
      if update_name_list = true_news(news, old_news)
update_name_list = update_name_list[0..5]
...

Delete the current files

masa@masa ~/ywesee/oddb.org $ rm -rf data/html/
masa@masa ~/ywesee/oddb.org $ rm -rf data/rss/
masa@masa ~/ywesee/oddb.org $ rm -rf log
masa@masa ~/ywesee/oddb.org $ mkdir log
masa@masa ~/ywesee/oddb.org $ touch log/fachinfo.txt
masa@masa ~/ywesee/oddb.org $ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   src/plugin/text_info.rb
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       log/
no changes added to commit (use "git add" and/or "git commit -a")

Reboot oddb.org/bin/oddbd

Run

currency/bin/currencyd
oddb.org/ext/fiparse/bin/fiparsed

Run update_textinfo_news

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for 3TC�, A. Vogel Rheuma-Tabletten, Abilify�, Abilify� Injektionsl�sung, Abseamed�, Aceril�- mite
Stored 4 Fachinfos
Ignored 1 Pseudo-Fachinfos
Ignored 1 up-to-date Fachinfo-Texts
Stored 2 Patinfos
Ignored 0 up-to-date Patinfo-Texts

Checked 0 companies
Unknown Iks-Numbers: 0
Fachinfos without iksnrs: 0
Session failures: 0
Download errors: 0
Parse Errors: 0

Check log/fachinfo.txt

masa@masa ~/ywesee/oddb.org $ head log/fachinfo.txt 
3TC�
A. Vogel Rheuma-Tabletten
Abilify�
Abilify� Injektionsl�sung
Abseamed�
Aceril�- mite
AcetaPhos� 750 mg
Acimethin�
Aclasta�
Actemra�

Note

It looks good

Experiment (delete one name from log/fachinfo.txt)

Result

Email ch.ODDB.org Report - Fach- und Patienteninfo News - 02/2011

Searched for Abilify�
Stored 0 Fachinfos
Ignored 0 Pseudo-Fachinfos
Ignored 2 up-to-date Fachinfo-Texts
Stored 0 Patinfos
Ignored 0 up-to-date Patinfo-Texts

Checked 0 companies
Unknown Iks-Numbers: 0
Fachinfos without iksnrs: 0
Session failures: 0
Download errors: 0
Parse Errors: 0

Note

Looks good

Final check (on browser)

Restore the database

Check Abilify Fachinformation on browser

Run

oddb.org/bin/oddbd
currency/bin/currenyd
oddb.org/ext/fiparse/bin/fiparsed

Run update_textinfo_news

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_textinfo_news

Check Abilify Fachinformation on browser

Note

The Fachinformation is updated
Not yet commit
Attach:patch.20110201.debug_fachnews.txt

Update test-cases

Check the current test-cases (before modification)

masa@masa ~/work/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/work/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
............................
Finished in 2.803692 seconds.

28 tests, 80 assertions, 0 failures, 0 errors

Note

No error and failure

Check the test-cases with the updated code

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
...E.F.......E....F....F....
Finished in 2.623608 seconds.

  1) Error:
test_extract_fachinfo_id(ODDB::TestTextInfoPlugin):
NoMethodError: undefined method `extract_fachinfo_id' for #<ODDB::TextInfoPlugin:0x7f9da86ce268>
    test/test_plugin/text_info.rb:780:in `test_extract_fachinfo_id'

  2) Failure:
test_fachinfo_news(ODDB::TestTextInfoPlugin) [test/test_plugin/text_info.rb:616]:
Exception raised:
Class: <NoMethodError>
Message: <"undefined method `to_html' for nil:NilClass">
---Backtrace---
/home/masa/ywesee/oddb.org/src/plugin/text_info.rb:106:in `fachinfo_news'
test/test_plugin/text_info.rb:617:in `test_fachinfo_news'
test/test_plugin/text_info.rb:616:in `test_fachinfo_news'
---------------

  3) Error:
test_import_news(ODDB::TestTextInfoPlugin):
NoMethodError: undefined method `to_html' for nil:NilClass
    /home/masa/ywesee/oddb.org/src/plugin/text_info.rb:106:in `fachinfo_news'
    /home/masa/ywesee/oddb.org/src/plugin/text_info.rb:158:in `import_news'
    test/test_plugin/text_info.rb:768:in `test_import_news'

  4) Failure:
test_old_fachinfo_news(ODDB::TestTextInfoPlugin) [test/test_plugin/text_info.rb:636]:
<["c413ce2d-a88e-4d71-b6b9-1c55e021edc0", "Amiodarone Winthrop\302\256/- Mite"]> expected but was
<"c413ce2d-a88e-4d71-b6b9-1c55e021edc0 Amiodarone Winthrop\302\256/- Mite">.

  5) Failure:
test_true_news(ODDB::TestTextInfoPlugin) [test/test_plugin/text_info.rb:658]:
<[]> expected but was
<[["01de437e-6568-4667-a3a6-00035098f59a",
  "Alcacyl\302\256 500 Instant-Pulver"],
 ["3ac0c14d-8c1a-4aed-9db6-f2dba58bc964", "Aldurazyme\302\256"],
 ["70893844-a876-4776-a61f-156e8465e47a", "Allopur\302\256"],
 ["3d808e28-3445-46e1-be00-68f053499bc1",
  "Allopurinol - 1 A Pharma100 mg/300 mg"],
 ["b287ecf9-84c2-48f0-b0c0-2dd9cff30d1f", "Amavita Acetylcystein 600"],
 ["78163fd4-6cf0-40ea-91b8-06c258722a7d", "Amavita Carbocistein"],
 ["1885f45a-b9df-4462-adb9-c46140859835", "Amavita Ibuprofen 400"],
 ["0a717d39-a873-4bff-87ab-b6b08b861da0", "Amavita Paracetamol 500"]]>.

28 tests, 70 assertions, 3 failures, 2 errors

Note

2 Errors, and 3 Failures

Update them one by one

1) Error: test_extract_fachinfo_id(ODDB::TestTextInfoPlugin)

Note

The method is deleted, since we do not use the Fachinformation ID anymore, so this test-case should be also deleted
Done

2) Failure: test_fachinfo_news(ODDB::TestTextInfoPlugin)

Note

The HTML code and logic have change
I should update the HTML source code for the test

Update test/data/html/text_info/News.html

    def test_fachinfo_news
      mapping = [
        [ 'News.html',
          :get,
          ODDB.config.text_info_newssource,
        ],
      ]
      agent = setup_mechanize mapping
      news = nil
      assert_nothing_raised do
        news = @plugin.fachinfo_news agent
      end
      assert_equal 1525, news.size
      assert_equal "3TC\302\256", news.first
    end

Result

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
.
Finished in 0.011875 seconds.

1 tests, 3 assertions, 0 failures, 0 errors

3) Error: test_import_news(ODDB::TestTextInfoPlugin)

Note

This is a 'join test', which means the method, import_news, calls the other tested methods
The HTML source code also should be updated

    def test_import_news
      logfile = File.join @vardir, 'fachinfo.txt'
      File.open logfile, 'w' do |fh|
        fh.puts "8a7f708c-c738-4425-a9a5-5ad294f20be4 Aclasta\302\256"
      end
      mapping = [
        [ 'News.html',
          :get,
          ODDB.config.text_info_newssource,
        ],
        [ 'SearchForm.html',
          :get,
          'http://textinfo.ch/Search.aspx',
        ],
        [ 'ResultProduct.html',
          :submit,
          'Search.aspx',
        ],
        [ 'Aclasta.de.html',
          :submit,
          'Result.aspx?lang=de',
        ],
        [ 'Aclasta.fr.html',
          :get,
          'Result.aspx?lang=fr',
        ],
      ]
      agent = setup_mechanize mapping
      @parser.should_receive(:parse_fachinfo_html).and_return FachinfoDocument.new
      @parser.should_receive(:parse_patinfo_html).and_return PatinfoDocument.new
      @app.should_receive(:sorted_fachinfos).and_return []
      success = @plugin.import_news agent
      expected = "Abilify\302\256\nAbilify\302\256 Injektionsl\303\266sung\nAbseamed\302\256\nAceril\302\256- mite\nAcetaPhos\302\256 750 mg\nAcimethin\302\256\nAclasta\302\256"
      assert_equal 5, @pages.size
      assert_equal expected, File.read(logfile)
      assert_equal true, success
    end

Result

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
.
Finished in 0.599557 seconds.

1 tests, 3 assertions, 0 failures, 0 errors

4) Failure: test_old_fachinfo_news(ODDB::TestTextInfoPlugin)

Note

The file format has change

    def test_old_fachinfo_news
      ## no file means no news
      assert_equal [], @plugin.old_fachinfo_news
      File.open File.join(@vardir, 'fachinfo.txt'), 'w' do |fh|
        fh.puts <<-EOS
Amiodarone Winthrop\302\256/- Mite
AcetaPhos\302\256 750 mg
        EOS
      end
      ## the file is parsed properly
      news = @plugin.old_fachinfo_news
      assert_equal 2, news.size
      assert_equal "Amiodarone Winthrop\302\256/- Mite", news.first
    end

Result

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
.
Finished in 0.000743 seconds.

1 tests, 3 assertions, 0 failures, 0 errors

5) Failure: test_true_news(ODDB::TestTextInfoPlugin)

Note

Actually this does not have to be tested, since the process is only the subtraction of Arrays
But the process of comparison has became a little bit different

    def test_true_news
      ## there are no news
      news = [
        "Abseamed\302\256",
        "Aclasta\302\256",
        "Alcacyl\302\256 500 Instant-Pulver",
        "Aldurazyme\302\256",
        "Allopur\302\256",
        "Allopurinol - 1 A Pharma100 mg/300 mg",
        "Amavita Acetylcystein 600",
        "Amavita Carbocistein",
        "Amavita Ibuprofen 400",
        "Amavita Paracetamol 500"
      ]
      old_news = [
        "Abseamed\302\256",
        "Aclasta\302\256",
      ]
      expected_news = [
        "Alcacyl\302\256 500 Instant-Pulver",
        "Aldurazyme\302\256",
        "Allopur\302\256",
        "Allopurinol - 1 A Pharma100 mg/300 mg",
        "Amavita Acetylcystein 600",
        "Amavita Carbocistein",
        "Amavita Ibuprofen 400",
        "Amavita Paracetamol 500"
      ]
      assert_equal expected_news, @plugin.true_news(news, old_news)
      ## clean disection
      old_news = [
        "Allopurinol - 1 A Pharma100 mg/300 mg",
        "Amavita Acetylcystein 600",
        "Amavita Carbocistein",
        "Amavita Ibuprofen 400",
        "Amavita Paracetamol 500"
      ]
      expected = [
        "Abseamed\302\256",
        "Aclasta\302\256",
        "Alcacyl\302\256 500 Instant-Pulver",
        "Aldurazyme\302\256",
        "Allopur\302\256",
      ]
      assert_equal expected, @plugin.true_news(news, old_news)
      ## recorded news don't appear on the news-page
      old_news = ["Amiodarone Winthrop\302\256/- Mite"]
      assert_equal news, @plugin.true_news(news, old_news)
  end

Result

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
.
Finished in 0.000697 seconds.

1 tests, 3 assertions, 0 failures, 0 errors

Check all the test-cases

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/text_info.rb 
/home/masa/ywesee/oddb.org/src/view/changelog.rb:26: warning: already initialized constant SORT_DEFAULT
Loaded suite test/test_plugin/text_info
Started
...........................
Finished in 3.131211 seconds.

27 tests, 77 assertions, 0 failures, 0 errors

Commit

Debug Fach- and Patinfo News job, since the web site is updated (oddb.org)

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110201-debug-fach_news-oddb_org

Confirm the current status

Merge the grabing script

Create a GitHub account

Simplify the code

Update test-cases

1) Error: test_extract_fachinfo_id(ODDB::TestTextInfoPlugin)

2) Failure: test_fachinfo_news(ODDB::TestTextInfoPlugin)

3) Error: test_import_news(ODDB::TestTextInfoPlugin)

4) Failure: test_old_fachinfo_news(ODDB::TestTextInfoPlugin)

5) Failure: test_true_news(ODDB::TestTextInfoPlugin)

Check all the test-cases

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110201-debug-fach_news-oddb_org

Confirm the current status

Merge the grabing script

Create a GitHub account

Simplify the code

Update test-cases

1) Error: test_extract_fachinfo_id(ODDB::TestTextInfoPlugin)

2) Failure: test_fachinfo_news(ODDB::TestTextInfoPlugin)

3) Error: test_import_news(ODDB::TestTextInfoPlugin)

4) Failure: test_old_fachinfo_news(ODDB::TestTextInfoPlugin)

5) Failure: test_true_news(ODDB::TestTextInfoPlugin)

Check all the test-cases

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler