view · edit · sidebar · attach · print · history

20110112-check-followers-autorun

<< | Index | >>

  1. Check log file of update_bsv bsv_followers online suspend
  2. How is the pharmacode update from RefData or MEDDATA? suspend
  3. Check update_bsv second mail prcess
Goal
  • Check autorun bsv_followers swissmedic_followers / 60%
Milestones
  1. Check log online
  2. Test locally
Summary
Commits
ToDo Tomorrow
Keep in Mind
  1. Limitation Ticket 248
  2. Check Ruby 1.8.6 with oniguruma patch on Win7
  3. swissmedic_followers debug
  4. On Ice
  5. emerge --sync

Check log file of update_bsv, bsv_followers online

It seems that bsv_followers method ran two times in this year 2011. 

 2011-01-02 08:48:19 CET getin update_bsv_followers
 2011-01-11 13:17:09 CET getin update_bsv_followers

Note

  • Looks no problem

Davatz-san said

  • The problem is that the second mail of update_bsv, entitled ch.ODDB.org Report - SL-Update (XML), does not come
  • For example, on the 2nd of January, the first mail came but the second mail did not come

Next

  • Run update_bsv locally
  • Compare the log files between local and online

Run update_bsv locally

Preparation (comment out update_bsv_followers)

src/util/updater.rb#update_bsv_followers 

    def update_bsv_followers

      LogFile.append('oddb/debug', " getin update_bsv_followers", Time.now)

=begin
      update_trade_status
      update_medwin_packages
      update_lppv
      update_price_feeds
      export_oddb_csv
      # export_oddb2_csv # Disabled 4.1.2010
      export_ouwerkerk
      export_generics_xls
      export_competition_xlss
=end
    end

Run

  • currencyd
  • bin/oddbd
  • bin/meddatad

Run umport_bsv job 

masa@masa ~/ywesee/oddb.org $ jobs/import_bsv

Result 

2011-01-12 09:37:46 CET getin update_bsv
2011-01-12 09:37:50 CET getin BsvXmlPlugin.update
2011-01-12 09:37:50 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2011-01-12 09:37:50 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2011-01-12 09:37:50 CET getin download_file
2011-01-12 09:37:54 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip
2011-01-12 09:37:54 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2011-01-12 09:37:54 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = false
2011-01-12 09:37:54 CET path = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip"
2011-01-12 09:49:27 CET return_value_BsvXmlPlugin.update = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip"
2011-01-12 09:49:27 CET getin log_notify_bsv
2011-01-12 09:49:27 CET date=Tue, 01 Feb 2011
2011-01-12 09:49:27 CET after pointer creating
2011-01-12 09:49:50 CET after log_info(plug)
2011-01-12 09:49:50 CET before @app.update
2011-01-12 09:49:50 CET after @app.update
2011-01-12 09:49:50 CET getin Log.notify (SL-Update)
2011-01-12 09:49:50 CET @files={}
2011-01-12 09:49:50 CET start outgoing process
2011-01-12 09:49:50 CET before send_mail(outgoing)
2011-01-12 09:49:50 CET getin send_mail
2011-01-12 09:49:50 CET @recipients=["mhatakeyama@ywesee.com"]
2011-01-12 09:49:50 CET config.smtp_server="smtp.gmail.com"
2011-01-12 09:49:50 CET config.smtp_port=587
2011-01-12 09:49:50 CET config.smtp_domain="ywesee.com"
2011-01-12 09:49:50 CET config.smtp_user="mhatakeyama@ywesee.com"
2011-01-12 09:49:50 CET config.smtp_pass="xxx"
2011-01-12 09:49:50 CET config.smtp_authtype=:plain
2011-01-12 09:49:51 CET getin Net::SMTP
2011-01-12 09:49:51 CET recipient=mhatakeyama@ywesee.com
2011-01-12 09:49:54 CET return_value_log_notify = ["mhatakeyama@ywesee.com"]
2011-01-12 09:49:54 CET getin Log.notify (SL-Update)
2011-01-12 09:49:54 CET @files={}
2011-01-12 09:49:54 CET start outgoing process
2011-01-12 09:49:54 CET before send_mail(outgoing)
2011-01-12 09:49:54 CET getin send_mail
2011-01-12 09:49:54 CET @recipients=["mhatakeyama@ywesee.com"]
2011-01-12 09:49:54 CET config.smtp_server="smtp.gmail.com"
2011-01-12 09:49:54 CET config.smtp_port=587
2011-01-12 09:49:54 CET config.smtp_domain="ywesee.com"
2011-01-12 09:49:54 CET config.smtp_user="mhatakeyama@ywesee.com"
2011-01-12 09:49:54 CET config.smtp_pass="xxx"
2011-01-12 09:49:54 CET config.smtp_authtype=:plain
2011-01-12 09:49:55 CET getin Net::SMTP
2011-01-12 09:49:55 CET recipient=mhatakeyama@ywesee.com
2011-01-12 09:49:57 CET return_value_log2_notify = ["mhatakeyama@ywesee.com"]
2011-01-12 09:49:57 CET getin update_bsv_followers

Compare log files

suspend

How is the pharmacode update from RefData or MEDDATA?

Email from Davatz-san Following question for the meddatad deamon

  • What happens if there is a Pharmacode in the Cache but the BSV-Followers (Swissmedic-Followers, meddatad) does not find a Pharmacode in Refdata.ch?
  • Will the Pharmacode in the Cache remain or will it be set to 0 again?

Memo

  • Temodal pharmacode was updated without running the updated update_bsv code.
  • Davatz-san executed update_bsv_followers and update_swissmedic_followers.
  • update_bsv_followers or update_swissmedic_followers may update the cache pharmacode from RefData or MedData
  • I would like to confirm it more clearly.

src/util/updater.rb#update_bsv_followers 

    def update_bsv_followers
      update_trade_status
      update_medwin_packages
...

src/util/updater.rb#update_swissmedic_followers 

    def update_swissmedic_followers
            update_trade_status
            update_medwin_packages
            reconsider_bsv :new_log => true
            update_medwin_companies
...

Note

  • 'update_medwin_packages' looks important

Experiment

  • Reload the oddb.org data 20101222
  • Check Temodal pharmacode on browser
  • Run only update_medwin_packages

Run

  • bin/oddbd
  • ext/meddata/bin/meddatad
  • currencyd
  • yusd

Confirmation

  • Temodal pharmacode is '0'

Run update_medwin_packages 

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_medwin_packages

Result

  • The pharmacode is not updated

suspend

Check update_bsv second mail prcess

Check xml on server 

3160927 2010-12-11 03:01 XMLPublications-2010.12.11.zip
3343728 2011-01-02 03:01 XMLPublications-2011.01.02.zip
3356354 2011-01-04 03:01 XMLPublications-2011.01.04.zip
3356354 2011-01-11 16:29 XMLPublications-2011.01.11.zip
3356354 2011-01-12 07:37 XMLPublications-2011.01.12.zip
3356354 2011-01-12 07:37 XMLPublications-latest.zip

Note

  • The xml file online was updated on 2011.01.02 and 2011.01.04

Compare the log files between local and online

On 2011.01.01 (online) 

2011-01-01 03:01:26 CET getin update_bsv
2011-01-01 03:01:31 CET getin BsvXmlPlugin.update
2011-01-01 03:01:31 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2011-01-01 03:01:31 CET save_dir   = /var/www/oddb.org/data/xml
2011-01-01 03:01:31 CET getin download_file
2011-01-01 03:01:33 CET save_file   = /var/www/oddb.org/data/xml/XMLPublications-2011.01.01.zip
2011-01-01 03:01:33 CET latest_file = /var/www/oddb.org/data/xml/XMLPublications-latest.zip
2011-01-01 03:01:33 CET File.exists?(/var/www/oddb.org/data/xml/XMLPublications-latest.zip) = true
2011-01-01 03:01:33 CET FileUtils.compare_file(/tmp/foo.9367.0, /var/www/oddb.org/data/xml/XMLPublications-latest.zip) = true
2011-01-01 03:01:33 CET path = nil
2011-01-01 03:01:33 CET return_value_BsvXmlPlugin.update = nil

Note

  • The system runs 'update_bsv' but the xml file is not new, so that the 'update_bsv' stopped

on 2011.01.02 (online) 

2011-01-02 03:01:14 CET getin update_bsv
2011-01-02 03:01:20 CET getin BsvXmlPlugin.update
2011-01-02 03:01:20 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2011-01-02 03:01:20 CET save_dir   = /var/www/oddb.org/data/xml
2011-01-02 03:01:20 CET getin download_file
2011-01-02 03:01:22 CET save_file   = /var/www/oddb.org/data/xml/XMLPublications-2011.01.02.zip
2011-01-02 03:01:22 CET latest_file = /var/www/oddb.org/data/xml/XMLPublications-latest.zip
2011-01-02 03:01:22 CET File.exists?(/var/www/oddb.org/data/xml/XMLPublications-latest.zip) = true
2011-01-02 03:01:22 CET FileUtils.compare_file(/tmp/foo.9340.0, /var/www/oddb.org/data/xml/XMLPublications-latest.zip) = false
2011-01-02 03:01:22 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2011.01.02.zip"
2011-01-02 04:06:30 CET return_value_BsvXmlPlugin.update = "/var/www/oddb.org/data/xml/XMLPublications-2011.01.02.zip"

Note

  • The xml file online was updated, and 'update_bsv' ran

Then 

2011-01-02 04:06:30 CET getin log_notify_bsv
2011-01-02 04:06:30 CET date=Sat, 01 Jan 2011
2011-01-02 04:06:30 CET after pointer creating
2011-01-02 04:07:37 CET after log_info(plug)
2011-01-02 04:07:37 CET before @app.update
2011-01-02 04:07:39 CET after @app.update
2011-01-02 04:07:39 CET getin Log.notify (SL-Update)
2011-01-02 04:07:39 CET @files={}
2011-01-02 04:07:39 CET start outgoing process
2011-01-02 04:07:39 CET before send_mail(outgoing)
2011-01-02 04:07:39 CET getin send_mail
2011-01-02 04:07:39 CET @recipients=["xxx", "xxx", "zdavatz@ywesee.com", "hwyss@ywesee.com", "mhatakeyama@ywesee.com"]

2011-01-02 04:07:39 CET config.smtp_server="smtp.gmail.com"
2011-01-02 04:07:39 CET config.smtp_port=587
2011-01-02 04:07:39 CET config.smtp_domain="ywesee.com"
2011-01-02 04:07:39 CET config.smtp_user="admin@ywesee.com"
2011-01-02 04:07:39 CET config.smtp_pass="xxx"
2011-01-02 04:07:39 CET config.smtp_authtype=:plain
2011-01-02 04:07:42 CET getin Net::SMTP
2011-01-02 04:07:42 CET recipient=xxx
2011-01-02 04:07:45 CET recipient=xxx
2011-01-02 04:07:47 CET recipient=zdavatz@ywesee.com
2011-01-02 04:07:49 CET recipient=hwyss@ywesee.com
2011-01-02 04:07:49 CET getin Log.notify (SL-Update)
2011-01-02 04:07:49 CET @files={}

Note

  • 'update_bsv' was done and a report mail was sent
  • but there is no name of mine, recipient=mhatakeyama@ywesee.com!!

Then 

2011-01-02 04:07:49 CET start outgoing process
2011-01-02 04:07:49 CET before send_mail(outgoing)
2011-01-02 04:07:49 CET getin send_mail
2011-01-02 04:07:49 CET @recipients=["zdavatz@ywesee.com", "hwyss@ywesee.com", "mhatakeyama@ywesee.com"]

Note

  • An email was sent again (This is probably the second mail) but the recipients were different from the first one

Then 

2011-01-02 08:48:19 CET getin update_bsv_followers

Note

  • About 5 hours later, 'update_bsv_followers' ran

Local 

2011-01-12 13:57:46 CET getin update_bsv
2011-01-12 13:57:50 CET getin BsvXmlPlugin.update
2011-01-12 13:57:50 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2011-01-12 13:57:50 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2011-01-12 13:57:50 CET getin download_file
2011-01-12 13:57:53 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip
2011-01-12 13:57:53 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2011-01-12 13:57:53 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = false
2011-01-12 13:57:54 CET path = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip"
2011-01-12 14:09:20 CET return_value_BsvXmlPlugin.update = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2011.01.12.zip"

Note

  • New xml file is saved and 'update_bsv' run

Then 

2011-01-12 14:09:20 CET getin log_notify_bsv
...
2011-01-12 14:09:45 CET getin Log.notify (SL-Update)
...
2011-01-12 14:09:48 CET return_value_log_notify = ["mhatakeyama@ywesee.com"]

2011-01-12 14:09:48 CET getin Log.notify (SL-Update)
...
2011-01-12 14:09:51 CET return_value_log2_notify = ["mhatakeyama@ywesee.com"]

2011-01-12 14:09:51 CET getin update_bsv_followers

Note

  • The 'Log.notify(SL-Update)' is the certificate of sending an email

The problem is (online) 

2011-01-02 04:06:30 CET return_value_BsvXmlPlugin.update = "/var/www/oddb.org/data/xml/XMLPublications-2011.01.02.zip"
...
2011-01-02 04:07:39 CET getin Log.notify (SL-Update)
...
2011-01-02 04:07:49 CET config.smtp_authtype=:plain
2011-01-02 04:07:42 CET getin Net::SMTP
2011-01-02 04:07:47 CET recipient=zdavatz@ywesee.com
2011-01-02 04:07:49 CET recipient=hwyss@ywesee.com
...
2011-01-02 04:07:49 CET getin Log.notify (SL-Update)
...
2011-01-02 04:07:49 CET config.smtp_authtype=:plain
2011-01-02 04:07:49 CET return_value_update_bsv=nil

Notes

  • 'return_value_update_bsv' is nil so 'update_bsv_followers' does not run anyway
  • 'Log.notify (SL-Update)' process is called two times but
  • there are not 'return_value_log_notify' and 'return_value_log_notify2'
  • Also, 'getin Net::SMTP' did not come in the second mail process

Check source code

src/util/log.rb#send_mail 

        def send_mail(multipart)
      LogFile.append('oddb/debug', " getin send_mail", Time.now)
      LogFile.append('oddb/debug', " @recipients=" + @recipients.inspect.to_s, Time.now)

      config = ODDB.config
      LogFile.append('oddb/debug', " config.smtp_server=" + config.smtp_server.inspect.to_s, Time.now)
      LogFile.append('oddb/debug', " config.smtp_port=" + config.smtp_port.inspect.to_s, Time.now)
      LogFile.append('oddb/debug', " config.smtp_domain=" + config.smtp_domain.inspect.to_s, Time.now)
      LogFile.append('oddb/debug', " config.smtp_user=" + config.smtp_user.inspect.to_s, Time.now)
      LogFile.append('oddb/debug', " config.smtp_pass=" + config.smtp_pass.inspect.to_s, Time.now)
      LogFile.append('oddb/debug', " config.smtp_authtype=" + config.smtp_authtype.inspect.to_s, Time.now)
      Net::SMTP.start(config.smtp_server, config.smtp_port, config.smtp_domain,
                      config.smtp_user, config.smtp_pass,
                      config.smtp_authtype) { |smtp|
      LogFile.append('oddb/debug', " getin Net::SMTP", Time.now)

                @recipients.each { |recipient|
      LogFile.append('oddb/debug', " recipient=" + recipient.to_s, Time.now)
                    multipart.to = [recipient]
                    smtp.sendmail(multipart.encoded,
                                                @mail_from || config.smtp_user, recipient)
                }
            }
        end

Note

  • It means that 'Net::SMTP.start' block did not run
  • because 'getin Net::SMTP' did not come

Question

  • If 'Net::SMTP.start' fails, what value is returned?

Experiment

src/util/log.rb 

LogFile.append('oddb/debug', " @recipients=" + @recipients.inspect.to_s, Time.now)
LogFile.append('oddb/debug', " self::class::MAIL_TO=" + self::class::MAIL_TO.to_s, Time.now)
LogFile.append('oddb/debug', " self::class=" + self::class.to_s, Time.now)
            @recipients = (@recipients + self::class::MAIL_TO).uniq
            outgoing.subject = subj
            outgoing.date = Time.now
            outgoing['User-Agent'] = 'ODDB Updater'

      LogFile.append('oddb/debug', " before send_mail(outgoing)", Time.now)
            send_mail(outgoing)
        end

Result (local)

log/oddb/debug/2011/01.log 

...
2011-01-12 16:09:50 CET getin Log.notify (SL-Update)
...
2011-01-12 16:09:50 CET @recipients=["mhatakeyama@ywesee.com"]
2011-01-12 16:09:50 CET self::class::MAIL_TO=mhatakeyama@ywesee.com
2011-01-12 16:09:50 CET self::class=ODDB::Log
...
2011-01-12 16:09:54 CET getin Log.notify (SL-Update)
...
2011-01-12 16:09:54 CET @recipients=["mhatakeyama@ywesee.com", "mhatakeyama@ywesee.com"]
2011-01-12 16:09:54 CET self::class::MAIL_TO=mhatakeyama@ywesee.com
2011-01-12 16:09:54 CET self::class=ODDB::Log
...

Note

  • This is quite normal as expected
  • I expect 'self::class' becomes different from ODDB::Log online when the second mail fails

Commit

view · edit · sidebar · attach · print · history
Page last modified on January 12, 2011, at 05:03 PM