20101101-update-update_bsv

<< Masa.20101102-debug-oddb-csv_export | 2010 | Masa.20101029-debug-bsv_follower-autorun >>

1. Check the server log of update_bsv (oddb.org)
2. Run import_bsv manually on production server
3. Update download_file method for update_bsv and update_swissmedic
4. Replace the download_file method by the current download_to method

Goal

1. Update update_bsv and update_swissmedic / 100%
2. Debug ComparisonDeCh / 70%

Milestones

1. Check server log 8:00
2. Run import_bsv manually on production server 10:20
   • Check the result later
3. Update download_file method 14:30
4. Replace the download_file method by the current download_to method
5. Test import_bsv locally
6. Push and pull on production server

Summary

• http://masay.ch/2010/11/01/update-update_bsv/

Commits

• Updated test_plugin/bsv_xml.rb not to create a test data file (oddb.org)
• Added download_file method for update_bsv and update_swissmedic methods (oddb.org)
• Replaced download_to method to download_file method (oddb.org)

ToDo Tomorrow

• Update update_swissmedic

Keep in Mind

• emerge --sync
• Debug ODDB::CsvExportPlugin
• Mon Nov 1 04:09:08 2010: ODDB::Export::Xls::ComparisonDeCh
• Update price feeds did not run automatically
• Replace get_latest_file (update_swissmedic) method by download_file method
• If update_bsv goes well on 1st Dec., let's think about deleting debug print code
• delete-first in update_bsv
• arrange files and back up script oldie
• arrange ydpm job list
• xls2odat argument files
• Search a fundamental cause of export_chde_xls bug
• adding a prefix for the pharmacode field in xls2odat
• Delete all SL-Entries before importing the new flags.
• 20100922-trace-limitation-data-flow-in-bsvXml?action=edit
  • 20100922 Options for LimitationNiveau
• 20100908 rpdf2txt GPLv3
• Tokyo Tyrant + ODBA
• admin tool hack
• Organize the information of yus README
• Organize the information of de.oddb.org README
• block sample http://q.hatena.ne.jp/1217573703 (japanese)

Attached Files

Check the server log of update_bsv (oddb.org)

$ cat oddb.org/log/oddb/debug/2010/10.log 
2010-10-30 03:01:22 CEST getin update_bsv
2010-10-30 03:01:28 CEST getin BsvXmlPlugin.update
2010-10-30 03:01:30 CEST path = "/var/www/oddb.org/data/xml/XMLPublications-2010.10.30.zip"
2010-10-30 03:01:30 CEST @latest = nil
2010-10-30 03:01:30 CEST File.exist?(@latest) = true
2010-10-30 03:01:30 CEST FileUtils.cmp(@latest, path) = true
2010-10-30 03:01:30 CEST FileUtils.rm /var/www/oddb.org/data/xml/XMLPublications-2010.10.30.zip 
2010-10-30 03:01:30 CEST return_value_BsvXmlPlugin.update = nil
2010-10-30 03:01:30 CEST return_value_update_bsv=nil

2010-10-31 03:01:14 CET getin update_bsv
2010-10-31 03:01:20 CET getin BsvXmlPlugin.update
2010-10-31 03:01:22 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2010.10.31.zip"
2010-10-31 03:01:22 CET @latest = nil
2010-10-31 03:01:22 CET File.exist?(@latest) = true
2010-10-31 03:01:22 CET FileUtils.cmp(@latest, path) = true
2010-10-31 03:01:22 CET FileUtils.rm /var/www/oddb.org/data/xml/XMLPublications-2010.10.31.zip 
2010-10-31 03:01:22 CET return_value_BsvXmlPlugin.update = nil
2010-10-31 03:01:22 CET return_value_update_bsv=nil

$ cat oddb.org/log/oddb/debug/2010/11.log 
2010-11-01 03:01:28 CET getin update_bsv
2010-11-01 03:01:34 CET getin BsvXmlPlugin.update
2010-11-01 03:01:36 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2010.11.01.zip"
2010-11-01 03:01:36 CET @latest = nil
2010-11-01 03:01:36 CET File.exist?(@latest) = true
2010-11-01 03:01:36 CET FileUtils.cmp(@latest, path) = true
2010-11-01 03:01:36 CET FileUtils.rm /var/www/oddb.org/data/xml/XMLPublications-2010.11.01.zip 
2010-11-01 03:01:36 CET return_value_BsvXmlPlugin.update = nil
2010-11-01 03:01:36 CET return_value_update_bsv=nil

Notes

• update_bsv is called everday, but update_bsv stops and update_bsv_followers does not run, because

2010-10-30 03:01:30 CEST File.exist?(@latest) = true
2010-10-30 03:01:30 CEST FileUtils.cmp(@latest, path) = true

• Namely, the latest file on production server is the same as online
• This is normal as I expected

Run import_bsv manually on production server

• Email A new xml file is up

 $ cd /var/www/oddb.org
 $ su
 # sudo -b -u xxx jobs/import_bsv

Notes

• We have to run jobs/import_bsv by the different user, because the log directory now belongs to the user.

Report mails

Log

New files

Update download_file method for update_bsv and update_swissmedic

ToDo

1. Update test/test_plugin/bsv_xml.rb
2. Update src/plugin/bsv_xml.rb
3. Test update_bsv locally
4. Update test/test_plugin/swissmedic.rb
5. Update src/plugin/swissmedic.rb
6. Test update_swissmedic locally
7. Commit

Check test cases

Run oddb.org/ext/meddata/bin/meddatad

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/bsv_xml.rb 
Loaded suite test/test_plugin/bsv_xml
Started
.........
Finished in 0.117352 seconds.

9 tests, 68 assertions, 0 failures, 0 errors

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/swissmedic.rb 
Loaded suite test/test_plugin/swissmedic
Started
................................
Finished in 1.201945 seconds.

32 tests, 110 assertions, 0 failures, 0 errors

Update test/test_plugin/bsv_xml.rb

Refer

• test/test_plugin/bsv_xml.rb#test_download

Commit

• Updated test_plugin/bsv_xml.rb not to create a test data file

Make a test case which will fail

    def test_download_file
      assert(false)
    end

Check it certainly fails

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/bsv_xml.rb 
Loaded suite test/test_plugin/bsv_xml
Started
.F........
Finished in 0.117761 seconds.

  1) Failure:
test_download_file(ODDB::TestBsvXmlPlugin) [test/test_plugin/bsv_xml.rb:696]:
<false> is not true.

10 tests, 69 assertions, 1 failures, 0 errors

Create test cases

    def test_download_file
      # Preparing variables
      target_url = @url
      save_dir = File.expand_path 'var', File.dirname(__FILE__)
      file_name = "XMLPublications.zip"

      online_file = @zip
      temp_file = File.join save_dir, 'temp.zip'
      save_file = File.join save_dir,
               Date.today.strftime("XMLPublications-%Y.%m.%d.zip")
      latest_file = File.join save_dir, 'XMLPublications-latest.zip'

      # Preparing mock objects
      flexstub(Tempfile).should_receive(:new).and_return do
        flexmock do |tempfile|
          tempfile.should_receive(:close)
          tempfile.should_receive(:unlink)
          tempfile.should_receive(:path).and_return(temp_file)
        end
      end

      fileobj = flexmock do |obj|
        obj.should_receive(:save_as).with(temp_file).and_return do
          FileUtils.cp online_file, temp_file   # instead of downloading
        end
        obj.should_receive(:save_as).with(save_file).and_return do
          FileUtils.cp online_file, save_file   # instead of downloading
        end
      end
      flexstub(Mechanize) do |mechclass|
        mechclass.should_receive(:new).and_return do
          flexmock do |mechobj|
            mechobj.should_receive(:get).and_return(fileobj)
          end
        end
      end

      # Downloading tests
      result = nil
      assert_nothing_raised do
        result = @plugin.download_file(target_url, save_dir, file_name)
      end
      assert_equal latest_file, result

      # Not-downloading tests
      assert_nothing_raised do
        result = @plugin.download_file(target_url, save_dir, file_name)
      end
      assert_equal nil, result

      # Check files
      assert File.exist?(save_file), "download to #{save_file} failed."
      assert File.exist?(latest_file), "download to #{latest_file} failed."
    ensure
      FileUtils.rm_r save_dir if File.exists? save_dir
    end

Make a new method (download_file)

    def download_file(target_url, save_dir, file_name)
      FileUtils.mkdir_p save_dir   # if there is it already, do nothing

      target_file = Mechanize.new.get(target_url)
      save_file = File.join save_dir,
               Date.today.strftime(file_name.gsub(/\./,"-%Y.%m.%d."))
      latest_file = File.join save_dir,
               Date.today.strftime(file_name.gsub(/\./,"-latest."))

      # download target_file temporarily
      temp = Tempfile.new('foo')
      temp_file = temp.path
      target_file.save_as temp_file

      # check and compare the latest file and save
      if(File.exists?(latest_file) && FileUtils.compare_file(temp_file, latest_file))
        return nil
      else
        target_file.save_as save_file
        FileUtils.cp(save_file, latest_file)
        return latest_file
      end
    rescue EOFError
      retries ||= 10
      if retries > 0
        retries -= 1
        sleep 10 - retries
        retry
      else
        raise
      end
    ensure
      temp.close
      temp.unlink
    end

Check test cases

masa@masa ~/ywesee/oddb.org $ ruby test/test_plugin/bsv_xml.rb 
Loaded suite test/test_plugin/bsv_xml
Started
..........
Finished in 0.144605 seconds.

10 tests, 74 assertions, 0 failures, 0 errors

Commit

• Added download_file method for update_bsv and update_swissmedic methods.

Replace the download_file method by the current download_to method

src/plugin/bsv_xml.rb

    def update
      save_dir = File.join ARCHIVE_PATH, 'xml'
      target_url = ODDB.config.url_bag_sl_zip
      file_name = "XMLPublications.zip"

      if(path = donwload_file(target_url, save_dir, file_name))
        _update path
      end
      path
    end

Result

• Email
  1. ch.ODDB.org Report - SL-Update (XML) - 11/2010 NoError
  2. ch.ODDB.org Report - SL-Update (XML) - 11/2010 NoError

Log

2010-11-01 15:38:01 CET getin update_bsv
2010-11-01 15:38:05 CET getin BsvXmlPlugin.update
2010-11-01 15:38:05 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-01 15:38:05 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2010-11-01 15:38:05 CET getin download_file
2010-11-01 15:38:09 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.01.zip
2010-11-01 15:38:09 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-01 15:38:09 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = false
2010-11-01 15:38:09 CET path = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.01.zip
2010-11-01 15:48:51 CET return_value_BsvXmlPlugin.update = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.01.zip"
2010-11-01 15:48:51 CET getin log_notify_bsv
2010-11-01 15:49:12 CET getin Log.notify (SL-Update)
2010-11-01 15:49:17 CET return_value_log_notify = ["mhatakeyama@ywesee.com"]
2010-11-01 15:49:17 CET getin Log.notify (SL-Update)
2010-11-01 15:49:21 CET return_value_log2_notify = ["mhatakeyama@ywesee.com"]
2010-11-01 15:49:21 CET return_value_update_bsv=["mhatakeyama@ywesee.com"]

New files

masa@masa ~/ywesee/oddb.org $ git status
# On branch work
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   src/plugin/bsv_xml.rb
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       data/xml/
#       log/
no changes added to commit (use "git add" and/or "git commit -a")

masa@masa ~/ywesee/oddb.org $ ls data/xml/ -al
insgesamt 6688
drwxr-xr-x 2 masa masa      16  1. Nov 15:38 .
drwxr-xr-x 9 masa masa      72  1. Nov 15:34 ..
-rw-r--r-- 1 masa masa 3420454  1. Nov 15:38 XMLPublications-2010.11.01.zip
-rw-r--r-- 1 masa masa 3420454  1. Nov 15:38 XMLPublications-latest.zip

Run it again

Log

masa@masa ~/ywesee/oddb.org $ vim log/oddb/debug/2010/11.log
2010-11-01 15:57:10 CET getin update_bsv
2010-11-01 15:57:14 CET getin BsvXmlPlugin.update
2010-11-01 15:57:14 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-01 15:57:14 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2010-11-01 15:57:14 CET getin download_file
2010-11-01 15:57:17 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.01.zip
2010-11-01 15:57:17 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-01 15:57:18 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = true
2010-11-01 15:57:18 CET FileUtils.compare_file(/tmp/foo.11392.0, /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = true
2010-11-01 15:57:18 CET path = nil
2010-11-01 15:57:18 CET return_value_BsvXmlPlugin.update = nil
2010-11-01 15:57:18 CET return_value_update_bsv=nil

Notes

• Looks good

Commit

• Replaced download_to method to download_file method.