20110907-update-importer-testcase-spreadsheet-testcases-migel

<< Masa.20110908-debug-importer-testcases-migel | 2011 | Masa.20110906-update-importer-testcases-migel >>

1. emerge --sync error
2. Update importer migel (products)
3. Update a testcase of spreadsheet
4. Testcases migel

Goal/Estimate/Evaluation:

• Update importer migel / 100% / 90%

Milestones

1. update importer
2. testcases spreadsheet
3. testcases migel

Summary

• http://masaywesee.wordpress.com/2011/09/07/import-migel-data-spec-migel/

Commits

• Fixed a frozen string bug thanks to dblock (Daniel Doubrovkine (dB.))

ToDo

• Testcases oddb.org, migel
  • at the same time, refactoring Group, Subgroup, Migelid because some code in the classes are totally same
• refactor rebuild_fulltext_index_tables
• delete_migelids in lib/migel/util/server.rb
• 'could not sort' warning -> this is caused by probably to_s method
• RangeError 0x3fd07d5c4ee0 is recycled object

emerge --sync error

Reference

• http://forums.funtoo.org/viewtopic.php?id=644
• http://www.gentooforum.de/artikel/19966/kdebase-startkde-div-probleme.html

My make.conf

• Attach:make.conf.20110907.txt

Update importer migel (products)

Log

...
569 / 571       Estimate total: 11.60 [h] It will be done in: 2.44 [m]
570 / 571       Estimate total: 11.58 [h] It will be done in: 1.22 [m]
571 / 571       Estimate total: 11.58 [h] It will be done in: -0.00 [m]

Run

• bin/migeld
• bin/admin

migel> Migel::Importer.new.import_all_products_from_csv('migel_product_de.csv', 'de', true)
migel> Migel::Importer.new.import_all_products_from_csv('migel_product_fr.csv', 'fr', true)

Log

...
11606 / 30742   Estimate total: 1.37 [h] It will be done in: 51.22 [m]
11607 / 30742   Estimate total: 1.37 [h] It will be done in: 51.31 [m]
11608 / 30742   Estimate total: 1.38 [h] It will be done in: 51.38 [m]
...

Result

• Error

21986 / 30742   Estimate total: 4.03 [h] It will be done in: 1.15 [h]
21987 / 30742   Estimate total: 4.03 [h] It will be done in: 1.15 [h]
/usr/lib64/ruby/site_ruby/1.8/odba/cache_entry.rb:42: [BUG] Segmentation fault
ruby 1.8.6 (2009-06-08) [x86_64-linux]

Abgebrochen

...
30741 / 30742   Estimate total: 8.90 [m] It will be done in: 0.00 [m]
30742 / 30742   Estimate total: 8.90 [m] It will be done in: -0.00 [m]

Note

• But there is only one error.

Setup index tables

• bin/admin

migel> init_migelids
migel> init_products
migel> rebuild_fulltext_index_tables

Attach:import_produt.20110907.jpg Δ

Update a testcase of spreadsheet

Email

• test for spreadsheet

Reference

• https://github.com/dblock/spreadsheet/commit/164dcfbb24097728f1a7453702c270107e725b7c

Task

• Pass the following testase failure by using flexmock, not necessary to update the spreadsheet code

masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby test/integration.rb 
Loaded suite test/integration
Started
E
Finished in 0.008582 seconds.

  1) Error:
test_write_frozen_string(Spreadsheet::TestIntegration):
Iconv::InvalidCharacter: "g"
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/encodings.rb:32:in `iconv'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/encodings.rb:32:in `internal'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/biff8.rb:43:in `_unicode_string'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:508:in `_write_sst'
    /usr/lib64/ruby/1.8/fileutils.rb:243:in `each_with_index'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:502:in `each'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:502:in `each_with_index'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:502:in `_write_sst'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:489:in `write_sst'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:439:in `write_from_scratch'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/excel/writer/workbook.rb:620:in `write_workbook'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/writer.rb:15:in `write'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/writer.rb:14:in `open'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/writer.rb:14:in `write'
    /home/masa/work/dblock-spreadsheet-164dcfb/lib/spreadsheet/workbook.rb:106:in `write'
    test/integration.rb:1309:in `test_write_frozen_string'

1 tests, 0 assertions, 0 failures, 1 errors

Note

• Ruby 1.9 passses this testcase
• Ruby 1.8 fails

Experiment

• test.rb

require 'iconv'
string = 'Frozen String'
client = 'UTF-16LE'
iconv = Iconv.new('UTF-16LE', client)
p iconv.iconv(string)

Result

masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby test.rb 
test.rb:6:in `iconv': "g" (Iconv::InvalidCharacter)
        from test.rb:6

Note

• It is the same error

Experiment

 require 'iconv'
 string = 'Frozen String.'
 iconv = Iconv.new('UTF-16LE', 'UTF-16LE')
 p iconv.iconv(string)

Run

masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby test.rb 
"Frozen String."

Note

• It works but I do not know why

Experiment

• test/integration.rb

    def test_write_frozen_string
      Spreadsheet.client_encoding = 'UTF-16LE'
      book = Spreadsheet::Workbook.new
      path = File.join @var, 'test_write_workbook.xls'
      sheet1 = book.create_worksheet
      str1 = "Frozen String.".freeze
      sheet1[0,0] = str1
      sheet1.row(0).push str1
      book.write path
    end

Run

masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby -I lib test/integration.rb 
Loaded suite test/integration
Started
E
Finished in 0.008138 seconds.

  1) Error:
test_write_frozen_string(Spreadsheet::TestIntegration):
TypeError: can't modify frozen string
    ./lib/spreadsheet/excel/writer/biff8.rb:66:in `<<'
    ./lib/spreadsheet/excel/writer/biff8.rb:66:in `_unicode_string'
    ./lib/spreadsheet/excel/writer/workbook.rb:508:in `_write_sst'
    /usr/lib64/ruby/1.8/fileutils.rb:243:in `each_with_index'
    ./lib/spreadsheet/excel/writer/workbook.rb:502:in `each'
    ./lib/spreadsheet/excel/writer/workbook.rb:502:in `each_with_index'
    ./lib/spreadsheet/excel/writer/workbook.rb:502:in `_write_sst'
    ./lib/spreadsheet/excel/writer/workbook.rb:489:in `write_sst'
    ./lib/spreadsheet/excel/writer/workbook.rb:439:in `write_from_scratch'
    ./lib/spreadsheet/excel/writer/workbook.rb:620:in `write_workbook'
    ./lib/spreadsheet/writer.rb:15:in `write'
    ./lib/spreadsheet/writer.rb:14:in `open'
    ./lib/spreadsheet/writer.rb:14:in `write'
    ./lib/spreadsheet/workbook.rb:106:in `write'
    test/integration.rb:1309:in `test_write_frozen_string'

1 tests, 0 assertions, 0 failures, 1 errors

Note

• There is no assertion in the testcase
• The testcase should be as follows

    def test_write_frozen_string
      Spreadsheet.client_encoding = 'UTF-16LE'
      book = Spreadsheet::Workbook.new
      path = File.join @var, 'test_write_workbook.xls'
      sheet1 = book.create_worksheet
      str1 = "Frozen String.".freeze
      sheet1[0,0] = str1
      sheet1.row(0).push str1
      assert_nothing_raised do
        book.write path
      end
    end

Run

masa@masa ~/work/dblock-spreadsheet-164dcfb/test $ ruby18 -rubygems -I ../lib integration.rb 
Loaded suite integration
Started
F
Finished in 0.010687 seconds.

  1) Failure:
test_write_frozen_string(Spreadsheet::TestIntegration) [integration.rb:1309]:
Exception raised:
Class: <TypeError>
Message: <"can't modify frozen string">
---Backtrace---
../lib/spreadsheet/excel/writer/biff8.rb:66:in `<<'
../lib/spreadsheet/excel/writer/biff8.rb:66:in `_unicode_string'
../lib/spreadsheet/excel/writer/workbook.rb:508:in `_write_sst'
/usr/lib64/ruby/1.8/fileutils.rb:243:in `each_with_index'
../lib/spreadsheet/excel/writer/workbook.rb:502:in `each'
../lib/spreadsheet/excel/writer/workbook.rb:502:in `each_with_index'
../lib/spreadsheet/excel/writer/workbook.rb:502:in `_write_sst'
../lib/spreadsheet/excel/writer/workbook.rb:489:in `write_sst'
../lib/spreadsheet/excel/writer/workbook.rb:439:in `write_from_scratch'
../lib/spreadsheet/excel/writer/workbook.rb:620:in `write_workbook'
../lib/spreadsheet/writer.rb:15:in `write'
../lib/spreadsheet/writer.rb:14:in `open'
../lib/spreadsheet/writer.rb:14:in `write'
../lib/spreadsheet/workbook.rb:106:in `write'
integration.rb:1310:in `test_write_frozen_string'
integration.rb:1309:in `test_write_frozen_string'
---------------

1 tests, 1 assertions, 1 failures, 0 errors

Experiment

• lib/spreadsheet/encoding.rb module Encodings

    if RUBY_VERSION >= '1.9'

...

    else
      require 'iconv'
      @@iconvs = {}
      def client string, internal='UTF-16LE'
        string = string.dup
        key = [Spreadsheet.client_encoding, internal]
        iconv = @@iconvs[key] ||= Iconv.new(Spreadsheet.client_encoding, internal)
        iconv.iconv string
      end
      def internal string, client=Spreadsheet.client_encoding
        string = string.dup
        key = ['UTF-16LE', client]
        iconv = @@iconvs[key] ||= Iconv.new('UTF-16LE', client)
        iconv.iconv string
      end
      def utf8 string, client=Spreadsheet.client_encoding
        string = string.dup
        key = ['UTF-8', client]
        iconv = @@iconvs[key] ||= Iconv.new('UTF-8', client)
        iconv.iconv string
      end
    end

Run

masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby -rubygems -I lib test/integration.rb 
Loaded suite test/integration
Started
..........................
Finished in 1.162569 seconds.

26 tests, 807 assertions, 0 failures, 0 errors
masa@masa ~/work/dblock-spreadsheet-164dcfb $ ruby1.9 -rubygems -I lib test/integration.rb 
Loaded suite test/integration
Started
..........................

Finished in 0.83873682 seconds.

26 tests, 807 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

Patch

• Attach:spreadsheet_20110907.patch.txt

Commit

• Fixed a frozen string bug thanks to dblock (Daniel Doubrovkine (dB.))

Testcases migel

1. util
   1. swissindex.rb (coverage: 100%)
   2. server.rb (coverage: 88.41%)
   3. multilingual.rb pass because it is similar to the multilingual.rb of de.oddb.org

Reference

• http://www.ultrasaurus.com/sarahblog/2011/04/repl-rspec-mocks/ (how to require rpsec depending on the version)
• http://blog.thefrontiergroup.com.au/2011/05/rspec-2-6-0-and-rcov/ (rspec2.6 + rcov) rake file is not necessary anymore for rcov