view · edit · sidebar · attach · print · history

Update log migel on Ruby 1.9.3

09.01.2012 export migel

Problem

  • The number of lines between migel_product_de.csv and swissINDEX_MiGel.csv are quite different

Note

  • swissINDEX_MiGel.csv comes from Updater.new(self).update_migel_nonpharma (oddb.org)
  • migel_product_de.csv comes from Importer.new.save_all_products (migel)

Commit

06.01.2012 update method

Error (update, migelde/fr.csv)

 migel> Migel::Util::Importer.new.update('/home/masa/ywesee/oddb.org/data/csv/migel_de.csv','de')
 -> undefined method `de=' for nil:NilClass
"/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `instance_eval'",
 "/home/masa/ywesee/migel/lib/migel/model/migelid.rb:40:in `each'",
 "/home/masa/ywesee/migel/lib/migel/model/migelid.rb:40:in `update_multilingual'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:148:in `update_migelid'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:67:in `block in update'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:54:in `each'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:54:in `update'",
 "(eval):1:in `block in _admin'",
 "/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `instance_eval'",
 "/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `block in _admin'"

Commit

15.12.2011 Date problem

Problem

masa@masa ~/work/migel $ ruby193 bin/admin 
migel> migelids.values.first.date
-> -4712-01-01

Note

  • This is the result of miss-converting of data in migrate_utf8

Commit

Procedure

  • Get migel code and date from the old database on Ruby 1.8.6
 migel> output_migel_code_and_date
  • copy migelcode_date.dat to the new migel directory for Ruby 1.9.3
  • Update dates
 migel> update_migelid_date
  • Save database
 pg_dump -U postgres migel.ruby193 > migel.ruby193.bak.20111215_after_update_date.sql
 gzip migel.ruby193.bak.20111215_after_update_date.sql
  • Send it online server and restore it
 # sudo -u postgres dropdb migel.ruby193; sudo -u postgres createdb -E UTF8 -T template0 migel.ruby193
 $ zcat migel.ruby193.bak.20111215_after_update_date.sql.gz | psql -U postgres migel.ruby193

missing pointer link

Commit

08.12.2011 Debug migel importer

Error (access http://oddb.masa.org/de/gcc/migel_search/migel_product/349001012)

error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
NoMethodError
undefined method `[]' for #<ODBA::FulltextIndex:0x000000218df948>
(druby://localhost:33000) /home/masa/work/migel/lib/migel/model_super.rb:166:in `data_origin'
(druby://localhost:33000) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/drbwrapper.rb:28:in `method_missing'
(druby://localhost:33000) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/namedcomponent.rb:43:in `data_origin'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/label.rb:57:in `initialize'

Delete the mgelid (34.90.01.01.2) and import again

migel> Migel::Model::Migelid.search_by_migel_code('34.90.01.01.2').first.delete
migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_de.csv','de')
migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_fr.csv','fr')

  (skip saving migel_product_de/fr.csv files, use the previous files)

migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_de.csv', 'de', true)
migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_fr.csv', 'fr', true)
migel> init_fulltext_index_tables

Result

  • The error is solved

07.12.2011 Check migel importer

Import process

1. Prepare migel_de.csv and migel_fr.csv
   Refer to http://dev.ywesee.com/wiki.php/Choddb/MiGelChangelog

2. update group, subgroup, migelid from a csv file
  migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_de.csv','de')
  migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_fr.csv','fr')

3. save all products data into a csv file
  migel> Migel::Util::Importer.new.save_all_products('migel_product_de.csv', 'de', true)
  migel> Migel::Util::Importer.new.save_all_products('migel_product_fr.csv', 'fr', true)
  (this process takes long time (several hours))

4. update data from the csv file
  migel> Migel::Util::Importer.new.import_all_products_from_csv('migel_product_de.csv', 'de', true)
  migel> Migel::Util::Importer.new.import_all_products_from_csv('migel_product_fr.csv', 'fr', true)

5. Rebuild index tables
  migel> init_fulltext_index_tables

Reference

Run importer (save products csv file)

  • ruby18 oddb.org/ext/swissindex/bin/swissindex_nonpharmad (=> swissindex_nonpharmad must run on Ruby 1.9.3 otherwise the date information will be broken)
  • ruby193 bin/migeld
  • bin/admin
migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)

Error

migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)
-> instance of DateTime needs to have method `marshal_load'
 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:585:in `load'",
 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:581:in `exclusive'",
 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:581:in `load'",
 "/home/masa/work/migel/lib/migel/util/swissindex.rb:22:in `search_migel_table'",
 "/home/masa/work/migel/lib/migel/util/importer.rb:220:in `get_products_by_migel_code'",

Refer to importer.rb.patch

Error

migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)
-> divided by 0
"/home/masa/work/migel/lib/migel/util/server.rb:20:in `instance_eval'",
 "/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb:35:in `Rational'",
 "/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb:35:in `_load'",
 "/usr/local/lib/ruby/1.9.1/drb/drb.rb:585:in `load'",
 "/usr/local/lib/ruby/1.9.1/drb/drb.rb:585:in `block in load'",
 "<internal:prelude>:10:in `synchronize'",

Refer to 18_19_loading_compatibility.rb.patch

Error (Date data)

  • migel_products_de.csv
 01.01.01.00.1,1624501,7612479001117,AMEDA Einhandmilchpumpe mit Flexishield,Nufer AG (Medizintechnik),,32.30,43.70,1,43.70,,A,-4712-01-01T12:00:00+00:00,-4712-01-01T12:00:00+00:00,DE
 01.01.01.00.1,4443763,7640109051744,ARDO AMARYLL Individuelle Handmilchpumpe,Ardo medical AG,,41.30,74.00,1,74.00,,A,-4712-01-01T12:00:00+00:00,-4712-01-01T12:00:00+00:00,DE
 ...

Refer to swissindex.rb.patch

  • oddb.org/ext/swissindex/src/swissindex.rb is copied under migel/lib/migel/ext/

IMPORTANT

  • oddb.org/ext/swissindex/bin/swissindex_nonpharmad must run on Ruby 1.9.3
  • since 18_19_loading_compatibility.rb does not work on the data transfer from Ruby 1.9 to Ruby 1.8 through druby

Error

migel> Migel::Util::Importer.new.update('data/csv/migel_de.csv','de')
-> wrong number of arguments (1 for 0)
["/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:41:in `block in update_multilingual'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:40:in `each'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:40:in `update_multilingual'",

Refer to migelid.rb.patch

Error (import_all_products_from_csv)

migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_de.csv', 'de', true)
-> undefined method `[]' for #<CSV:0x0000000ba16ef8>
["/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/usr/local/lib/ruby/1.9.1/csv.rb:1367:in `open'",
 "/home/masa/work/migel/lib/migel/util/importer.rb:345:in `import_all_products_from_csv'",
 "(eval):1:in `block in _admin'",
 "/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/home/masa/work/migel/lib/migel/util/server.rb:19:in `block in _admin'"]

Refer to importer.rb.patch

02.12.2011 Update migel for Ruby 1.9.3

Error (Product view, example http://oddb.masa.org/de/gcc/migel_search/migel_product/349001012)

error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
Encoding::CompatibilityError
incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
/home/masa/work/oddb.org_1/src/view/migel/product.rb:83:in `gsub'
/home/masa/work/oddb.org_1/src/view/migel/product.rb:83:in `description'
/home/masa/work/oddb.org_1/src/view/migel/product.rb:116:in `product_text'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:66:in `create'
error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
Encoding::CompatibilityError
incompatible character encodings: ASCII-8BIT and UTF-8
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:112:in `block in template_html'
/usr/local/lib/ruby/1.9.1/cgi/html.rb:433:in `block in html'
(eval):1069:in `html'

Commit

Error (Migel 0-9)

error in SBSM::Session#to_html: /de/gcc/migel_alphabetical/range/0-9
NoMethodError
undefined method `each_char' for [" ", "\"", "1", "7", ""]:Array
/home/masa/work/oddb.org_1/src/util/interval.rb:95:in `load_model'
/home/masa/work/oddb.org_1/src/util/interval.rb:104:in `filter'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:242:in `call'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:242:in `view'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:179:in `to_html'

Refer to interval.rb.patch

Error (Migel A-Z)

error in SBSM::Session#to_html: /de/gcc/migel_alphabetical/
Encoding::CompatibilityError
incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/iso-latin1.rb:52:in `match'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/iso-latin1.rb:52:in `downcase'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/interval.rb:83:in `comparison_value'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/interval.rb:97:in `block (2 levels) in load_model'

Refer to iso-latin1.rb.patch

Error (search_migel_migelid)

masa@masa ~/work/migel $ ruby193 bin/admin
bin/admin:9: warning: variable $KCODE is no longer effective; ignored
migel> search_migel_migelid('Wartungskosten','de')
-> class Date needs to have method `_load'
migel> exit
-> Goodbye

Refer to server.rb.patch

Error (bin/admin)

masa@masa ~/work/migel $ ruby193 bin/admin 
bin/admin:9: warning: variable $KCODE is no longer effective; ignored
/usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- encoding/character/utf-8 (LoadError)
        from /usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from bin/admin:15:in `<main>'

Refer to admin.patch

Error (boot migel)

masa@masa ~/work/migel $ ruby193 bin/migeld
bin/migeld:6: warning: variable $KCODE is no longer effective; ignored
/usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:300:in `block in load_driver': Unable to load driver 'pg' (underlying error: wrong constant name pg) (DBI::InterfaceError)
        from /usr/local/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:242:in `load_driver'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:160:in `_get_full_driver'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:145:in `connect'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/connection_pool.rb:60:in `block in _connect'

Refer to odba.rb.patch

Patches

  1. Attach:odba.rb.patch.20111202.txt
  2. Attach:admin.patch.20111202.txt
  3. Attach:server.rb.patch.20111202.txt
  4. Attach:importer.rb.patch.20111207.txt
  5. Attach:18_19_loading_compatibility.rb.patch.20111207.txt
  6. Attach:swissindex.rb.patch.20111207.txt
  7. Attach:migelid.rb.patch.20111207.txt

Scripts

  1. Attach:migel_patches.sh.20111207.txt
  2. Attach:oddb_patches.sh.20111207.txt
view · edit · sidebar · attach · print · history
Page last modified on January 09, 2012, at 10:03 AM