dev.ywesee.com - Choddb/MiGelChangelog

view · edit · sidebar · attach · print · history

Update log migel on Ruby 1.9.3

09.01.2012 export migel

Problem

The number of lines between migel_product_de.csv and swissINDEX_MiGel.csv are quite different

Note

swissINDEX_MiGel.csv comes from Updater.new(self).update_migel_nonpharma (oddb.org)
migel_product_de.csv comes from Importer.new.save_all_products (migel)

Commit

06.01.2012 update method

Error (update, migelde/fr.csv)

 migel> Migel::Util::Importer.new.update('/home/masa/ywesee/oddb.org/data/csv/migel_de.csv','de')
 -> undefined method `de=' for nil:NilClass

"/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `instance_eval'",
 "/home/masa/ywesee/migel/lib/migel/model/migelid.rb:40:in `each'",
 "/home/masa/ywesee/migel/lib/migel/model/migelid.rb:40:in `update_multilingual'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:148:in `update_migelid'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:67:in `block in update'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:54:in `each'",
 "/home/masa/ywesee/migel/lib/migel/util/importer.rb:54:in `update'",
 "(eval):1:in `block in _admin'",
 "/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `instance_eval'",
 "/home/masa/ywesee/migel/lib/migel/util/server.rb:18:in `block in _admin'"

Commit

Fix undefined method error, undefined method 'de=' for nil:NilClass, in Importer#update method (migel)

15.12.2011 Date problem

Problem

masa@masa ~/work/migel $ ruby193 bin/admin 
migel> migelids.values.first.date
-> -4712-01-01

Note

This is the result of miss-converting of data in migrate_utf8

Commit

Updated migelid dates after migrate_utf8 for Ruby 1.9.3 (migel)

Procedure

Get migel code and date from the old database on Ruby 1.8.6

 migel> output_migel_code_and_date

copy migelcode_date.dat to the new migel directory for Ruby 1.9.3
Update dates

 migel> update_migelid_date

Save database

 pg_dump -U postgres migel.ruby193 > migel.ruby193.bak.20111215_after_update_date.sql
 gzip migel.ruby193.bak.20111215_after_update_date.sql

Send it online server and restore it

 # sudo -u postgres dropdb migel.ruby193; sudo -u postgres createdb -E UTF8 -T template0 migel.ruby193
 $ zcat migel.ruby193.bak.20111215_after_update_date.sql.gz | psql -U postgres migel.ruby193

missing pointer link

Commit

Fix pointer/pointer in migel product view (migel)

08.12.2011 Debug migel importer

Error (access http://oddb.masa.org/de/gcc/migel_search/migel_product/349001012)

error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
NoMethodError
undefined method `[]' for #<ODBA::FulltextIndex:0x000000218df948>
(druby://localhost:33000) /home/masa/work/migel/lib/migel/model_super.rb:166:in `data_origin'
(druby://localhost:33000) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/drbwrapper.rb:28:in `method_missing'
(druby://localhost:33000) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/namedcomponent.rb:43:in `data_origin'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/label.rb:57:in `initialize'

Delete the mgelid (34.90.01.01.2) and import again

migel> Migel::Model::Migelid.search_by_migel_code('34.90.01.01.2').first.delete

migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_de.csv','de')
migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_fr.csv','fr')

  (skip saving migel_product_de/fr.csv files, use the previous files)

migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_de.csv', 'de', true)
migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_fr.csv', 'fr', true)
migel> init_fulltext_index_tables

Result

The error is solved

07.12.2011 Check migel importer

Import process

1. Prepare migel_de.csv and migel_fr.csv
   Refer to http://dev.ywesee.com/wiki.php/Choddb/MiGelChangelog

2. update group, subgroup, migelid from a csv file
  migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_de.csv','de')
  migel> Migel::Util::Importer.new.update('/var/www/migel/data/csv/migel_fr.csv','fr')

3. save all products data into a csv file
  migel> Migel::Util::Importer.new.save_all_products('migel_product_de.csv', 'de', true)
  migel> Migel::Util::Importer.new.save_all_products('migel_product_fr.csv', 'fr', true)
  (this process takes long time (several hours))

4. update data from the csv file
  migel> Migel::Util::Importer.new.import_all_products_from_csv('migel_product_de.csv', 'de', true)
  migel> Migel::Util::Importer.new.import_all_products_from_csv('migel_product_fr.csv', 'fr', true)

5. Rebuild index tables
  migel> init_fulltext_index_tables

Reference

Run importer (save products csv file)

ruby18 oddb.org/ext/swissindex/bin/swissindex_nonpharmad (=> swissindex_nonpharmad must run on Ruby 1.9.3 otherwise the date information will be broken)
ruby193 bin/migeld
bin/admin

migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)

Error

migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)
-> instance of DateTime needs to have method `marshal_load'

 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:585:in `load'",
 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:581:in `exclusive'",
 "(druby://localhost:50002) /usr/lib64/ruby/1.8/drb/drb.rb:581:in `load'",
 "/home/masa/work/migel/lib/migel/util/swissindex.rb:22:in `search_migel_table'",
 "/home/masa/work/migel/lib/migel/util/importer.rb:220:in `get_products_by_migel_code'",

Refer to importer.rb.patch

Error

migel> Migel::Util::Importer.new.save_all_products('migel_products_de.csv', 'de', true)
-> divided by 0

"/home/masa/work/migel/lib/migel/util/server.rb:20:in `instance_eval'",
 "/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb:35:in `Rational'",
 "/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb:35:in `_load'",
 "/usr/local/lib/ruby/1.9.1/drb/drb.rb:585:in `load'",
 "/usr/local/lib/ruby/1.9.1/drb/drb.rb:585:in `block in load'",
 "<internal:prelude>:10:in `synchronize'",

Refer to 18_19_loading_compatibility.rb.patch

Error (Date data)

migel_products_de.csv

 01.01.01.00.1,1624501,7612479001117,AMEDA Einhandmilchpumpe mit Flexishield,Nufer AG (Medizintechnik),,32.30,43.70,1,43.70,,A,-4712-01-01T12:00:00+00:00,-4712-01-01T12:00:00+00:00,DE
 01.01.01.00.1,4443763,7640109051744,ARDO AMARYLL Individuelle Handmilchpumpe,Ardo medical AG,,41.30,74.00,1,74.00,,A,-4712-01-01T12:00:00+00:00,-4712-01-01T12:00:00+00:00,DE
 ...

Refer to swissindex.rb.patch

oddb.org/ext/swissindex/src/swissindex.rb is copied under migel/lib/migel/ext/

IMPORTANT

oddb.org/ext/swissindex/bin/swissindex_nonpharmad must run on Ruby 1.9.3
since 18_19_loading_compatibility.rb does not work on the data transfer from Ruby 1.9 to Ruby 1.8 through druby

Error

migel> Migel::Util::Importer.new.update('data/csv/migel_de.csv','de')
-> wrong number of arguments (1 for 0)

["/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:41:in `block in update_multilingual'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:40:in `each'",
 "/home/masa/work/migel/lib/migel/model/migelid.rb:40:in `update_multilingual'",

Refer to migelid.rb.patch

Error (import_all_products_from_csv)

migel> Migel::Util::Importer.new.import_all_products_from_csv('data/csv/migel_product_de.csv', 'de', true)
-> undefined method `[]' for #<CSV:0x0000000ba16ef8>

["/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/usr/local/lib/ruby/1.9.1/csv.rb:1367:in `open'",
 "/home/masa/work/migel/lib/migel/util/importer.rb:345:in `import_all_products_from_csv'",
 "(eval):1:in `block in _admin'",
 "/home/masa/work/migel/lib/migel/util/server.rb:19:in `instance_eval'",
 "/home/masa/work/migel/lib/migel/util/server.rb:19:in `block in _admin'"]

Refer to importer.rb.patch

02.12.2011 Update migel for Ruby 1.9.3

Error (Product view, example http://oddb.masa.org/de/gcc/migel_search/migel_product/349001012)

error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
Encoding::CompatibilityError
incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
/home/masa/work/oddb.org_1/src/view/migel/product.rb:83:in `gsub'
/home/masa/work/oddb.org_1/src/view/migel/product.rb:83:in `description'
/home/masa/work/oddb.org_1/src/view/migel/product.rb:116:in `product_text'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:66:in `create'

error in SBSM::Session#to_html: /de/gcc/migel_search/migel_product/349001012
Encoding::CompatibilityError
incompatible character encodings: ASCII-8BIT and UTF-8
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:112:in `block in template_html'
/usr/local/lib/ruby/1.9.1/cgi/html.rb:433:in `block in html'
(eval):1069:in `html'

Commit

Added migrate_utf8 for force_encoding every String data into utf-8 for Ruby 1.9. (migel)

Error (Migel 0-9)

error in SBSM::Session#to_html: /de/gcc/migel_alphabetical/range/0-9
NoMethodError
undefined method `each_char' for [" ", "\"", "1", "7", "�"]:Array
/home/masa/work/oddb.org_1/src/util/interval.rb:95:in `load_model'
/home/masa/work/oddb.org_1/src/util/interval.rb:104:in `filter'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:242:in `call'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:242:in `view'
/usr/local/lib/ruby/site_ruby/1.9.1/sbsm/state.rb:179:in `to_html'

Refer to interval.rb.patch

Error (Migel A-Z)

error in SBSM::Session#to_html: /de/gcc/migel_alphabetical/
Encoding::CompatibilityError
incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/iso-latin1.rb:52:in `match'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/iso-latin1.rb:52:in `downcase'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/interval.rb:83:in `comparison_value'
(druby://localhost:10000) /home/masa/work/oddb.org/src/util/interval.rb:97:in `block (2 levels) in load_model'

Refer to iso-latin1.rb.patch

Error (search_migel_migelid)

masa@masa ~/work/migel $ ruby193 bin/admin
bin/admin:9: warning: variable $KCODE is no longer effective; ignored
migel> search_migel_migelid('Wartungskosten','de')
-> class Date needs to have method `_load'
migel> exit
-> Goodbye

Refer to server.rb.patch

Error (bin/admin)

masa@masa ~/work/migel $ ruby193 bin/admin 
bin/admin:9: warning: variable $KCODE is no longer effective; ignored
/usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- encoding/character/utf-8 (LoadError)
        from /usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from bin/admin:15:in `<main>'

Refer to admin.patch

Error (boot migel)

masa@masa ~/work/migel $ ruby193 bin/migeld
bin/migeld:6: warning: variable $KCODE is no longer effective; ignored
/usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:300:in `block in load_driver': Unable to load driver 'pg' (underlying error: wrong constant name pg) (DBI::InterfaceError)
        from /usr/local/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:242:in `load_driver'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:160:in `_get_full_driver'
        from /usr/local/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi.rb:145:in `connect'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/connection_pool.rb:60:in `block in _connect'

Refer to odba.rb.patch

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

Update log migel on Ruby 1.9.3

09.01.2012 export migel

06.01.2012 update method

15.12.2011 Date problem

missing pointer link

08.12.2011 Debug migel importer

07.12.2011 Check migel importer

02.12.2011 Update migel for Ruby 1.9.3

Patches

Scripts

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

Update log migel on Ruby 1.9.3

09.01.2012 export migel

06.01.2012 update method

15.12.2011 Date problem

missing pointer link

08.12.2011 Debug migel importer

07.12.2011 Check migel importer

02.12.2011 Update migel for Ruby 1.9.3

Patches

Scripts

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler