view · edit · sidebar · attach · print · history

Index>

20140618-ruby-21x

Summary

  • Conversion to ruby 2.1.x
  • Make migel run unit tests under Ruby 1.9.3

Commits

oddb.org branch ruby_21x

migel

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • One unit-test for searchbar fails and might be a clue why searching does not work correctly.
  • Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?
  • Make ext/swissindex/test/test_swissindex.rb and skipping tests in migel-gem pass.

---

Conversion to ruby 2.1.x

Trying to start oddb.org with Ruby 2.1.x.

  • Must correct Apache to use mod_ruby with 2.1
  • Analyse the problem, why I changed line 208 /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb to if RUBY_VERSION < "1.9" && (cuki = request.cookies[self::class::PERSISTENT_COOKIE_NAME]). Maybe I should better check whether the classs is a DRb::DRbUnknown.
  • Changing RubyAddPath /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib in /etc/apache2/vhosts.d/00_default_ssl_vhost.conf and /etc/apache2/vhosts.d/oddb.conf to /usr/local/lib64/ruby/gems/2.1.0/gems/sbsm-1.2.5/lib after saving copy of the conf files as *.conf.ruby_19x

After the following changes

  • adding commit Added monkey patching CSV::Cell
  • adding a new line 208 return if request.cookies.is_a?(DRb::DRbUnknown) to /usr/local/lib/ruby/gems/2.1.0/gems/sbsm-1.2.5/lib/sbsm/session.rb
  • Adding a line return if object.class == ODDB::App in method odba_add_reference(object) of def odba_add_reference(object) (/usr/local/lib64/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache_entry.rb)

bin/oddbd started without problem and a short manual test dit not find any problems. Could log in via yus (still under 1.8.7). Migel does not work (as not yet ported to 2.1.x).

After adding commit Avoid problems when log directory or pifile not present I am able to start import_daily. While import is running I will start porting migel to 2.1.x.

Running jobs/export_csv results in 

2014-06-23 14:55:08 +0200: safe_export oddb.csv starting
/usr/share/applications/kde4/kate.desktop 2014-06-23 15:12:20 +0200: safe_export oddb.csv completed
2014-06-23 15:12:20 +0200: safe_export oddb2.csv starting
2014-06-23 15:14:20 +0200: safe_export oddb2.csv completed
ls -l data/*/*csv
-rw-r--r-- 1 apache apache 5892545 23. Jun 15:12 data/csv/oddb.2014-06-01.csv
-rw-r--r-- 1 apache apache 6241817 23. Jun 15:14 data/csv/oddb2.2014-06-01.csv
-rw-r--r-- 1 apache apache 5892545 23. Jun 15:11 data/downloads/oddb.csv
-rw-r--r-- 1 apache apache 6241817 23. Jun 15:14 data/downloads/oddb2.csv
niklaus@oddb-ci2 /v/w/oddb.org> diff ./data/downloads/oddb2.csv /var/www/oddb.org.ruby193/data/downloads/
niklaus@oddb-ci2 /v/w/oddb.org> diff ./data/downloads/oddb.csv /var/www/oddb.org.ruby193/data/downloads/
<.. now with 2.1.2>
/usr/local/bin/ruby jobs/export_csv
2014-06-23 15:18:49 +0200: safe_export oddb.csv starting
2014-06-23 15:27:31 +0200: safe_export oddb.csv completed
2014-06-23 15:27:31 +0200: safe_export oddb2.csv starting
2014-06-23 15:28:01 +0200: safe_export oddb2.csv completed
niklaus@oddb-ci2 /v/w/oddb.org> diff ./data/downloads/oddb.csv /var/www/oddb.org.ruby193/data/downloads/
niklaus@oddb-ci2 /v/w/oddb.org> diff ./data/downloads/oddb2.csv /var/www/oddb.org.ruby193/data/downloads/
niklaus@oddb-ci2 /v/w/oddb.org> ls -l data/*/*csv
-rw-r--r-- 1 apache apache 5892545 23. Jun 15:27 data/csv/oddb.2014-06-01.csv
-rw-r--r-- 1 apache apache 6241817 23. Jun 15:28 data/csv/oddb2.2014-06-01.csv
-rw-r--r-- 1 apache apache 5892545 23. Jun 15:27 data/downloads/oddb.csv
-rw-r--r-- 1 apache apache 6241817 23. Jun 15:28 data/downloads/oddb2.csv

export_csv seems be okay with 2.1.2 and took only 9 minutes and 13 seconds compared to 19 minutes and 12 seconds. Therefore it as almost twice as fast!

Running export_daily and comparing results with last month. E.g 

2014-05-07 09:00:12 CEST safe_export Mail Patinfo-Invoices starting
2014-05-07 09:15:04 CEST safe_export Mail Patinfo-Invoices completed

2014-06-23 15:37:22 +0200: safe_export Mail Patinfo-Invoices starting
2014-06-23 15:50:40 +0200: safe_export Mail Patinfo-Invoices completed

Patinfo-Invoices took 13 minutes and 18 seconds compared to 14 minutes and 52 seconds. Speed-up is about 10%. Export-daily finished with message 2014-06-23 16:14:05 +0200: restart_export_server. Done sleeping 60 seconds

Kill -9 import_daily and restarted it. Now it seems to go further and memory consumption is presently at 3770 MB, about the same as with Ruby 1.9.3 at this stage of the import.

Running also the following scripts:

  • export_teilbarkeit
  • export_fachinfo_yaml
  • export_fachinfo_chapter

Import_daily finished at 18:15 (after almost 3 hours) with out of memory 

/var/www/oddb.org/src/model/text.rb:395:in `block in wrap': failed to allocate memory (NoMemoryError)
        from /var/www/oddb.org/src/model/text.rb:377:in `each'
        from /var/www/oddb.org/src/model/text.rb:377:in `wrap'
        from /var/www/oddb.org/src/model/text.rb:351:in `block (2 levels) in to_s'
        from /var/www/oddb.org/src/model/text.rb:350:in `collect'
        from /var/www/oddb.org/src/model/text.rb:350:in `block in to_s'
        from /var/www/oddb.org/src/model/text.rb:345:in `collect'
        from /var/www/oddb.org/src/model/text.rb:345:in `to_s'
        from /var/www/oddb.org/src/model/text.rb:431:in `block in to_s'
        from /var/www/oddb.org/src/model/text.rb:431:in `collect'
        from /var/www/oddb.org/src/model/text.rb:431:in `to_s'
        from /var/www/oddb.org/src/model/text.rb:490:in `block in to_s'
        from /var/www/oddb.org/src/model/text.rb:490:in `collect'
        from /var/www/oddb.org/src/model/text.rb:490:in `to_s'
        from /var/www/oddb.org/src/model/fachinfo.rb:116:in `unwanted_effect_text'
        from (eval):3:in `block in proc_resolve_search_term'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:146:in `call'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:146:in `search_term'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:149:in `search_terms'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:174:in `update_origin'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:163:in `update'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:590:in `block in update_indices'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:589:in `each'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:589:in `update_indices'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:515:in `store'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:274:in `odba_isolated_store'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:395:in `block in odba_store_unsaved'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `each'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `odba_store_unsaved'
        from /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:380:in `odba_store'
        from /var/www/oddb.org/src/util/persistence.rb:279:in `issue_update'
        from /var/www/oddb.org/src/util/oddbapp.rb:133:in `block in update'
        from /var/www/oddb.org/src/util/failsafe.rb:10:in `call'
        from /var/www/oddb.org/src/util/failsafe.rb:10:in `failsafe'
        from /var/www/oddb.org/src/util/oddbapp.rb:131:in `update'
        from /var/www/oddb.org/src/util/oddbapp.rb:1575:in `update'
        from /var/www/oddb.org/src/plugin/text_info.rb:144:in `store_fachinfo'
        from /var/www/oddb.org/src/plugin/text_info.rb:188:in `block in update_fachinfo'
        from /var/www/oddb.org/src/plugin/text_info.rb:180:in `each'
        from /var/www/oddb.org/src/plugin/text_info.rb:180:in `update_fachinfo'
        from /var/www/oddb.org/src/plugin/text_info.rb:1065:in `parse_and_update'
        from /var/www/oddb.org/src/plugin/text_info.rb:1317:in `block in import_stuff'
        from /var/www/oddb.org/src/plugin/text_info.rb:1312:in `each'
        from /var/www/oddb.org/src/plugin/text_info.rb:1312:in `import_stuff'
        from /var/www/oddb.org/src/plugin/text_info.rb:1366:in `import_swissmedicinfo_by_index'
        from /var/www/oddb.org/src/plugin/text_info.rb:1434:in `import_swissmedicinfo'
        from /var/www/oddb.org/src/util/updater.rb:525:in `block in update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:501:in `call'
        from /var/www/oddb.org/src/util/updater.rb:501:in `wrap_update'
        from /var/www/oddb.org/src/util/updater.rb:519:in `update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:308:in `update_textinfo_swissmedicinfo'
        from /var/www/oddb.org/src/util/updater.rb:193:in `run'
        from jobs/import_daily:13:in `block in <module:Util>'
        from /var/www/oddb.org/src/util/job.rb:42:in `call'
        from /var/www/oddb.org/src/util/job.rb:42:in `run'
        from jobs/import_daily:12:in `<module:Util>'
        from jobs/import_daily:11:in `<module:ODDB>'
        from jobs/import_daily:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted

Make migel run unit tests under Ruby 1.9.3

When running bundle exec rspec spec/suite.rb for migel I get the following problem 

/opt/src/migel/lib/fixes/singular.rb:7:in `require': cannot load such file -- facet/string/singular (LoadError)
        from /opt/src/migel/lib/fixes/singular.rb:7:in `<top (required)>'
        from /opt/src/migel/lib/migel/model_super.rb:6:in `require'
        from /opt/src/migel/lib/migel/model_super.rb:6:in `<top (required)>'
        from /opt/src/migel/spec/model/group_spec.rb:7:in `require'
        from /opt/src/migel/spec/model/group_spec.rb:7:in `<top (required)>'
        from /opt/src/migel/spec/suite.rb:14:in `require'
        from /opt/src/migel/spec/suite.rb:14:in `block in <top (required)>'
        from /home/niklaus/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/find.rb:41:in `block in find'
        from /home/niklaus/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/find.rb:40:in `catch'
        from /home/niklaus/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/1.9.1/find.rb:40:in `find'
        from /opt/src/migel/spec/suite.rb:12:in `<top (required)>'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `each'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/configuration.rb:896:in `load_spec_files'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:22:in `run'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:80:in `run'
        from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:17:in `block in autorun'

Fixing facets to use version 1.8.54 fixed the problem. See commit Made unit-tests pass unter ruby 1.9.3 again.

Running under Ruby 2.1.2 we fail like 

/home/niklaus/.rvm/gems/ruby-2.1.2/gems/rmail-1.0.0/lib/rmail.rb:41:in `require': /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rmail-1.0.0/lib/rmail/address.rb:694: invalid multibyte escape: /\A[\200-\377\w!$%&\'*+\/=?^_\`{\}|~#-]+/m (SyntaxError)
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rmail-1.0.0/lib/rmail.rb:41:in `<top (required)>'
        from /opt/src/migel/lib/migel/util/mail.rb:5:in `require'
        from /opt/src/migel/lib/migel/util/mail.rb:5:in `<top (required)>'
        from /opt/src/migel/lib/migel/util/importer.rb:9:in `require'
        from /opt/src/migel/lib/migel/util/importer.rb:9:in `<top (required)>'
        from /opt/src/migel/spec/util/importer_spec.rb:11:in `require'
        from /opt/src/migel/spec/util/importer_spec.rb:11:in `<top (required)>'
        from /opt/src/migel/spec/suite.rb:14:in `require'
        from /opt/src/migel/spec/suite.rb:14:in `block in <top (required)>'
        from /home/niklaus/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/find.rb:47:in `block (2 levels) in find'
        from /home/niklaus/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/find.rb:46:in `catch'
        from /home/niklaus/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/find.rb:46:in `block in find'
        from /home/niklaus/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/find.rb:42:in `each'
        from /home/niklaus/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/find.rb:42:in `find'
        from /opt/src/migel/spec/suite.rb:12:in `<top (required)>'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/configuration.rb:1057:in `load'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/configuration.rb:1057:in `block in load_spec_files'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/configuration.rb:1057:in `each'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/configuration.rb:1057:in `load_spec_files'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/runner.rb:97:in `setup'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/runner.rb:85:in `run'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/runner.rb:70:in `run'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/lib/rspec/core/runner.rb:38:in `invoke'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/gems/rspec-core-3.0.2/exe/rspec:4:in `<top (required)>'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/bin/rspec:23:in `load'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/bin/rspec:23:in `<main>'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `eval'
        from /home/niklaus/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `<main>'

Conclusion we want to eliminate rmail and use only mail (maybe also in a more recent version).

But first we must cleanup the 19 errors under Ruby 1.9 for which I opened the branch unittests. 

rspec ./spec/model/migelid_spec.rb:107 # Migel::Model::Migelid when limitation_text is updated limitation_text.de should be 'limitation text'
rspec ./spec/model/migelid_spec.rb:113 # Migel::Model::Migelid when limitation_text is updated limitation_text.fr should be nil
rspec ./spec/model/migelid_spec.rb:116 # Migel::Model::Migelid when limitation_text is updated limitation_text.parent should be migelid
rspec ./spec/util/importer_spec.rb:108 # Migel::Util::Importer Examples update_migelid should update a Migelid instance when it is found in the database
rspec ./spec/util/importer_spec.rb:135 # Migel::Util::Importer Examples update_migelid should create a Migelid instance when it is not found in the database
rspec ./spec/util/importer_spec.rb:243 # Migel::Util::Importer Examples get_products_by_migel_code should search product data from online migel server
rspec ./spec/util/importer_spec.rb:280 # Migel::Util::Importer Examples save_all_products should save all the products data into a file
rspec ./spec/util/importer_spec.rb:410 # Migel::Util::Importer Examples reimport_missing_data should update products
rspec ./spec/util/importer_spec.rb:204 # Migel::Util::Importer Examples update method updates group, subgroup, and migelid instances normal update
rspec ./spec/util/importer_spec.rb:221 # Migel::Util::Importer Examples update method updates group, subgroup, and migelid instances delete group
rspec ./spec/util/importer_spec.rb:228 # Migel::Util::Importer Examples update method updates group, subgroup, and migelid instances delete subgroup
rspec ./spec/util/importer_spec.rb:235 # Migel::Util::Importer Examples update method updates group, subgroup, and migelid instances delete migelid
rspec ./spec/util/importer_spec.rb:359 # Migel::Util::Importer Examples update_product example update_products_by_migel_code should update products searched by online migel server
rspec ./spec/util/importer_spec.rb:376 # Migel::Util::Importer Examples update_product example import_all_products_from_csv should update products by a csv file
rspec ./spec/util/importer_spec.rb:490 # Migel::Util::Importer Examples #report should == ["Saved file: ", "Total     1 Migelids (    0 Migelids have products /     0 Migelids have no products)", "Saved  Products", "Save time length: ", "", "Migelids with products (0)", "", "Migelids without products (0)"]
rspec ./spec/util/importer_spec.rb:539 # Migel::Util::Importer Examples #report_save_all_products 
rspec ./spec/util/server_spec.rb:205 # Migel::Util::Server Examples _admin should return method result in the argument 'result' (Array)
rspec ./spec/util/server_spec.rb:213 # Migel::Util::Server Examples _admin should return error message when StandardError happens during the method execution
rspec ./spec/util/swissindex_spec.rb:27 # Migel::Util::Swissindex Migel::Util::Swissindex examples search_migel_table should return an array of hash

Skipping 5 tests. Committed

(Note: migel does not run under 1.8.7 anymore)

But as oddb.org fails running ext/swissindex/test/test_swissindex.rb I think the migration to an external migel was never completed with respect to run unit-tests/rspecs in oddb.org and migel. I should probably cleanup this problem once. Added it to the keep-in-mind list.

After adding a csv monkey patch for migel. I could access migel from oddb-ci2. But know I have to think why my import_daily did not yet finish.

view · edit · sidebar · attach · print · history
Page last modified on June 24, 2014, at 08:19 AM