view · edit · sidebar · attach · print · history

Index>

20140903-fix-migel-update-job

Summary

  • Add import new migel-XLS file to import_daily

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • One unit-test for searchbar fails and might be a clue why searching does not work correctly.
  • Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?
  • Make ext/swissindex/test/test_swissindex.rb and skipping tests in migel-gem pass.

---

Add import new migel-XLS file to import_daily

After discussion with Zeno it is clear for me that I adapt the migel/jobs/* files for the new importer. /var/www/migel/jobs/update_migel_products_with_report is scheduled via crontab to run twice a year. What I do not like about the jobs is that the contain code like 

  Importer.new.save_all_products('data/csv/migel_product_de.csv', 'de', true)
  Importer.new.save_all_products('data/csv/migel_product_fr.csv', 'fr', true)
  Importer.new.import_all_products_from_csv(File.expand_path('../../data/csv/migel_product_de.csv', __FILE__), 'de', true)
  Importer.new.import_all_products_from_csv(File.expand_path('../../data/csv/migel_product_fr.csv', __FILE__), 'fr', true)

In my new method update_all I already have a local Hash for the languages. Will change it as a Constant to be exported from the Migel module. And adding a unified method save_all_products_all_languages(options = {:report => false, :estimate => false}) to replace import_all_products_from_csv while keeping backward compatibility.

Must now find a way to make the skipped test for the importer run in spec/importer_spec.rb. Stubbing the DRbObject works using the following snippet 

    @server = double('swissindex_nonpharmad', :connect => true)
    allow_any_instance_of(DRbObject).to receive(:session).and_return(@server)

Making the tests pass I discovered an errorr in lib/migel/util/importer.rb reported_save_all_products where I changed 

-                   elsif line[0] = /^(\d\d)\.(\d\d)\.(\d\d)\.(\d\d)\.(\d)$/
+                   elsif line[0] =~ /^(\d\d)\.(\d\d)\.(\d\d)\.(\d\d)\.(\d)$/

Now all spec test passed and I am happy with them.

Before running import jub I verified that http://oddb-ci2.dyndns.org/de/gcc/migel_search/migel_limitation/346001001 contains 

Limitation: 2 Sets pro Jahr; Kinder 0-12
Jahre
Indikation: mittelschwere bis schwere Atopische Dermatitis, welche eine kontinuierliche Behandlung mit Emollienten und topischen Steroiden bedarf.
Verschreibung nur durch Fachärzte in Pädiatrie, Dermatologie und Allergologie
Gültig bis 31.12.2012; in Evaluation

This is wrong and should be fixed after running sudo -u apache /usr/local/bin/ruby jobs/update_migel_products_with_report. As I have not added any test for my new method save_all_products_all_languages, I get errors. Therefore I must add some more spec tests. I think they fail because I added 'it' to languages.

No. Problem was that there were even more pending specs, which pointed to problem in the code, e.g. did not handle the case when the logger was empty. See changes in lib/migel/util/server.rb. Also the variables :migel_codes_with_products, :migel_codes_without_products had to be declared and used correctly. Conclusion is that the jobs did not work since about mid 2012.

Adding an rspec for save_all_products_all_languages proved more work than needed as the Migel::Util::Mail is not derived from Mail, but a separate Module (not class). Therefore some refactoring is needed, as I want to be able to use Mail.defaults do delivery_method :test end to simplify testing. Deleting unused method Mail.notify_admins(subject, lines)

Okay now I can test sending emails and I see errors when running the import. Fixing 

       +can't modify frozen String
       +/opt/src/migel/lib/migel/plugin/swissindex.rb:31:in `upcase!'

by replacing lang.upcase! by lang = lang.upcase

Now running jobs/update_migel_products_with_report begins to eat CPU and time, but chokes after a few seconds with 

/usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.7/lib/mail/configuration.rb:25:in `delivery_method': undefined method `new' for :smtp_connection:Symbol (NoMethodError)
        from /var/www/migel/lib/migel/util/mail.rb:26:in `block in notify_admins_attached'
        from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.7/lib/mail/mail.rb:106:in `instance_eval'
        from /usr/local/lib/ruby/gems/1.9.1/gems/mail-2.2.7/lib/mail/mail.rb:106:in `defaults'
        from /var/www/migel/lib/migel/util/mail.rb:25:in `notify_admins_attached'
        from /var/www/migel/lib/migel/util/importer.rb:238:in `reported_save_all_products'

Probably my configuration is not correct, as I never did run a migel job under oddb-ci2. Creating a etc/migel.yml with correct data on oddb-ci2.

Now I received an error e-mail, because of a missing @ before migel_codes_without_products. Correcting this error and relaunching the job. Now waiting for the jobs to complete. I see regularly lines like 126 / 579 Estimate total: 15.32 [m] It will be done in: 11.99 [m] After some Time received the mail for the de. But the content for http://oddb-ci2.dyndns.org/de/gcc/migel_search/migel_limitation/346001001 did not change and still display the old value.

E-Mail details look good for me 

Wed Sep  3 14:55:19 2014: migel Migel::Util::Importer#reported_save_all_products(de)
Saved file: /var/www/migel/data/csv/migel_products_de.csv
Total   579 Migelids (  374 Migelids have products /   205 Migelids have no products)
Saved 32183 Products

Migelids with products (374)
http://ch.oddb.org/de/gcc/migel_search/migel_product/010101001
<..>
Attached file migel_products_de.csv.gz

The attached csv does not contain a header line and the first line looks like 010101001,1624501,7612479001117,AMEDA Einhandmilchpumpe mit Flexishield,Nufer Medical AG ,,32.30,43.70,1,43.70,1 Stk,A,,2010-07-09T00:00:00+00:00,DE Asking Zeno whether this is okay.

Waiting for import of fr/it to finish before restarting all ch.oddb services. Got also report for 'fr', but for 'it' I have to wait a long time. How long? (2h 49')

Okay. Look at our problem via bin/admin on thinpower and got 

migel> Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.name.de
-> Therapeutische Bandage in Bekleidungsform aus Seide mit kovalent gebundener antimikrobieller Funktion 
migel> Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.date
-> 2008-08-01
Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.limitation_text.de[250..500]
-> tologie und Allergologie 
Gültig bis 31.12.2012; in Evaluation

on oddb-ci I get 

migel> Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.name.de
-> Therapeutische Bandage in Bekleidungsform aus Seide mit kovalent gebundener antimikrobieller Funktion 
migel> Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.date
-> -4712-01-01
Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.limitation_text.de[200..500]
-> schreibung nur durch Fachärzte in Pädiatrie, Dermatologie und Allergologie 
Gültig bis 31.12.2012; in Evaluation

It looks like that the limitation_text did not get updated. Waiting for end of import to check again.

At the end of the upgrade I got another error 

ODBA::Stub was unable to replace Hash#32355 from Migel::Model::Migelid:#880
/var/www/migel/lib/migel/util/importer.rb:127:in `block in update': undefined method `delete' for nil:NilClass (NoMethodError)
        from /var/www/migel/lib/migel/util/importer.rb:120:in `each'
        from /var/www/migel/lib/migel/util/importer.rb:120:in `update'
        from /var/www/migel/lib/migel/util/importer.rb:93:in `block in update_all'
        from /var/www/migel/lib/migel/util/importer.rb:83:in `each'
        from /var/www/migel/lib/migel/util/importer.rb:83:in `update_all'
        from jobs/update_migel_products_with_report:14:in `block in <module:Util>'
        from /var/www/migel/lib/migel/util/job.rb:22:in `call'
        from /var/www/migel/lib/migel/util/job.rb:22:in `run'
        from jobs/update_migel_products_with_report:12:in `<module:Util>'
        from jobs/update_migel_products_with_report:11:in `<module:Migel>'
        from jobs/update_migel_products_with_report:10:in `<main>'

But the changes seem to come through as migel/bin/admin shows now 

migel> Migel::Model::Migelid.search_by_migel_code('34.60.01.00.1').first.limitation_text.de[200..500]
-> hreibung nur durch Facha&#776;rzte fu&#776;r Pa&#776;diatrie, Dermatologie und Allergologie

And also the link http://oddb-ci2.dyndns.org/de/gcc/migel_search/migel_product/346001001 shows now

Pushed commit Added importing Migel.xls. Fixed some small errors. Pulling the changes on thinpower and starting import there. Using sudo -u apache /usr/local/bin/ruby /var/www/migel/jobs/update_migel_products_with_report. Must change crontab if import is successful.

After running it a second time on oddb-ci2 I got 

cp -p /var/www/migel/data/csv/MiGeL.xls /var/www/migel/data/csv/MiGeL-2014.09.04.xls
mv /var/www/migel/data/csv/MiGeL.xls /var/www/migel/data/csv/MiGeL-latest.xls
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:588:in `gsub': invalid byte sequence in UTF-8 (ArgumentError)
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:588:in `update_fulltext_index'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:390:in `do_update_index'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:71:in `block (3 levels) in fill'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:70:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:70:in `block (2 levels) in fill'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:69:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:69:in `block in fill'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:66:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:66:in `fill'
        from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:335:in `fill_index'
        from /var/www/migel/lib/migel/util/server.rb:328:in `rebuild_fulltext_index_table'
        from /var/www/migel/lib/migel/util/server.rb:386:in `rebuild_fulltext_index_tables'
        from /var/www/migel/lib/migel/util/server.rb:279:in `init_fulltext_index_tables'
        from jobs/update_migel_products_with_report:15:in `block in <module:Util>'
        from /var/www/migel/lib/migel/util/job.rb:22:in `call'
        from /var/www/migel/lib/migel/util/job.rb:22:in `run'
        from jobs/update_migel_products_with_report:12:in `<module:Util>'
        from jobs/update_migel_products_with_report:11:in `<module:Migel>'
        from jobs/update_migel_products_with_report:10:in `<main>'

Must have look at these errors next week.

view · edit · sidebar · attach · print · history
Page last modified on September 04, 2014, at 07:57 AM