view · edit · sidebar · attach · print · history

20130903-export-doctors-2

<< | Index | >>


Summary

  • error in export doctors.csv
  • more cleanups in unit tests

Commits

Index


Is there a problem in src/util/validator.rb

Currently we have two errors when running test/test_util/validator.rb

  1) Error:
test_search_query(TestOddbValidator):
NoMethodError: undefined method `u' for #<ODDB::Validator:0x00000002cf8120>
    /opt/src/oddb.org/src/util/validator.rb:578:in `search_query'
    /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/sbsm-1.2.3/lib/sbsm/validator.rb:146:in `perform_validation'
    /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/sbsm-1.2.3/lib/sbsm/validator.rb:119:in `validate'
    /opt/src/oddb.org/test/test_util/validator.rb:72:in `test_search_query'

  2) Error:
test_search_query_shorter_than_3(TestOddbValidator):
NoMethodError: undefined method `u' for #<ODDB::Validator:0x00000002d2c948>
    /opt/src/oddb.org/src/util/validator.rb:578:in `search_query'
    /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/sbsm-1.2.3/lib/sbsm/validator.rb:146:in `perform_validation'
    /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/sbsm-1.2.3/lib/sbsm/validator.rb:119:in `validate'
    /opt/src/oddb.org/test/test_util/validator.rb:75:in `test_search_query_shorter_than_3'

39 tests, 63 assertions, 0 failures, 2 errors, 0 skips

This small patch fixes the problem

index 7f67e73..2ad4f73 100644
--- a/src/util/validator.rb
+++ b/src/util/validator.rb
@@ -575,7 +575,7 @@ module ODDB
         value = @@utf8.iconv(value)
                          result = @@utf8.iconv(result)
                        end
-                       result = u result
+      result = result.force_encoding('utf-8')
                        if(result.length > 2)
                                result
                        else

I tried to reproduce the problemm on oddb-ci.dyndns.org by Executing a search with 'po', but got the @@Es ist ein Fehler aufgetreten: Ihr Such-Stichwort ergibt ein sehr grosses Resultat. Bitte verwenden Sie mindestens 3 Buchstaben.@@

Apply the patch and rerunning the same query. No problem.

Found a few definitions for a method u on my VM via

grep -rw "def u" ~/.rvm/
/home/vagrant/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/matrix/lup_decomposition.rb:    def u
/home/vagrant/.rvm/src/ruby-1.9.3-p0/test/ruby/test_m17n_comb.rb:    def u(str) str.dup.force_encoding("UTF-8") end
/home/vagrant/.rvm/src/ruby-1.9.3-p0/test/ruby/test_m17n.rb:    def u(str) str.dup.force_encoding("UTF-8") end
/home/vagrant/.rvm/src/ruby-1.9.3-p0/test/ruby/test_string.rb:  def u(str)
/home/vagrant/.rvm/src/ruby-1.9.3-p0/lib/matrix/lup_decomposition.rb:    def u
/home/vagrant/.rvm/gems/ruby-1.9.3-p0/gems/character-encodings-0.4.1/lib/encoding/character/utf-8.rb:  def u(str)
/home/vagrant/.rvm/gems/ruby-1.9.3-p0/gems/odba-1.1.0/lib/odba/18_19_loading_compatibility.rb:  def u str

Inside my /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/bin/ruby I found it only once /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/matrix/lup_decomposition.rb: def u

Therefore I think my patch is correct and compatible with newer version of Ruby. But I wondering why in our test-suite the 18_19_loading_compatibility.rb is not loaded. Anyway in 18_19_loading_compatibility.rb the method "u" returns an unchanged string. Therefore I could simplify my patch to remove the line completely.

Pushed commit Remove obsolete call to u

fix error with doctors.csv

Running with ruby 1.9.3-p0 on my VM the job sudo -u apache /usr/local/rvm/rubies/ruby-1.9.3-p0/bin/ruby /var/www/oddb.org/jobs/mail_index_therapeuticus_csv 2>&1 | tee /vagrant/mail_index_therapeuticus_csv.log

I don't get any errors but at least 1902 warnings /usr/local/rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:313: warning: regexp match /.../n against to UTF-8 string which might help

more cleanups in unit tests

With the following command I see the status of each test-suite by parsing its log

 2013-09-02 19:13:53 +0200: Now testing /opt/src/oddb.org/test/../ext/suite.rb true
239 tests, 593 assertions, 17 failures, 21 errors, 0 pendings, 0 omissions, 0 notifications
2013-09-02 19:14:33 +0200: Now testing /opt/src/oddb.org/test/test_state/suite.rb true
796 tests, 816 assertions, 54 failures, 83 errors, 0 skips
2013-09-02 19:14:43 +0200: Now testing /opt/src/oddb.org/test/test_view/suite.rb true
271 tests, 299 assertions, 33 failures, 23 errors, 1 skips
2013-09-02 19:14:48 +0200: Now testing /opt/src/oddb.org/test/test_command/suite.rb true
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
2013-09-02 19:14:55 +0200: Now testing /opt/src/oddb.org/test/test_remote/suite.rb true
6 tests, 6 assertions, 1 failures, 0 errors, 0 skips
2013-09-02 19:14:58 +0200: Now testing /opt/src/oddb.org/test/test_custom/suite.rb true
43 tests, 43 assertions, 10 failures, 0 errors, 0 skips
2013-09-02 19:15:02 +0200: Now testing /opt/src/oddb.org/test/test_plugin/suite.rb true
446 tests, 658 assertions, 14 failures, 72 errors, 0 skips
2013-09-02 19:15:45 +0200: Now testing /opt/src/oddb.org/test/test_model/suite.rb true
661 tests, 1380 assertions, 22 failures, 1 errors, 0 skips
2013-09-02 19:15:52 +0200: Now testing /opt/src/oddb.org/test/test_util/suite.rb true
312 tests, 457 assertions, 17 failures, 6 errors, 0 skips

Will clean-up the unit test using the following strategy.

  • Fix failures where nr errors <10 && failures 10
  • Fix errors where sorted by errors ascending

Pushed a few commits

Found one place where we still had a non-UTF-8 string in the source code. Fixed it with commit Fixed UTF-8 error in ydim.rb and unit-tests. I think our problem when exporting the doctors.csv could stem from another occurrence of the same problem. Is there maybe a header or a database column still defined with a non-UTF-8 string?

view · edit · sidebar · attach · print · history
Page last modified on September 11, 2013, at 02:02 PM