view · edit · sidebar · attach · print · history

< Niklaus.20161030-delete_bauerfeind_takes_too_long | Index | Niklaus.20160824-bauerfeind-problems >>

20160825-oddb2xml-zip-problem

Summary

Commits

Index

Oddb2xml has problems zipping

Got the following error e-mail

/opt/ywesee/oddb2xml_download: line 10: /opt/ywesee/oddb2xml_download.rb: No such file or directory
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:5: warning: already initialized constant Net::NTLM::VERSION::MAJOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:53: warning: previous definition of MAJOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:6: warning: already initialized constant Net::NTLM::VERSION::MINOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:54: warning: previous definition of MINOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:7: warning: already initialized constant Net::NTLM::VERSION::TINY
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:55: warning: previous definition of TINY was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:8: warning: already initialized constant Net::NTLM::VERSION::STRING
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:56: warning: previous definition of STRING was here
/opt/ywesee/oddb2xml_download: line 17: 33266 Killed                  oddb2xml -e -I 45 -c zip
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:5: warning: already initialized constant Net::NTLM::VERSION::MAJOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:53: warning: previous definition of MAJOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:6: warning: already initialized constant Net::NTLM::VERSION::MINOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:54: warning: previous definition of MINOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:7: warning: already initialized constant Net::NTLM::VERSION::TINY
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:55: warning: previous definition of TINY was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:8: warning: already initialized constant Net::NTLM::VERSION::STRING
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:56: warning: previous definition of STRING was here
/opt/ywesee/oddb2xml_download: line 32: 33585 Killed                  oddb2xml -e -I 50 -c zip
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:5: warning: already initialized constant Net::NTLM::VERSION::MAJOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:53: warning: previous definition of MAJOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:6: warning: already initialized constant Net::NTLM::VERSION::MINOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:54: warning: previous definition of MINOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:7: warning: already initialized constant Net::NTLM::VERSION::TINY
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:55: warning: previous definition of TINY was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:8: warning: already initialized constant Net::NTLM::VERSION::STRING
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:56: warning: previous definition of STRING was here
/opt/ywesee/oddb2xml_download: line 44: 33605 Killed                  oddb2xml -e -I 55 -c zip
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:5: warning: already initialized constant Net::NTLM::VERSION::MAJOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:53: warning: previous definition of MAJOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:6: warning: already initialized constant Net::NTLM::VERSION::MINOR
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:54: warning: previous definition of MINOR was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:7: warning: already initialized constant Net::NTLM::VERSION::TINY
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:55: warning: previous definition of TINY was here
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:8: warning: already initialized constant Net::NTLM::VERSION::STRING
/opt/ywesee/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:56: warning: previous definition of STRING was here
/opt/ywesee/oddb2xml_download: line 56: 33624 Killed                  oddb2xml -e -c zip

Looks as if oddb2xml -e -c zip got killed. Trying to reproduce the probleme here. But why was the process killed? Did it use too much memory or cpu time?

The problem is, that the current directory contains a transfer.zip, which should only be present in the download sub-directory.

Adding the following lines to the rspec of Oddb2xml::ZurroseDownloader

      expect(File.exist?(File.join(Oddb2xml::Downloads, 'transfer.dat'))).to eq(true)
      expect(File.exist?(File.join(Oddb2xml::Downloads, 'transfer.zip'))).to eq(true)
      expect(File.exist?('transfer.zip')).to eq(false)

Remarked the following things:

  • download_as did not copy the downloade files into the downloads directory
  • There were few spec tests, to check whether downloads went into the download directory, as some files are used from the current directory.

Pushed commits

Job import Bauerfeind ended with

 sudo -u apache jobs/import_migel_bauerfeind 

resulted with

2016-08-25 03:06:08 +0200: delete_all_bauerfeind_products. 70450 4046445241017 4046445241017 17.03.02.00.1
2016-08-25 03:06:28 +0200: delete_all_bauerfeind_products. 70451 4046445241024 4046445241024 17.03.02.00.1
2016-08-25 03:06:43 +0200: Deleted 942 products from 17.03.02.00.1
2016-08-25 03:12:19 +0200: Deleted 2009 products. Having 4 active_bauerfeind_products of 732
/var/www/migel/lib/migel/util/csv_importer.rb:168:in `block in delete_all_inactive_bauerfeind_products': undefined local variable or method `migel_code' for #<Migel::Util::CsvImporter:0x0000000291da28> (NameError)
        from /var/www/migel/lib/migel/util/csv_importer.rb:167:in `each'
        from /var/www/migel/lib/migel/util/csv_importer.rb:167:in `delete_all_inactive_bauerfeind_products'
        from /var/www/migel/lib/migel/util/csv_importer.rb:88:in `import_all_products_from_csv'
        from jobs/import_migel_bauerfeind:21:in `block in <module:Util>'
        from /var/www/migel/lib/migel/util/job.rb:22:in `call'
        from /var/www/migel/lib/migel/util/job.rb:22:in `run'
        from jobs/import_migel_bauerfeind:14:in `<module:Util>'
        from jobs/import_migel_bauerfeind:13:in `<module:Migel>'
        from jobs/import_migel_bauerfeind:12:in `<main>'

restarting with

  sudo -u apache jobs/import_migel_bauerfeind

this results in the same error

2016-08-25 10:11:45 +0200: import_all_products_from_csv: options are {:report=>true, :estimate=>true, :filename=>"/var/www/migel/data/csv/update_migel_bauerfeind.csv"}
2016-08-25 10:11:45 +0200: import_all_products_file_name are "/var/www/migel/data/csv/update_migel_bauerfeind.csv"
2016-08-25 10:11:45 +0200: import_all_products_from_csv: file_name /var/www/migel/data/csv/update_migel_bauerfeind.csv lang de estimate true
2016-08-25 10:22:30 +0200: Deleted 0 products. Having 4 active_bauerfeind_products of 732
/var/www/migel/lib/migel/util/csv_importer.rb:168:in `block in delete_all_inactive_bauerfeind_products': undefined local variable or method `migel_code' for #<Migel::Util::CsvImporter:0x00000001f31c58> (NameError)
        from /var/www/migel/lib/migel/util/csv_importer.rb:167:in `each'
        from /var/www/migel/lib/migel/util/csv_importer.rb:167:in `delete_all_inactive_bauerfeind_products'
        from /var/www/migel/lib/migel/util/csv_importer.rb:88:in `import_all_products_from_csv'
        from jobs/import_migel_bauerfeind:21:in `block in <module:Util>'
        from /var/www/migel/lib/migel/util/job.rb:22:in `call'
        from /var/www/migel/lib/migel/util/job.rb:22:in `run'
        from jobs/import_migel_bauerfeind:14:in `<module:Util>'
        from jobs/import_migel_bauerfeind:13:in `<module:Migel>'
        from jobs/import_migel_bauerfeind:12:in `<main>'

Pushed the fix Don't use undefined variable migel_code

view · edit · sidebar · attach · print · history
Page last modified on August 25, 2016, at 10:35 AM