view · edit · sidebar · attach · print · history

20101208-update-spreadsheet-gem-debug-ebook

<< Masa.20101209-debug-ebook-compendium_ch_oddb_org_firefox_epub | 2010 | Masa.20101207-update-outline-spreadsheet >>

Download file for testcase
Create a new gem package
Debug ebook errors
Install spreadsheet on Windows 7
Update Rakefile spreadsheet
Resume debug ebook
Traceback the error message

Goal

Test cases / 80%

Milestones

~~Tag commit~~
~~Download files for test~~
~~Rails test~~
~~New gem package~~ suspend
~~Debug Ebook errors~~ suspend
Debug Ebook

Summary

http://masay.ch/2010/12/08/update-spreadsheet-gem-debug-ebook/

Commits

Updated README.txt, Rakefile, and gempackage

ToDo Tomorrow

Debug Ebook error / Traceback error message

Keep in Mind

Test case spreadsheet outline
jeweler
YARD + rspec
Author info in code
VirtualBox kernel module
todos
BAG: Files published obviously only on the first ch.ODDB.org Report - Error: Generikaliste - 12/2010 log passwd
Testcase of 'Object Stream' parser (for rpdf2txt/lib/rpdf2txt/parser.rb#rebuild_object_catalogue method) 20101126-update-rpdf2txt
Testcases of lib/oddb/html/state/global.rb#grant_download, lib/oddb/html/view/download.rb#to_html
Debug testcases in test/export/test_server.rb de.oddb.org
A bug import_gkv Tue Nov 16 02:00:10 2010: de.oddb.org Zubef (PDF)
Compression (refer to lib/oddb/export/server.rb), Test cases (grant_download, Logging, Reporting)
Log Error: on production server, de.oddb.org/log/import_dimdi, import_pharmnet
On Ice
emerge --sync

Download file for testcase]]

Link

spreadsheet 0.6.4.3 http://rubyforge.org/frs/?group_id=678

I cannot find the file for the test fails.

Create a new gem package

Reference

Package management (Japanese) http://jp.rubyist.net/magazine/?0010-PackageManagement
Hoe (Package management) (Japanese) http://blog.s21g.com/articles/211

Hoe sample

$ sudo gem install hoe
$ sow sample
$ rake -T

Ways to create a gem package

Directly write a gemspec file
hoe
newgem
cutagem
gemity
jeweler
bones
etc.

Memo

jeweler looks better

spreadsheet.gemspec

masa@masa ~/ywesee/spreadsheet $ cat spreadsheet.gemspec
require "rubygems"
require "rake"

spec = Gem::Specification.new do |s|
   s.name        = "spreadsheet"
   s.version     = "0.6.5"
   s.summary     = "The Spreadsheet Library is designed to read and write Spreadsheet Documents"
   s.description = "As of version 0.6.0, only Microsoft Excel compatible spreadsheets are supported"
   s.author      = "Masaomi Hatakeyama, Zeno R.R. Davatz"
   s.email       = "mhatakeyama@ywesee.com, zdavatz@ywesee.com"
   s.platform    = Gem::Platform::RUBY
   s.files       = Dir.glob("{bin,lib,test}/**/*") + Dir.glob("*.txt")
   s.test_file   = "test/suite.rb"
   s.executables << 'xlsopcodes'
   s.add_dependency('iconv')
   s.add_dependency('ruby-ole')
   s.homepage    = "http://scm.ywesee.com/?p=spreadsheet/.git;a=summary"
end

if $0 == __FILE__
   Gem.manage_gems
   Gem::Builder.new(spec).build
end

Make a gem package

$ gem build spreadsheet.gemspec

Note

I cannot 'gem push' with my account because of 'spreadsheet' is already up

suspend

Debug ebook errors

Ebook compendium_ch.oddb.org.firefox.epub could not be completed (01.12.2010 01:02:34) (fwd)

Subject: Ebook compendium_ch.oddb.org.firefox.epub could not be completed
   (01.12.2010 01:02:34)

NoMethodError: undefined method `metadata' for nil:NilClass
/usr/lib64/ruby19/site_ruby/1.9.1/ebps/conversion/fachinfo_yaml.rb:8:in `block in <module:EBPS>'
/usr/lib64/ruby19/1.9.1/yaml.rb:217:in `call'
/usr/lib64/ruby19/1.9.1/yaml.rb:217:in `transfer'
/usr/lib64/ruby19/1.9.1/yaml.rb:217:in `node_import'
/usr/lib64/ruby19/1.9.1/yaml.rb:217:in `load_documents'
/usr/lib64/ruby19/1.9.1/yaml.rb:217:in `each_document'
/usr/lib64/ruby19/site_ruby/1.9.1/ebps/conversion/fachinfo_yaml.rb:83:in `import'
./oddb_firefox:58:in `<main>'

Reference

http://dev.ywesee.com/wiki.php/Main/CHoddb

Confirm the same error in my local

grep check

masa@masa ~/ywesee/ebps $ grep -r compendium *
masa@masa ~/ywesee/ebps $ 

masa@masa ~/ywesee/oddb.org $ grep -r compendium *
�bereinstimmungen in Bin�rdatei doc/resources/downloads/examples/compendium_ch.oddb.org.firefox.epub.
�bereinstimmungen in Bin�rdatei doc/resources/downloads/examples/compendium_ch.oddb.org.stanza.epub.
ext/fiparse/test/test_fachinfo_pdf.rb:                                  assert_nil(/.*compendium.*/i.match(ch_str), ch_str)
ext/fulltext/data/dicts/french/fulltext.dict:compendium/S
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub' => 17,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.htc.prc'      => 17,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.kindle.mobi'  => 17,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.stanza.epub'  => 17,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub' => 30,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.htc.prc'      => 30,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.kindle.mobi'  => 30,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.stanza.epub'  => 30,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub',
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.htc.prc',
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.kindle.mobi',
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.stanza.epub',
src/view/centeredsearchform.rb:          'download[compendium_ch.oddb.org.firefox.epub]' => 1,
src/view/centeredsearchform.rb:          'download[compendium_ch.oddb.org.htc.prc]'      => 1,
src/view/centeredsearchform.rb:          'download[compendium_ch.oddb.org.kindle.mobi]'  => 1,
src/view/centeredsearchform.rb:          'download[compendium_ch.oddb.org.stanza.epub]'  => 1,
src/view/user/download_export.rb:    example('compendium_ch.oddb.org.firefox.epub')
src/view/user/download_export.rb:    example('compendium_ch.oddb.org.htc.prc')
src/view/user/download_export.rb:    example('compendium_ch.oddb.org.kindle.mobi')
src/view/user/download_export.rb:    link = example('compendium_ch.oddb.org.stanza.epub')
src/view/user/download_export.rb:    checkbox_with_filesize('compendium_ch.oddb.org.firefox.epub')
src/view/user/download_export.rb:    checkbox_with_filesize('compendium_ch.oddb.org.htc.prc')
src/view/user/download_export.rb:    checkbox_with_filesize('compendium_ch.oddb.org.kindle.mobi')
src/view/user/download_export.rb:    checkbox_with_filesize('compendium_ch.oddb.org.stanza.epub')
src/view/user/download_export.rb:               once('compendium_ch.oddb.org.firefox.epub')
src/view/user/download_export.rb:               once('compendium_ch.oddb.org.htc.prc')
src/view/user/download_export.rb:               once('compendium_ch.oddb.org.kindle.mobi')
src/view/user/download_export.rb:               once('compendium_ch.oddb.org.stanza.epub')

Notes

These errors come from 'oddb.org' not 'epbs'
But this email subject says 'from EBPS'

Look at these files

masa@masa ~/ywesee/oddb.org $ grep -r compendium_ch.oddb.org.firefox.epub *
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub' => 17,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub' => 30,
src/util/oddbconfig.rb:    'compendium_ch.oddb.org.firefox.epub',
src/view/centeredsearchform.rb:          'download[compendium_ch.oddb.org.firefox.epub]' => 1,
src/view/user/download_export.rb:    example('compendium_ch.oddb.org.firefox.epub')
src/view/user/download_export.rb:    checkbox_with_filesize('compendium_ch.oddb.org.firefox.epub')
src/view/user/download_export.rb:               once('compendium_ch.oddb.org.firefox.epub')

Focus on src/view/user/download_export.rb

ODDB::View::User::DownloadExportInnerComposite < HtmlGrid::Composite

  def example_fachinfo_firefox_epub(model, session)
    example('compendium_ch.oddb.org.firefox.epub')
  end

...


  def fachinfo_epub_firefox(model, session)
    checkbox_with_filesize('compendium_ch.oddb.org.firefox.epub')
  end

...

    def price_fachinfo_firefox_epub(model, session)
        once('compendium_ch.oddb.org.firefox.epub')
    end

suspend

Install spreadsheet on Windows 7

Reference

git for Windows msysGit http://code.google.com/p/msysgit/

Error

C:\Users\masa\spreadsheet>gem build spreadsheet.gemspec
WARNING:  no rubyforge_project specified
  Successfully built RubyGem
  Name: spreadsheet
  Version: 0.6.5
  File: spreadsheet-0.6.5.gem
C:\Users\masa\spreadsheet>gem install spreadsheet-0.6.5.0.gem
ERROR:  Could not find a valid gem 'spreadsheet-0.6.5.0.gem' (>= 0) in any repos
itory

Suceeded in installing

C:\Users\masa\spreadsheet>gem install rake
C:\Users\masa\spreadsheet>gem install hoe
C:\Users\masa\spreadsheet>gem install ruby-ole
C:\Users\masa\spreadsheet>rake -I lib gem
C:\Users\masa\spreadsheet>cd pkg
C:\Users\masa\spreadsheet\pkg>gem install spreadsheet-0.6.5.0.gem
Successfully installed spreadsheet-0.6.5.0
1 gem installed
Installing ri documentation for spreadsheet-0.6.5.0...
Installing RDoc documentation for spreadsheet-0.6.5.0...

C:\Users\masa\spreadsheet\pkg>gem search spreadsheet

*** LOCAL GEMS ***

spreadsheet (0.6.5.0)

Update Rakefile spreadsheet

Commit

Updated README.txt, Rakefile, and gempackage

Resume debug ebook

Memo

'./oddb_firefox' in the error message is '/var/ebps/bin/oddb_firefox'

/var/ebps/bin/oddb_firefox:58


begin

  import_from = EBPS.config.import_from
  export_to = EBPS.config.export_to
  require "ebps/conversion/#{import_from}"
  require "ebps/conversion/#{export_to}"

  EBPS.config.preprocess.each do |name, data|
    require "ebps/preprocess/#{name}"
    result = identify_module(EBPS::Preprocess, name).preprocess(data)
    log.push sprintf('Preprocess: %s', result)
  end

  importer = identify_module EBPS::Conversion, import_from
  exporter = identify_module EBPS::Conversion, export_to

  model = []
  source_path = EBPS.config.source
  inputs = ""
  if File.ftype(source_path) == 'directory'
    inputs << "Input files for conversion:"
    Dir.foreach(source_path) do |name|
      path = File.join source_path, name
      if File.ftype(path) == 'file'
        inputs << "\n" << path
        File.open(path) do |fh|
          model.concat importer.import(fh, path)
        end
      end
    end
  else
    inputs << "Input file for conversion: " << source_path
    source = File.open(source_path)
    model = importer.import source, source_path     <= HERE Line.58
  end
...

Question

How (When) was this code called?

Experiment

masa@masa ~/work/ebps_work_20101108 $ ruby1.9 /var/ebps/bin/oddb_firefox 
/usr/lib64/ruby/site_ruby/1.9.1/ebps/util/mail.rb:14:in `basename': can't convert nil into String (TypeError)
        from /usr/lib64/ruby/site_ruby/1.9.1/ebps/util/mail.rb:14:in `notify'
        from /var/ebps/bin/oddb_firefox:85:in `ensure in <main>'
        from /var/ebps/bin/oddb_firefox:85:in `<main>'

There is a etc/crontab on production server of ebps

mhatakeyama@server /var/ebps/bin $ cat /etc/crontab 
...
# export ebooks
0  1  1 * *     xxx  /var/ebps/bin/run_ch.oddb_ebooks
0  1  1 * *     xxx  /var/ebps/bin/run_de.oddb_ebooks
30  6  1 * *    xxx  /var/ebps/bin/run_emediat_ebooks

mhatakeyama@server /var/ebps/bin $ cat /var/ebps/bin/run_ch.oddb_ebooks 
#!/bin/bash

# download ch.oddb.org-fi-data
cd /var/ebps/data/yaml
scp user@xxx:/var/www/oddb.org/data/downloads/fachinfo.yaml fachinfos.ch.oddb.yaml

# start conversion
cd /var/ebps/bin
./oddb_firefox&
./oddb_stanza&
./oddb_kindle&

wait # for all processes to finish

Notes

'oddb_firefox' is called from crond once a month (on the 1st day)

Experiment

config.yaml

masa@masa ~/work/ebps $ cat config.yml 
---
smtp_server: smtp.gmail.com
smtp_user: xxx@ywesee.com
smtp_pass: 'xxx'
report_to:
 - mhatakeyama@ywesee.com
#source: /home/masa/work/data.yml
source: /home/masa/work/fachinfos.ch.oddb.yaml
target: /home/masa/work/sample.epub
import_from: import_module_sample
export_to: epub
language: de
title: Firmenliste Schweiz

converter_for_firefox.rb

#!/usr/bin/env ruby19

$: << File.expand_path('../lib', File.dirname(__FILE__))

require 'ebps'
require 'ebps/util/mail'

def identify_module namespace, name
  module_name = name.gsub /(^|_)./ do |match| match.upcase.delete '_' end
  namespace.const_get module_name
end

begin

  log = []

  # Read modules
  import_from = EBPS.config.import_from
  export_to = EBPS.config.export_to
  require "ebps/conversion/#{import_from}"
  require "ebps/conversion/#{export_to}"

  importer = identify_module EBPS::Conversion, import_from
  exporter = identify_module EBPS::Conversion, export_to

  # Import process
  model = []
  source_path = EBPS.config.source
  inputs = ""
  if File.ftype(source_path) == 'directory'
    inputs << "Input files for conversion:"
    Dir.foreach(source_path) do |name|
      path = File.join source_path, name
      if File.ftype(path) == 'file'
        inputs << "\n" << path
        File.open(path) do |fh|
          model.concat importer.import(fh, path)
        end
      end
    end
  else
    inputs << "Input file for conversion: " << source_path
    source = File.open(source_path)
    model = importer.import source, source_path
  end
  log.push inputs

  # Export process
  target = EBPS.config.target
  exporter.export model, target

  log.push sprintf("The Ebook %s was generated and stored in %s",
                   File.basename(target), File.dirname(target))

rescue StandardError => err
  log.push err
ensure
  EBPS::Util::Mail.notify log
end

Run converter

masa@masa ~/work $ ls -al
drwxr-xr-x  7 masa masa      4096  8. Dez 16:00 ebps
-rwxr-xr-x  1 masa masa 391877556  8. Dez 15:57 fachinfos.ch.oddb.yaml

masa@masa ~/work/ebps $ ruby1.9 -I lib converter_for_firefox.rb config="config.yml"

Note

'fachinfos.ch.oddb.yaml' is the file downloaded from the production server
- /var/www/oddb.org/data/downloads/fachinfo.yaml

Result

Input file for conversion: /home/masa/work/fachinfos.ch.oddb.yaml

NoMethodError: undefined method `title' for #<Syck::DomainType:0x00000000f122f0>
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:436:in `block (2 levels) in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `each'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `sort_by'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `block in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `call'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `with_tmpdir'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:431:in `export'
/home/masa/work/ebps/lib/ebps/conversion/epub.rb:32:in `export'
converter_for_firefox.rb:50:in `<main>'

Experiment

masa@masa ~/work/ebps $ ruby1.9 -I lib oddb_firefox config="config.yml"

Result

The same error
Ebook sample.epub could not be completed (08.12.2010 16:32:13)

Input file for conversion: /home/masa/work/fachinfos.ch.oddb.yaml

NoMethodError: undefined method `title' for #<Syck::DomainType:0x0000000293c568>
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:436:in `block (2 levels) in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `each'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `sort_by'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `block in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `call'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `with_tmpdir'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:431:in `export'
/home/masa/work/ebps/lib/ebps/conversion/epub.rb:32:in `export'
oddb_firefox:72:in `<main>'

Consideration

'import_module_sample' in config.yml may be wrong

Experiment

config.yml

---
smtp_server: smtp.gmail.com
smtp_user: xxx@ywesee.com
smtp_pass: 'xxx'
report_to:
 - mhatakeyama@ywesee.com
#source: /home/masa/work/data.yml
source: /home/masa/work/fachinfos.ch.oddb.yaml
target: /home/masa/work/sample.epub
#import_from: import_module_sample
import_from: fachinfo_yaml
export_to: epub
language: de
title: Firmenliste Schweiz

Run oddb_firefox again

masa@masa ~/work/ebps $ ruby1.9 -I lib oddb_firefox config="config.yml"

Result

The same error as the server came!!!
https://mail.google.com/a/ywesee.com/#inbox/12cc6a433019af52

NoMethodError: undefined method `metadata' for nil:NilClass
/home/masa/work/ebps/lib/ebps/conversion/fachinfo_yaml.rb:8:in `block in <module:EBPS>'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `call'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `transfer'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `node_import'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `load_documents'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `each_document'
/home/masa/work/ebps/lib/ebps/conversion/fachinfo_yaml.rb:83:in `import'
oddb_firefox:58:in `<main>'

Traceback the error message

Experiment

ebps/lib/ebps/conversion/fachinfo_yaml.rb

require 'ebps/text/document'
require 'yaml'

module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      doc = descr[EBPS.config.language]
print "doc=", doc, "\n"
      doc.metadata.update val
      doc
    end
  end
...

Run again

masa@masa ~/work/ebps $ ruby1.9 -I lib oddb_firefox config="config.yml"

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101208-update-spreadsheet-gem-debug-ebook

Download file for testcase]]

Create a new gem package

Debug ebook errors

Confirm the same error in my local

Install spreadsheet on Windows 7

Update Rakefile spreadsheet

Resume debug ebook

Traceback the error message

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101208-update-spreadsheet-gem-debug-ebook

Download file for testcase]]

Create a new gem package

Debug ebook errors

Confirm the same error in my local

Install spreadsheet on Windows 7

Update Rakefile spreadsheet

Resume debug ebook

Traceback the error message

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler