<< | Index | >>
Files
Note
Run
ruby193 -I ../oddb/lib bin/oddbd
Error
/home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': /home/masa/ywesee/oddb.org.ruby193/src/plugin/who.rb:56: invalid multibyte escape: /\240/ (SyntaxError) from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/util/updater.rb:24:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/state/admin/init.rb:8:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/state/admin/global.rb:5:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/state/admin/selectsubstance.rb:5:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/state/admin/activeagent.rb:8:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:12:in `block in require_r' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:7:in `each' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:7:in `require_r' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:17:in `block in require_r' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:7:in `each' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:7:in `require_r' from /home/masa/ywesee/oddb.org.ruby193/src/state/states.rb:23:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/util/session.rb:8:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:22:in `<top (required)>' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require' from bin/oddbd:13:in `<main>'
Experiment
def extract_text(node)
unless(node.children.any? { |br| br.element? && br.name != 'br' })
html = node.inner_html
if RUBY_VERSION < '1.9'
#html.gsub! /\240/, ''
end
html.gsub(/\s+/, ' ').gsub(/\s*<br\s*\/?>\s*/, "\n").strip
end
end
Result
Problem
Error
error in SBSM::Session#to_html: /de/gcc/fachinfo/swissmedicnr/57455 NoMethodError undefined method `collect' for nil:NilClass /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:70:in `sections' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:118:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:42:in `block (3 levels) in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `each' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `inject' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `block (2 levels) in to_html' (eval):373:in `div' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:39:in `block in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `each' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `each_with_index' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:132:in `block in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:125:in `call' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:125:in `template_tags' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:111:in `block (2 levels) in template_html' (eval):1086:in `body' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:110:in `block in template_html' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/cgi/html.rb:437:in `block in html' (eval):1069:in `html' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/cgi/html.rb:437:in `html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:109:in `template_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:131:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/state.rb:179:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:450:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:170:in `block in drb_process' <internal:prelude>:10:in `synchronize' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:168:in `drb_process' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
Note
rebulid_indices
all Indices Created in total: 0.28421154971 h (about 10 minutes)
Access with data yesterday (before migerate_to_utf8 running last night)
Error
error in SBSM::Session#to_html: /de/gcc/fachinfo/swissmedicnr/57455 Encoding::CompatibilityError incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string) /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:77:in `match' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:77:in `block (2 levels) in sections' (eval):1103:in `p' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:71:in `block in sections' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:70:in `collect' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:70:in `sections' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:120:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:42:in `block (3 levels) in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `each' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `inject' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `block (2 levels) in to_html' (eval):373:in `div' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:39:in `block in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `each' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `each_with_index' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:38:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/composite.rb:277:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:132:in `block in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:125:in `call' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:125:in `template_tags' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:111:in `block (2 levels) in template_html' (eval):1086:in `body' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:110:in `block in template_html' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/cgi/html.rb:437:in `block in html' (eval):1069:in `html' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/cgi/html.rb:437:in `html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:109:in `template_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/template.rb:131:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/state.rb:179:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:450:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:170:in `block in drb_process' <internal:prelude>:10:in `synchronize' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/sbsm-1.0.0/lib/sbsm/session.rb:168:in `drb_process' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop' /home/masa/bin/ruby193rc1/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
Experiment
def sections(context, sections) section_attr = { 'style' => @lookandfeel.section_style } subhead_attr = { 'style' => self.class::SUB_STYLE } sections.collect { |section| context.p(section_attr) { head = context.span(subhead_attr) { self.escape(section.subheading) } section.subheadingp section.subheading
p section.subheading.encoding
Result
"" #<Encoding:ASCII-8BIT> "" #<Encoding:ASCII-8BIT> "" #<Encoding:ASCII-8BIT> "" #<Encoding:ASCII-8BIT> "Erwachsene: \xC3\xBCbliche Einzel- bzw. Tagesdosis\n" #<Encoding:ASCII-8BIT>
Check on bin/admin
ch.oddb> ODBA.cache.fetch('27322904').descriptions.values[0].usage.sections[1].subheading -> Erwachsene: übliche Einzel- bzw. Tagesdosis ch.oddb> ODBA.cache.fetch('27322904').descriptions.values[0].usage.sections[1].subheading.encoding -> ASCII-8BIT
Experiment
ch.oddb> ODBA.cache.fetch('27322904').descriptions.values[0].usage.sections[1].subheading.force_encoding('utf-8') -> Erwachsene: übliche Einzel- bzw. Tagesdosis ch.oddb> ODBA.cache.fetch('27322904').descriptions.values[0].usage.sections[1].subheading.encoding -> UTF-8
Access
Error
error in SBSM::Session#to_html: /de/gcc/fachinfo/swissmedicnr/57455 Encoding::CompatibilityError incompatible character encodings: UTF-8 and ASCII-8BIT /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:82:in `block (2 levels) in sections' (eval):1103:in `p' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:71:in `block in sections' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:70:in `collect' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:70:in `sections' /home/masa/ywesee/oddb.org.ruby193/src/view/chapter.rb:120:in `to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:42:in `block (3 levels) in to_html' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `each' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `inject' /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.0/lib/htmlgrid/divcomposite.rb:40:in `block (2 levels) in to_html' (eval):373:in `div' ...
Note
Run migrate_to_utf8 again with the latest downloaded data
Note
After migrate_to_utf8 running, if oddbd runs on Ruby 1.8, some data will be gone
Problem
strip script (Python)
Run
masa@masa ~/work $ python --version Python 2.6.4 masa@masa ~/work $ python kindlestrip.py compendium_ch.oddb.org_kindle_v1.2.mobi compendium_ch.oddb.org_kindle_v1.2.strip.mobi masa@masa ~/work $ ls -lh -rw-r--r-- 1 masa masa 52M 19. Okt 15:22 compendium_ch.oddb.org_kindle_v1.2.mobi -rw-r--r-- 1 masa masa 18M 19. Okt 15:23 compendium_ch.oddb.org_kindle_v1.2.strip.mobi
Note
Task
oddb_kindle.yaml
... postprocess: - - system_call - command_lines: - - '/home/masa/ywesee/ebps/data/books/' - 'python /home/masa/bin/kindlestrip.py test_kindle_v1.2.mobi test_kindle_v1.2.strip.mobi'
Run
masa@masa ~/ywesee/ebps $ ruby1.9 -I lib bin/oddb_kindle config="etc/oddb_kindle.yml" kindlegen_args='-c2' kindlegen_path='kindlegen_v1.2'
Log
************************************************** * Amazon.com kindlegen(Linux) V1.2 build 33307 * * A command line e-book compiler * * Copyright Amazon.com 2011 * ************************************************** option: -c2: Kindle Huffdic compression Info(prcgen): Added metadata dc:Title "20111019_0 Test Kindle 1.2" Info(prcgen): Added metadata dc:Date "2011-10-19" Info(prcgen): Added metadata dc:Creator "ch.oddb.org" Info(prcgen): Parsing files 0000009 Info(prcgen): Resolving hyperlinks Info(prcgen): Building table of content URL: /tmp/d20111019-3036-36vq6a/toc.ncx Info(pagemap): No Page map found in the book Info(prcgen): Computing UNICODE ranges used in the book Info(prcgen): Found UNICODE range: Basic Latin [20..7E] Info(prcgen): Found UNICODE range: Latin-1 Supplement [A0..FF] Info(prcgen): Found UNICODE range: General Punctuation - Windows 1252 [2013..2014] Info(prcgen): Found UNICODE range: Basic Greek [370..3FF] Info(prcgen): Found UNICODE range: Mathematical Operators [2200..22FF] Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000001 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000002 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000004 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000008 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000016 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000032 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000064 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000128 Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass 0000154 Info(prcgen/compress): Compression pass 0000001 Info(prcgen/compress): Compression pass 0000002 Info(prcgen/compress): Compression pass 0000003 Info(prcgen/compress): Text compressed to (in % of original size): 056.84% Info(prcgen/compress): Compression dictionary statistics: 000019698 bytes 000001663 entries Info(prcgen/compress): Compression pass 0000004 Info(prcgen/compress): Text compressed to (in % of original size): 053.23% Info(prcgen/compress): Compression dictionary statistics: 000016820 bytes 000001300 entries Info(prcgen/compress): Compression pass 0000005 Info(prcgen/compress): Text compressed to (in % of original size): 051.35% Info(prcgen/compress): Compression dictionary statistics: 000014674 bytes 000001056 entries Info(prcgen/compress): Compression pass 0000006 Info(prcgen/compress): Text compressed to (in % of original size): 047.16% Info(prcgen/compress): Compression dictionary statistics: 000011120 bytes 000000863 entries Info(prcgen/compress): Compression pass 0000007 Info(prcgen/compress): Text compressed to (in % of original size): 046.77% Info(prcgen/compress): Compression dictionary statistics: 000010024 bytes 000000709 entries Info(prcgen/compress): Compression pass 0000008 Info(prcgen/compress): Text compressed to (in % of original size): 046.91% Info(prcgen/compress): Compression dictionary statistics: 000009256 bytes 000000588 entries Info(prcgen/compress): Advanced compression successful (decoded and verified). Info(prcgen): Final stats - text compressed to (in % of original size): 046.91% Info(prcgen): The document identifier is: "20111019_0_Test_Kindle_1.2" Info(prcgen): The file format version is V6 Info(prcgen): Saving MOBI file Info(prcgen): MOBI File successfully generated! (postprocess)KindleStrip v1.20. Written 2010-2011 by Paul Durrant.
done
Header Bytes: 53524353000000100000002f00000001
Result
masa@masa ~/ywesee/ebps $ ls -lh data/books/ -rw-r--r-- 1 masa masa 324K 19. Okt 16:10 test_kindle_v1.2.mobi -rw-r--r-- 1 masa masa 186K 19. Okt 16:10 test_kindle_v1.2.strip.mobi
Note
kindlegen_path: /home/masa/bin/kindlegen_v1.2 kindlegen_args: -c2 postprocess: - - system_call - command_lines: - - '/home/masa/ywesee/ebps/data/books/' - 'python /home/masa/bin/kindlestrip.py test_kindle_v1.2.mobi test_kindle_v1.2.strip.mobi'
Task
Note