view · edit · sidebar · attach · print · history

Index>

20160504-ydim-html-to-utf8

Summary

  • Migrate ydim-html to UTF8

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)
  • BSV-Plugin does not send mail. Is it sending two mails at once?
  • remove obsolete pdf_patinfo in model, admin/views
  • remove obsolete ext from html-grid
  • Fix sbsm/htmlgrid when DEFAULT_FLAVOUR is nil

Migrate ydim-html to UTF

Continuing work of yesterday. Looks like we have some weird data in the DB. Eg. my debug output prints 

child has encoding ISO-8859-1 Grünenthal Pharma AG
child has encoding ISO-8859-1 8756  Mitlödi 
child has encoding UTF-8 Frau
 62: Migrating Array []
child has encoding ISO-8859-1 Christine.Unsinn@grunenthal.com
 62: Migrating Array ["SÀndli 2  "]
child has encoding UTF-8 SÀndli 2  
child has encoding ISO-8859-1 Unsinn
child has encoding UTF-8 dt_info
 62: Migrating YDIM::Debitor #<YDIM::Debitor:0x007f6dbe193c68>

ArgumentError: invalid byte sequence in UTF-8 when updating index 'ydim_debitor_name' with a YDIM::Debitor
["/mnt/src/ydim/vendor/ruby/2.3.0/bundler/gems/odba-065f94642746/lib/odba/index.rb:278:in `downcase'", "/mnt/src/ydim/vendor/ruby/2.3.0/bundler/gems/odba-065f94642746/lib/odba/index.rb:278:in `block in search_terms'", "/mnt/src/ydim/vendor/ruby/2.3.0/bundler/gems/odba-065f94642746/lib/odba/index.rb:278:in `collect'", "/mnt/src/ydim/vendor/ruby/2.3.0/bundler/gems/odba-065f94642746/lib/odba/index.rb:278:in `search_terms'"]

Most of the stuff could be fixed, but found 

[1] pry(#<YDIM::Server>)> child
=> "G\xE9rard Brugger"
[2] pry(#<YDIM::Server>)> sanitize_utf8(child)
=> "G\xE9rard Brugger"
[7] pry(#<YDIM::Server>)> child.valid_encoding?
=> false
18] pry(#<YDIM::Server>)> child.force_encoding(Encoding::ISO_8859_1).clone.encode(Encoding::UTF_8)
=> "Gérard Brugger"

Fixed some more border cases. Now the script runs find (Takes about two minutes, if nothing has to be changed). Verifying visual output. Generated PDF does not contain logo and some header info.

Reloading the databse and running migration again (as my first test 'eated' some invalid characters, which should no longer be the case). This looks fine.

Attacking the problem with missing Header/Logo in PDFs (eg. invoice 12438, Grünenthal). The following errors must be corrected:

  • Replace english name like "Description" by german "Beschreibung"
  • Ywesee logo is missing
  • Ywesee address and info like MwSt, IBAN is missing
  • Invoice is shown as "#012438" insted of "Rechnung 12438"
  • Missing space between "1 x Patinfo" and address
  • Invoice date is shown as 01.01.-4712 instead of 18.12.2015
  • Missing line "Bedingungen: zahlbar in 10 Tagen"
  • Missing greeting lines "Ohne Ihre Gegenmeldung .. 043 540 05 50"

The display of the invoice looks okay. See

When looking at https://ydim.oddb-ci2.dyndns.org/de/ydim/autoinvoice/unique_id/10030 I got an ArgumentError invalid byte sequence in UTF-8 After adding autoinvoices to the table to be migrated, this page displays correctly, too.

Dateformat is wrong (shows 01.01.-4712 in https://ydim.oddb-ci2.dyndns.org/de/ydim/debitor/unique_id/334 for a paid invoice.

Okay. This roots in bad values read from the database, e.g pry shows the following when invoice in the view invoices. 

from (pry):7:in `date'
[8] pry(#<YDIM::Html::View::InvoiceInnerComposite>)> model.items.first.time.to_date
=> #<Date: 2009-10-15 ((2455120j,0s,0n),+0s,2299161j)>
[9] pry(#<YDIM::Html::View::InvoiceInnerComposite>)> model.date
=> #<Date: -4712-01-01 ((0j,43200s,(172800000000000/4910239)n),+0s,2299161j)>
[10] pry(#<YDIM::Html::View::InvoiceInnerComposite>)> model.date.year
=> -4712
[11] pry(#<YDIM::Html::View::InvoiceInnerComposite>)> model.items.first.time.year
=> 2009

Therefore I will patch the model.date when being displayed. No, I think I must patch the value inside the YDIM::Invoice class. Added a test case, ported all unit test of ydim to Minitest.

Pushed commits

Merged the changes from branch ruby2x into my master branch with commit Merged changes from ruby2x

Must also push some changes made to htmlgrid and sbsm.

Must fix a few unit test failures in sbsm before pushing it.

view · edit · sidebar · attach · print · history
Page last modified on May 04, 2016, at 03:22 PM