view · edit · sidebar · attach · print · history

20110128-check-migrate-ramaze_oddb

<< | Index | >>


  1. Make class diagram (data structure) in each project suspend
  2. Read ticket#16 Import existing ch.oddb.org-data from the old database into 1.9

Goal
  • Understand data structure / 70%
Milestones
  1. Install ascii-diagram
  2. Read ticket#16
  3. Read the code 'migrate'
Summary
Commits
ToDo Tomorrow
Keep in Mind
  1. Check drug data structure (class diagram) in ch.oddb.org, de.oddb.org, and ramaze oddb
  2. Check rdbi instead of dbi for ODBA
  3. Encoding woes (from Davatz-san)
  4. Feedback: This option indicates that the regular expression is parsed as 'UTF8' (from Davatz-san)
  5. pg on Ubuntu - see http://dev.ywesee.com/wiki.php/Gem/Pg (from Davatz-san)
  6. On Ice
  7. emerge --sync

Make class diagram (data structure) in each project

Dia install message

Package libpng12 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpng12.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libpng12', required by 'cairo', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GTK_CFLAGS
and GTK_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

Re-install cairo

 sudo emerge cairo

Try install dia again

 sudo emerge dia

Read ticket#16 Import existing ch.oddb.org-data from the old database into 1.9

http://trac.ywesee.com/ticket/16

Notes

  • The class structure (class-hierarchy) is totally the same in de.oddb.org and ramaze oddb, which is an extended version from ch.oddb.org
  • So, the data structure is available for both de.oddb.org and ch.oddb.org even if actually the source data is different
  • But there are subtle differences in the old (oddb.org) and the new (de.oddb.org, ramaze)
  • The 'migrate' script absorbs the difference
  • migration diagram http://trac.ywesee.com/attachment/ticket/16/data_migration_diagram_ch.oddb.org_ruby-1.8_ruby-1.9.txt
    • key word: ODBA::DRbWrapper
    • key point: it is impossible to directly transfer the marshal data, then it converts the data through ODBA::DRbWrapper
    • The migration does loading objects from the old database and re-creating new objects for the new database

Try to run migrate with -E:UTF-8 option

 ruby1.9 -E:UTF-8 bin/migrate 

Result

  • the same error

Note (This is the first I should do)

 The good news is, that I've managed to overcome the irregularities in the DRb connection 
 (by introducing a new method 

 OddbApp#each_remote_company?, which iterates over all companies wrapped in a ODBA::DRbWrapper

 ). Using this, I am now able to cleanly access Company data.

Note

  • But there is not such a method defined in 'oddb.org', so I should define it
  • I should understand how to use ODBA::DRbWrapper class
  • I have so far created 'remote_each_company' method in OddbApp class with 'extended(DRb::DRbUndumped)' for 'call by reference'
  • This may be wrong

Experiment

oddb.org/src/util/oddbapp.rb

    def remote_each_atc_class(&block)
        ODBA::DRbWrapper.new(@atc_classes).each_value(&block)
        #@atc_classes.extend(DRb::DRbUndumped).each_value(&block)
    end

Run oddb.org/bin/oddbd

Run ch.oddb.org/bin/migrate

Result

  • The error is same
  • ODBA::DRbWrapper is basically the same to include DRb::DRbUndumped in the object to be called by reference through drb

Memo

  • Important concept of the migration is changing the data structure (class structure)

From

 +---------+     +--------------+     +----------+     +---------+
 | Company |1---n| Registration |1---n| Sequence |1---n| Package |
 +---------+     +--------------+     +----------+     +---------+

To

        +--------------+        +------+ +------+
        | Registration |        |  FI  | |  PI  |
        +--------------+        +------+ +------+
          n         n                1     1
          |         |                |     |
          1         1                n     n
  +---------+     +---------+     +----------+     +---------+
  | Company |1---n| Product |1---n| Sequence |1---n| Package |
  +---------+     +---------+     +----------+     +---------+
                                        n
                                        |
                                        1
                                    +-------+     +-------+
                                    |  ATC  |1---n|  DDD  |
                                    +-------+     +-------+

About Price

 +-----------+     +-----------+     +----------------+
 |           |     |  Price    |     |  Money         |
 |  Package  |1---n|   - type  |1---n|   - valid_from |
 |           |     |           |     |   - amount     |
 +-----------+     +-----------+     +----------------+

Note

  • exactly: exfactory/pp
  • valid_from: from when is this price valid (independent of the source of the price).
  • amount: yes, the amount, e.g. 12.30.

Check flow of migrate script

  1. migrate_atc: ODDB::Drugs::Atc, ODDB::Drugs::Ddd
    1. migrate_multilingual
  2. migrate_company: ODDB::Business::Company
    1. migrate_product: ODDB::Drugs::Product, ODDB::Regulatory::Registration, ODDB::Regulatory::Authority
      1. migrate_sequence: ODDB::Drugs::Sequence,
        1. migrate_atc: ODDB::Drugs::Atc, ODDB::Drugs::Ddd
        2. migrate_composition: ODDB::Drugs::Composition
          1. migrate_active_agent: ODDB::Drugs::ActiveAgent
        3. migrate_package: ODDB::Drugs::Package, ODDB::Util::Code, ODDB::Business::Price
          1. migrate_part: ODDB::Drugs::Part, ODDB::Drugs::Unit
          2. migrate_migrate_feedback:
        4. migrate_m10l_document: ODDB::Text::Document, ODDB::Text::Chapter, ODDB::Text::Paragraph,

Experiment

oddb.org/stc/util/oddbapp.rb

    def remote_each_atc_class(&block)
        ODBA::DRbWrapper.new(@atc_classes).each_value(&block)
    end
    def remote_each_company(&block)
        ODBA::DRbWrapper.new(@companies).each_value(&block)
    end

Run oddb.org/bin/oddbd

Run ch.oddb.org/bin/migrate (by Ruby 1.9.1)

Result

masa@masa ~/ywesee/ch.oddb.org $ /home/masa/bin/ruby191/bin/ruby bin/migrate 
...
(many errors)
...

Note

  • It took more than five hours
  • I know the grammatical reason of the errors but I have not completely understood the fundamental cause of the errors
  • There is no description about the errors in the ticket#16

Result

Note

  • Looks good
  • There are a little bit differences from online data but they are same to the data in ch.oddb.org online
view · edit · sidebar · attach · print · history
Page last modified on January 29, 2011, at 10:39 AM