view · edit · sidebar · attach · print · history

20100908 test import

<< Masa.20100909-job-import_gkv | 2010 | Masa.20100907-install-ruby >>

Gkv
test_import
Review ticket 242

Goal

test_import / 80%

Milestones

test_import one by one
1. ~~invalidator.size failure~~ 8:45
2. ~~assert_instance_of(Array, report)~~ pass
3. ~~assert_equal(1, Drugs::Product.instances.size)~~ 15:50
4. ~~assert_equal(1, confirmed.size)~~ 16:10
~~Commit~~ 16:10
~~Reading~~
1. ~~setup~~
2. ~~simulate_import~~

Summary

http://masay.ch/2010/09/08/test-import/

Commits

Corrected test_gkv.rb (test_import, test_import__ml)

ToDo Tomorrow

20100907 Ticket 242, Comment 9 - notification Mail
Dimdi-Import-Error.

Keep in Mind

20100907 Ticket 242, Comment 9 - notification Mail
20100908 rpdf2txt GPLv3
setup script for de.oddb.org

Attached Files

Cannot switch ruby19

masa@masa /etc/portage $ sudo eselect ruby set ruby19
Passwort: 
!!! Error: Can't use that profile. No suitable Ruby interpreter found.
exiting

I do not know why.

suspend

I did it but got some warning about gem.

masa@masa ~/work/de.oddb.org $ sudo eselect ruby set ruby1.9
Passwort: 
!!! Warning: Could not set gem symlink
It appears you do not have RubyGems installed for this profile.
If you need RubyGems, emerge dev-ruby/rubygems with the appropriate RUBY_TARGETS setting.

Successfully switched to profile:
  ruby1.9                   

masa@masa ~/work/de.oddb.org $ sudo eselect ruby set ruby18
Successfully switched to profile:
  ruby18

Gkv

Gkv stands for "gesetzliche Krankenversicherung"
Gesetzliche Gesundheitsversicherung means "Health Insurance by Law"

test_import

assert_equal 2, @invalidator.invalidated.size

git checkout fc894bb1f561ffe9807cc5ad45bc39cef4f024e4
check @invalidator.size
@system.ivalidate obj.uid in lib/oddb/import/gkv.rb counts up invalidate length

  def save(obj)
    obj.save
    @system.invalidate obj.uid
p @system.invalidate.size
  end

This @system.invalidate obj.uid statement is deteled in 1b0624b8fc4aa8985b5013c3d2e942bd98b3426f
So, I guess Class invalidator in test_gkv.rb is not necessary.

assert_equal(1, Drugs::Product.instances.size)

Confirm failure

masa@masa ~/work/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
F
Finished in 0.019584 seconds.

  1) Failure:
test_import(ODDB::Import::TestGkv) [test/import/test_gkv.rb:115]:
<1> expected but was
<3>.

1 tests, 3 assertions, 1 failures, 0 errors

Reading setup method

  def simulate_import
    handler = GkvHandler.new @import.method(:process_page)
    File.read(@path).each do |line|
      handler.send_flowing_data line
      handler.send_line_break
    end
    handler.send_page
    @import.postprocess
    @import.report
  end

Comment

@path == "/home/masa/work/de.oddb.org/test/import/data/txt/gkv/gkv_p1.txt"
Drugs::Product.instances is a class variable?
- it is defined in test/stub/model.rb
@import.class == Gkv

/home/masa/work/de.oddb.org/test/import/data/txt/gkv/gkv_p1.txt

Zuzahlungsbefreite Arzneimittel nach � 31 Abs. 3 Satz 4 SGB V
sortiert nach Arzneimittelname
Produktstand   01.09.2009
grö�<9f>e                                                                                                                                                                                                                                                                                                                                                         inkl.MwSt
Wirkstoff(e)                    Wirkstärke               Packungs-)n(                                                                    Arzneimittelname                          PZN           Hersteller                                                                                                                                 Darreichungsform                Apothekenverkaufspreis
ABSEAMED 10000I.E./1ML                4000741      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                             10000   I.E.              6X1     ml           Fertigspritzen                                            611,53
ABSEAMED 1000I.E./0.5ML               4000646      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               1000   I.E.           6X0.5     ml           Fertigspritzen                                             64,20
ABSEAMED 2000I.E./1ML                  4000652      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               2000   I.E.              6X1     ml           Fertigspritzen                                            119,04
ABSEAMED 3000I.E./0.3ML               4000669      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               3000   I.E.           6X0.3     ml           Fertigspritzen                                            173,94
ABSEAMED 4000I.E./0.4ML               4000681      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               4000   I.E.           6X0.4     ml           Fertigspritzen                                            228,83
ABSEAMED 5000I.E./0.5ML               4000698      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               5000   I.E.           6X0.5     ml           Fertigspritzen                                            283,70
ABSEAMED 6000I.E./0.6ML               4000729      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               6000   I.E.           6X0.6     ml           Fertigspritzen                                            338,57
ABSEAMED 8000I.E./0.8ML               4000735      MEDICE ARZN.GMBH&CO.KG    Epoetin alfa                               8000   I.E.           6X0.8     ml           Fertigspritzen                                            448,34
ACC 200                                              3867219      HEXAL AG                                     Acetylcystein                               200   mg                 50     St            Brausetabletten                                          12,74
ACC 200                                              3867225      HEXAL AG                                     Acetylcystein                               200   mg               100     St            Brausetabletten                                          15,42
ACC 200                                              4789763      HEXAL AG                                     Acetylcystein                               200   mg                 20     St            Brausetabletten                                          11,01
Seite 1 von 1095

Check Drugs::Product.instances.size test/import/test_gkv.rb

  def simulate_import
p  Drugs::Product.instances.size
    handler = GkvHandler.new @import.method(:process_page)
    File.read(@path).each do |line|
      handler.send_flowing_data line
      handler.send_line_break
    end
p  Drugs::Product.instances.size
    handler.send_page
p  Drugs::Product.instances.size
    @import.postprocess
    @import.report
  end

  def test_import
    existing = Drugs::Package.new
    existing.add_code(Util::Code.new(:cid, '4000741', 'DE'))
    existing.add_part(Drugs::Part.new)
    existing.save
    sequence = Drugs::Sequence.new
    product = Drugs::Product.new
    product.name.de = 'A product'
    existing.sequence = sequence
p  Drugs::Product.instances.size
    sequence.product = product
p  Drugs::Product.instances.size
    assert_nil(existing.code(:zuzahlungsbefreit))
puts
    ## simulate a call to @import.import
    report = simulate_import

Result

masa@masa ~/work/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
0
1

1
1
3
F
Finished in 0.01983 seconds.

  1) Failure:
test_import(ODDB::Import::TestGkv) [test/import/test_gkv.rb:120]:
<1> expected but was
<3>.

1 tests, 3 assertions, 1 failures, 0 errors

This means

sequence.product = product: Drugs::Product.instances +1
handler.send_page: Drugs::Product.instances +2

Hypothesis

simulate_import method was added after some change

Check commit that counts up Drugs::Product.instances in simulate_import method

First commit of import_gkv.rb
/home/masa/work/de.oddb.org/test/import/data/txt/gkv/gkv_p1.txt is the same
There is the simulate_import method but no counting up Drugs::Product.instances in simulate_import method

From this commit 4adb046616d3cf37ac9644ad2cb26d6c29801a63 2009-09-07 Hannes Wyss Import unknown packages, even if data quality cannot.. simulate_import method counts up Drugs::Product.instances.size

Look at diff and added (+ green) lines carefully http://scm.ywesee.com/?p=de.oddb.org/.git;a=blobdiff;f=lib/oddb/import/gkv.rb;h=d14707b3f0f17937c3ee440a3d2e95eca3f994be;hp=9404dfec1eac950c7a03c0a5af7eef205e663937;hb=4adb046616d3cf37ac9644ad2cb26d6c29801a63;hpb=fc894bb1f561ffe9807cc5ad45bc39cef4f024e4

Check report

First commit 8da2441beead2c66bf2d8f887ce0100a00494c18 2009-09-04 Hannes Wyss Import Zuzahlungs-Befreiung from the GKV-PDF

 "Imported  11 FB-Entries on 08.09.2010:",
 "Visited    0 existing FB-Entries",
 "Visited    0 existing Companies",
 "Visited    0 existing Substances",
 "Created    1 new FB-Entries",
 "Assigned   0 Companies",
 "Ignored   10 Unknown Packages"

Fail commit 4adb046616d3cf37ac9644ad2cb26d6c29801a63 2009-09-07 Hannes Wyss Import unknown packages, even if data quality cannot...

 "Imported    11 Zubef-Entries on 08.09.2010:",
 "Visited      0 existing Zubef-Entries",
 "Visited      9 existing Companies",
 "Visited      9 existing Substances",
 "Created     11 new Zubef-Entries",
 "Created      2 new Products",
 "Created     10 new Sequences",
 "Created      2 new Companies",
 "Created      2 new Substances",
 "Assigned     0 Chemical Equivalences",
 "Assigned     0 Companies",
 "Created      1 Incomplete Packages:",
 "http://de.oddb.org/de/drugs/package/pzn/4789763"

Comment

Why is the Drugs::Product.instances.size different?

Tag set

first: First commit 8da2441beead2c66bf2d8f887ce0100a00494c18 2009-09-04 Hannes Wyss Import Zuzahlungs-Befreiung from the GKV-PDF
fail: Fail commit 4adb046616d3cf37ac9644ad2cb26d6c29801a63 2009-09-07 Hannes Wyss Import unknown packages, even if data quality cannot...

Check simulate_method carefully

first: lib/oddb/import/gkv.rb

  def send_page
p Drugs::Product.instances.size
    @callback.call @rows
p Drugs::Product.instances.size

Result

masa@masa ~/work/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
1
1
.
Finished in 0.002832 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

fail: lib/oddb/import/gkv.rb

  def send_page
p Drugs::Product.instances.size
    @callback.call @rows
p Drugs::Product.instances.size

Result

masa@masa ~/work/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
1
3
.
Finished in 0.026326 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

Consideration

simulate_import -> hander(GkvHandler).send_page -> @callback(Method).call @row
where is the next process.
@callback.call == Gkv#process_page
- See sample below

def test
  p "w"
end

a = method(:test)
a.call             #=> "w"

Gkv#process_page

  def process_page rows
    rows.each do |row|
      import_row row
    end
  end

Consideration

when is rows (@rows) set?
1. Gkv#send_line_break
2. data (@current_line) comes from send_flowing_data
Gkv#send_flowing_data and Gkv#send_line_break are called from simulate_import method

Gkv#send_line_break

  def send_flowing_data(data)
    @current_line << data
  end
  def send_line_break
    data = @current_line.strip.split /\s{3,}/
    if /^\d{6,}$/.match(data[1].to_s)
      ## ensure consistent row-length, so we can append additional substances
      #  to the tail
      data[9] ||= nil
      @rows.push data
    else
      @rows.push :doubtful
    end
    reset
  end

def simulate_import

  def simulate_import
    handler = GkvHandler.new @import.method(:process_page)
    File.read(@path).each do |line|
      handler.send_flowing_data line
      handler.send_line_break
    end
    handler.send_page

@rows (after send_flowing_data and send_line_break methods)

@rows.size=17
[:doubtful,
 :doubtful,
 :doubtful,
 :doubtful,
 :doubtful,
 ["ABSEAMED 10000I.E./1ML",
  "4000741",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "10000",
  "I.E.",
  "6X1",
  "ml",
  "Fertigspritzen",
  "611,53"],
 ["ABSEAMED 1000I.E./0.5ML",
  "4000646",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "1000",
  "I.E.",
  "6X0.5",
  "ml",
  "Fertigspritzen",
  "64,20"],
 ["ABSEAMED 2000I.E./1ML",
  "4000652",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "2000",
  "I.E.",
  "6X1",
  "ml",
  "Fertigspritzen",
  "119,04"],
 ["ABSEAMED 3000I.E./0.3ML",
  "4000669",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "3000",
  "I.E.",
  "6X0.3",
  "ml",
  "Fertigspritzen",
  "173,94"],
 ["ABSEAMED 4000I.E./0.4ML",
  "4000681",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "4000",
  "I.E.",
  "6X0.4",
  "ml",
  "Fertigspritzen",
  "228,83"],
 ["ABSEAMED 5000I.E./0.5ML",
  "4000698",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "5000",
  "I.E.",
  "6X0.5",
  "ml",
  "Fertigspritzen",
  "283,70"],
 ["ABSEAMED 6000I.E./0.6ML",
  "4000729",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "6000",
  "I.E.",
  "6X0.6",
  "ml",
  "Fertigspritzen",
  "338,57"],
 ["ABSEAMED 8000I.E./0.8ML",
  "4000735",
  "MEDICE ARZN.GMBH&CO.KG",
  "Epoetin alfa",
  "8000",
  "I.E.",
  "6X0.8",
  "ml",
  "Fertigspritzen",
  "448,34"],
 ["ACC 200",
  "3867219",
  "HEXAL AG",
  "Acetylcystein",
  "200",
  "mg",
  "50",
  "St",
  "Brausetabletten",
  "12,74"],
 ["ACC 200",
  "3867225",
  "HEXAL AG",
  "Acetylcystein",
  "200",
  "mg",
  "100",
  "St",
  "Brausetabletten",
  "15,42"],
 ["ACC 200",
  "4789763",
  "HEXAL AG",
  "Acetylcystein",
  "200",
  "mg",
  "20",
  "St",
  "Brausetabletten",
  "11,01"],
 :doubtful]

import_row method extract specific data from this @rows in process_page method

lib/oddb/gkv.rb

process_page

  def process_page rows
    rows.each do |row|
      import_row row
    end
  end

import_row (too long!! need refactoring!)

xxx

Experiment

  def process_page rows
p "get-in process_page"
previous = Drugs::Product.instances.size
current  = previous
    rows.each_with_index do |row,i|
      import_row row
print i, " Drug::Product.instance.size=",Drugs::Product.instances.size,"\n"
current = Drugs::Product.instances.size
if previous < current
  pp row
end
previous = current
    end
  end

Result

"get-in process_page"
0 Drug::Product.instance.size=1
1 Drug::Product.instance.size=1
2 Drug::Product.instance.size=1
3 Drug::Product.instance.size=1
4 Drug::Product.instance.size=1
5 Drug::Product.instance.size=1
6 Drug::Product.instance.size=2
["ABSEAMED 1000I.E./0.5ML",
 "4000646",
 "MEDICE ARZN.GMBH&CO.KG",
 "Epoetin alfa",
 "1000",
 "I.E.",
 "6X0.5",
 "ml",
 "Fertigspritzen",
 "64,20"]
7 Drug::Product.instance.size=2
8 Drug::Product.instance.size=2
9 Drug::Product.instance.size=2
10 Drug::Product.instance.size=2
11 Drug::Product.instance.size=2
12 Drug::Product.instance.size=2
13 Drug::Product.instance.size=3
["ACC 200",
 "3867219",
 "HEXAL AG",
 "Acetylcystein",
 "200",
 "mg",
 "50",
 "St",
 "Brausetabletten",
 "12,74"]
14 Drug::Product.instance.size=3
15 Drug::Product.instance.size=3
16 Drug::Product.instance.size=3

Point

I have to understand why these two row data is saved in this commit
and why these data is NOT saved in the first commit
what is the difference in (probably) in import_row method

Hypothesis: 4adb046616d3cf37ac9644ad2cb26d6c29801a63 2009-09-07 Hannes Wyss Import unknown packages, even if data quality cannot...

-    if((company = import_company(row)) && product && product.company != company)
+    if(product && (company = import_company(row)) && product.company != company)

Answer

  def import_row(row)
    package = import_package(row)

import_package method looks counting up Drugs::Product.instances.size

import_package

  def import_package(row)
    pzn = row.at(1).to_i
    return if(pzn == 0)
    pzn = u(pzn.to_s)
    package = Drugs::Package.find_by_code(:type => 'cid',
                                          :value => pzn,
                                          :country => 'DE')
    @created_pzn = nil
    if(package.nil?)
      @created_pzn = pzn
      package = Drugs::Package.new
      package.add_code(Util::Code.new(:cid, pzn, 'DE'))
      part = Drugs::Part.new
      part.package = package
      save part
      product = import_product(package, row)

import_product method looks counting up Drugs::Product.instances.size

  def import_product(package, row)
    name = product_name(row)
    search = name.dup
    product = nil
    candidates = []
    until(product || search.empty? || candidates.size > 1)
      candidates = Drugs::Product.search_by_name(search)
      if(candidates.size == 1)
        product = candidates.first
        product.name.add_synonym(name) && save(product)
      end
      search.sub!(/(\s|^)\S*$/, '')
    end
    if(product.nil?)
      @created_products += 1
      product = Drugs::Product.new
      product.name.de = name
      save product
    end
    product
  end

'save product' looks counting up Drugs::Product.instances.size

suspend

Correct test case

test/import/test_gkv.rb

test_import

assert_equal(3, Drugs::Product.instances.size) corrected
assert_equal(true, Drugs::Product.instances.include?(product)) pass
assert_equal(1, product.sequences.size) pass
assert_instance_of(Util::Code, code) pass
assert_equal(true, code.value) pass
assert_equal(1, confirmed.size) fail

assert_equal(1, confirmed.size)

Confirm failure

masa@masa ~/ywesee/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
.FF.....
Finished in 0.047509 seconds.

  1) Failure:
test_import(ODDB::Import::TestGkv) [test/import/test_gkv.rb:130]:
<1> expected but was
<11>.

  2) Failure:
test_import__ml(ODDB::Import::TestGkv) [test/import/test_gkv.rb:181]:
<1> expected but was
<3>.

8 tests, 20 assertions, 2 failures, 0 errors

I have to understand @confirmed_pzns

Backtrace @confirmed_pzns

test/import/test_gkv.rb

test_import

confirmed = @import.instance_variable_get('@confirmed_pzns')
assert_equal(1, confirmed.size)

Check lib/oddb/import/gkv.rb

@confirmed_pzns is hash
@confirmed_pzns is stored in import_row method

  def import_row(row)
    if row == :doubtful
      @doubtful_pzns.push @created_pzn if @created_pzn
      @created_pzn = nil
      return
    end
    @count += 1
    package = import_package(row)
    return if package.nil?
    @confirmed_pzns.store(package.code(:cid).value, true)

the row data other than :doubtful is stored

Consideration

the total size of @rows is 17
the number of :doubtful is 6
so it should be 11 good
there is :doubtful in the last of @rows list
but actually I do not perfectly understand @confirm_pzns meaning

Correct

assert_equal(11, confirmed.size) corrected

Confirm all the test

masa@masa ~/ywesee/de.oddb.org $ ruby test/import/test_gkv.rb 
Loaded suite test/import/test_gkv
Started
........
Finished in 0.089895 seconds.

8 tests, 43 assertions, 0 failures, 0 errors

Review ticket 242

Refer

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20100908 test import

Cannot switch ruby19

Gkv

test_import

assert_equal 2, @invalidator.invalidated.size

assert_equal(1, Drugs::Product.instances.size)

Reading setup method

Correct test case

assert_equal(1, confirmed.size)

Review ticket 242

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20100908 test import

Cannot switch ruby19

Gkv

test_import

assert_equal 2, @invalidator.invalidated.size

assert_equal(1, Drugs::Product.instances.size)

Reading setup method

Correct test case

assert_equal(1, confirmed.size)

Review ticket 242

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler