20110830-debug-memory-leak-migel-overwriting-original-generic-flag-oddb_org
<< | Index | >>
- Import migel data in both 'de' and 'fr'
- Overwriting Original/Generic flag
- Fix View
- Testcases oddb.org Original/Generic flag process
Goal/Estimate/Evaluation:
- Debug memory leak migel import / 100% / 90%
- Overwriting original generic flag / 80% / 98%
- Milestones
Debug memory leak migel importOriginal Generic flag- Update Migel search function
- Testcases Generic flag oddb.org
- Testcases migel
- Summary
- Commits
- Added Overwrite BAG-XML checkbox in the view of registration (oddb.org)
- Updated the testcases for bsv_xml.rb (oddb.org)
- ToDo
- bbmb.ch debug
- Refactor Group, Subgroup, Migelid because some code in the classes are totally same
- Check Group and Subgroup de and fr view
- Testcases oddb.org,
swissindex.rb, migel
Import migel data in both 'de' and 'fr'
Hypothesis
- migel_fr.csv may influence on the memory leak
Experiment
- Restore database
masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel masa@masa ~/ywesee/migel $ bin/migeld
migel> Migel::Importer.new.update('data/csv/migel_de.csv', 'de')
migel> Migel::Importer.new.update_all_products('de', true)
Result
- 0.9% memory is used after 7 minuts
Note
- update takes about 20 minutes
- Attach:plot_estimate_time.rb.20110830.txt
Experiment
- Import fr data
masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel masa@masa ~/ywesee/migel $ bin/migeld
migel> Migel::Importer.new.update('data/csv/migel_fr.csv', 'fr')
migel> Migel::Importer.new.update_all_products('fr', true)
Result
Note
- It looks same as de import
Experiment
- Import both de and fr
masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel masa@masa ~/ywesee/migel $ bin/migeld
migel> Migel::Importer.new.update('data/csv/migel_de.csv', 'de')
migel> Migel::Importer.new.update('data/csv/migel_fr.csv', 'fr')
migel> Migel::Importer.new.update_all_products('de', true)
Note
- It is getting longer and longer in the later data to be updated
- It takes over 2 hours
- The memory footprint is increasing before migel product iteration process
Hypothesis
- Migel::Model::Migelid.all method may be the cause
Experiment
migel> Migel::Model::Migelid.all.length
Result
Experiment
migel> ODBA.cache.index_keys('migel_model_migelid_migel_code').length
-> 571
migel> open("/home/masa/work/test.dat","w"){|f| f.print ODBA.cache.index_keys('migel_model_migelid_migel_code').join("\n")}
Result
masa@masa ~/work $ head test.dat 01.01.01.00.1 01.01.02.00.2 01.01.02.01.2 01.01.02.02.2 01.02.01.00.2 01.02.01.01.1 03.01.01.00.1 03.01.02.00.1 03.02.01.00.2 03.02.01.01.2 ...
Note
- much faster to output all migel code
Experiment
- lib/migel/importer.rb
def update_all_products(lang = 'DE', estimate = false)
lang.upcase!
start_time = Time.now
index_table_name = 'migel_model_migelid_migel_code'
#total = Migel::Model::Migelid.all.length
migel_codes = ODBA.cache.index_keys('migel_model_migelid_migel_code')
total = migel_codes.length
count = 0
#Migel::Model::Migelid.all.each do |migelid|
migel_codes.each do |migel_code|
update_products_by_migel_code(migel_code, lang)
count += 1
puts estimate_time(start_time, total, count) if estimate
end
end
Run
migel> Migel::Importer.new.update_all_products('de', true)
Result
- Let's see tomorrow
Overwriting Original/Generic flag
Note
- There are two generic_type variables,
- Registration@generic_type
- Package@sl_generic_type
Experiment
- src/model/registration.rb
attr_accessor
...
:keep_generic_type
- src/view/admin/registration.rb
...
class RegistrationForm < View::Form
COMPONENTS = {
#[0,2] => :generic_type,
[0,2,0] => :generic_type,
[1,2] => 'keep',
[0,2,1] => :keep_generic_type,
...
CSS_MAP = {
[1,2] => 'list bold',
- src/util/validator.rb
BOOLEAN = [
...
:keep_generic_type
- src/state/admin/registration.rb
def update keys = [ ... :keep_generic_type
Result
ch.oddb> registration('61848').keep_generic_type.class
-> TrueClass
Next
- Update process (bsv.rb)
- src/plugin/bsv.rb
when 'OrgGenCode'
gtype = GENERIC_TYPES[@text]
@reg_data.store(:generic_type, (gtype || :unknown)) unless @registration.keep_generic_type
Run
- Updater.new(self).update_bsv
Result
Plugin: ODDB::BsvXmlPlugin
Error: NoMethodError
Message: undefined method `keep_generic_type' for nil:NilClass
@report: {:pharmacode_oddb=>"3365788", :swissmedic_no5_oddb=>"57363", :pharmacode_bag=>"3365788", :deductible=>:deductible_g, :name_base=>"Aclasta", :atc_class=>"M05BA08", :name_descr=>"Inf L\303\266s 5 mg/100ml ", :swissmedic_no5_bag=>"57363", :generic_type=>:unknown, :swissmedic_no8_bag=>"57363001"}
Backtrace:
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:475:in `tag_end'
/usr/lib64/ruby/1.8/rexml/parsers/streamparser.rb:26:in `parse'
/usr/lib64/ruby/1.8/rexml/document.rb:201:in `parse_stream'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:964:in `update_preparations'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `send'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `_update'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:787:in `get_input_stream'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `_update'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1154:in `each'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1154:in `each'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1297:in `each'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1434:in `foreach'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1413:in `open'
/usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1432:in `foreach'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:636:in `_update'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:631:in `update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:251:in `update_bsv'
/home/masa/ywesee/oddb.org/src/util/updater.rb:476:in `call'
/home/masa/ywesee/oddb.org/src/util/updater.rb:476:in `wrap_update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:249:in `update_bsv'
(eval):1:in `_admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1468:in `_admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1466:in `initialize'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1466:in `new'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1466:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
bin/oddbd:38
- src/plugin/bsv_xml.rb
when 'OrgGenCode'
gtype = GENERIC_TYPES[@text]
unless (registration = @app.registration(@iksnr) and registration.keep_generic_type)
@reg_data.store(:generic_type, (gtype || :unknown))
end
Run
- Updater.new(self).update_bsv
Fix View
- src/view/admin/registration.rb
COMPONENTS = {
...
[0,3] => :keep_generic_type,
[2,2] => :expiration_date,
[2,3] => :renewal_flag,
[0,4,0] => :complementary_select,
[0,4,1] => :complementary_type,
[2,4] => :market_date,
[0,5] => :index_therapeuticus,
[2,5] => :manual_inactive_date,
[0,6] => :ith_swissmedic,
[2,6] => :inactive_date,
[0,7] => :indication,
[2,7] => :patented_until,
[0,8] => :export_flag,
[2,8] => :ignore_patent,
[3,8] => :violates_patent,
[0,9] => :parallel_import,
[2,9] => :vaccine,
...
COMPONENT_CSS_MAP = {
[1,0,1,8] => 'standard',
[3,0,1,8] => 'standard',
}
CSS_MAP = {
[0,0,6,10] => 'list',
[0,9] => 'list',
}
COLSPAN_MAP = { [3,7] => 3 }
def reorganize_components
if(@model.is_a?(Persistence::CreateItem))
components.store([1,10], :submit)
css_map.store([1,10], 'list')
else
components.update({
[0,10] => 'fi_upload_instruction0',
[2,10] => :fachinfo_label,
[3,10,0] => :fachinfo,
[3,10,1] => :assign_fachinfo,
[0,11] => 'fi_upload_instruction1',
[1,11] => :language_select,
[2,11] => :textinfo_update,
[0,12] => 'fi_upload_instruction2',
[1,12] => :fachinfo_upload,
[2,12] => :activate_fachinfo,
[0,13] => 'fi_upload_instruction3',
[1,13,0]=> :submit,
[1,13,1]=> :new_registration,
[2,13] => :deactivate_fachinfo,
})
colspan_map.store([3,10], 3)
colspan_map.store([0,10], 2)
css_map.store([0,10], 'list bg bold')
css_map.store([1,10], 'list bg')
css_map.store([2,10,2,4], 'list')
css_map.store([0,11,2,3], 'list bg')
component_css_map.store [3,12,1,2], 'standard'
Before
After
Commit
Testcases oddb.org Original/Generic flag process
Commit
masa@masa ~/ywesee/oddb.org/test/test_plugin $ ruby bsv_xml.rb /usr/lib64/ruby/gems/1.8/gems/savon-0.8.6/lib/savon/soap/xml.rb:154: warning: parenthesize argument(s) for future version /usr/lib64/ruby/gems/1.8/gems/httpi-0.9.0/lib/httpi/response.rb:71: warning: parenthesize argument(s) for future version bsv_xml.rb:218: warning: already initialized constant MEDDATA_SERVER Loaded suite bsv_xml Started ............................................................................ Finished in 0.240469 seconds. 76 tests, 156 assertions, 0 failures, 0 errors