Index>

20140611-sax-parser-for-fi-pi-import

Summary

• Use sax-parser to fix problem "Out of memory" for import_daily
• jobs/import_swissmedic fails with undefined method `sequences' for "format_type":String

Commits

(from sax_parser branch)

• https://github.com/ngiger/oddb.org/commit/c5e6be7b7bb804673c321a18a6d301073ffd4e63 Log end of parse_fach|patinfo. Take 6
• https://github.com/ngiger/oddb.org/commit/d42e89bd6ad319a141beb4678e7f430935781f62 Use sax-parser for extract_matched_content. Take 11
• https://github.com/ngiger/oddb.org/commit/d4ba5f023ff368c935ecf6f9e5d0a75274559e3a Use sax-machine to parse Aips-xml. Third try

Index

• Use sax-parser to fix problem "Out of memory" for import_daily
• jobs/import_swissmedic fails with undefined method `sequences' for "format_type":String

Keep in Mind

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
• The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
• One unit-test for searchbar fails and might be a clue why searching does not work correctly.
• Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?

---

Use sax-parser to fix problem "Out of memory" for import_daily

We must now find

• a way to get the IKSNRs of all names found via "Neue Einträge" when running jobs/update_textinfo_swissmedicinfo
• a way to get the IKSNRs of all companies when running jobs/update_textinfo_swissmedicinfo company1 company2

Must also look that import work whether reparse is given or not

• jobs/update_textinfo_swissmedicinfo --no-download --target=both CBI (has20 entries)
• jobs/update_textinfo_swissmedicinfo --no-download --target=both --reparse CBI (has20 entries)
• jobs/update_textinfo_swissmedicinfo --no-download --target=pi 49232
• jobs/update_textinfo_swissmedicinfo --no-download --target=fi 49232
• jobs/update_textinfo_swissmedicinfo --no-download --target=both 49232
• jobs/update_textinfo_swissmedicinfo --no-download --target=both --reparse 49232 (has 5 entries)
• jobs/update_textinfo_swissmedicinfo # should call import_swissmedicinfo_by_index, we have no unit-test for it!!!
• jobs/update_swissmedicno_fi_pi
• jobs/import_daily

I must add unit-tests for these combination in test/test_plugin/text_info_swissmedicinfo.rb

There is a problem when adding --reparse. It seems that the delegation to the DRB does not work correctly. Unfortunately this is mocked in the unit tests.

Saving my work in a separate branch sax_parser. See commits

• Log end of parse_fach|patinfo. Take 6
• Use sax-parser for extract_matched_content. Take 11
• Use sax-machine to parse Aips-xml. Third try

update_swissmedicno_fi_pi seems to detect quite a lot of errors. Will rerun it once my refactoring will be empty.

2014-06-11 17:30:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1300:in `check_swissmedicno_fi_pi': check_swissmedicno_fi_pi 739/775 iksnrs_to_import  are  
00274 00277 00278 00279 00280 00282 00284 00286 00288 00294 00295 00296 00299 00300 00305 00306 00309 00313 00314 00315 00316 00328 00330 00332 00334 00335 00338 00339 00343 00372 00437 00643 00691 11066 1229105863 1229107845 1229107847 1229109223 1229109224 12537 13290 16397 18056 19943 19945 20883 21213 21840 22582 23397 23895 24424 24926 24959 26061 28840 29221 29413 29729 29730 30300 30433 30629 30630 30658 31286 31312 32827 33037 33512 33617 3363895078 34134 34135 34282 34284 34286 34370 34762 34834 34914 35157 35532 35587 35622 35644 35797 35922 35950 36212 36492 36503 36506 36508 36509 36575 36590 36591 36595 36596 36597 36598 36599 36605 36606 36607 36639 36642 36645 36652 36687 36694 36698 36846 36850 36851 36863 37908 38045 38096 38115 38130 38148 38359 38423 38470 38596 39001 39045 39387 39497 39523 39552 39560 39622 39645 39648 39694 40034 40414 40450 40558 40905 41017 41756 42035 42048 42084 42357 42417 42450 42482 42647 42694 43216 43225 43522 43556 43593 43709 43779 43836 43838 43869 43870 43996 44143 44207 44371 44372 44438 44654 44689 44788 44939 45010 45197 45228 45229 45275 45370 45427 45519 45730 45786 45859 45954 45979 46051 46090 46129 46180 46205 46318 46344 46345 46351 46429 46489 46513 46535 46567 46619 46620 46621 46668 46768 46857 46859 46861 46898 47122 47503 47504 47505 47601 47721 47795 47796 47915 48057 48179 48523 48642 48706 48914 48981 49057 49156 49182 49404 49454 49586 49588 49606 49744 49783 49786 49854 50240 50264 50426 50474 50726 50787 51079 51135 51231 51407 51458 51680 51682 51684 51685 51750 51995 52121 52214 52231 52273 52324 52377 52418 52515 52557 52558 52559 52560 52562 52594 52599 52608 52624 52759 52799 52866 53051 53075 53149 53210 53388 53417 53418 53471 53493 53514 53691 53694 53699 53703 53708 53713 53748 53788 53793 53826 53841 53848 53855 53881 53978 53979 54002 54010 54070 54095 54113 54255 54260 54268 54276 54280 54281 54447 54493 54521 54541 54635 54785 54810 54823 54831 54859 54872 54948 55042 55082 55113 55137 55262 55295 55380 55413 55429 55431 55463 55506 55623 55651 55666 55697 55807 55811 55829 55837 55863 55875 55897 55898 55918 56009 56028 56029 56056 56075 56236 56318 56328 56381 56387 56411 56521 56548 56716 56750 56882 56951 57148 57322 57499 57517 57559 57566 57567 57570 57619 57657 57660 57681 57714 57715 57740 57769 57800 57823 57949 57982 57992 57994 58051 58125 58129 58167 58268 58275 58292 58336 58385 58387 58416 58417 58436 58616 58631 58660 58697 58758 58773 58878 58899 58921 59166 59170 59189 59190 59218 59258 59303 59336 59374 59376 59408 59502 60026 60131 60184 60210 60211 60341 60406 60443 60463 60469 60718 60838 60839 60840 60843 60856 60858 60870 60873 60880 60886 60889 60904 60906 60911 60912 61227 61229 61267 61281 61331 61424 61488 61489 61495 61541 61550 61560 61561 61562 61563 61564 61566 61568 61569 61570 61571 61572 61573 61574 61575 61576 61577 61578 61579 61580 61581 61582 61584 61585 61586 61587 61588 61589 61590 61591 61593 61595 61596 61597 61598 61599 61600 61601 61602 61603 61605 61607 61609 61610 61611 61612 61613 61614 61615 61616 61617 61618 61619 61621 61622 61623 61671 61672 61673 61690 61693 61694 61695 61696 61697 61698 61699 61700 61701 61702 61703 61704 61705 61706 61707 61708 61709 61710 61711 61712 61713 61714 61716 61717 61718 61719 61720 61723 61724 61725 61728 61729 61730 61731 61733 61735 61736 61737 61738 61739 61740 61741 61742 61743 61744 61745 61746 61747 61748 61749 61750 61751 61752 61753 61754 61755 61756 61757 61758 61759 61760 61761 61762 61763 61764 61765 61766 61767 61770 61771 61772 61773 61774 61775 61776 61777 61778 61779 61782 61783 61784 61785 61786 61787 61789 61790 61791 61792 61793 61794 61795 61797 61799 61800 61801 61802 61803 61804 61805 61806 61807 61808 61809 61810 61811 61812 61818 61819 61821 61822 61823 61824 61825 61826 61827 61828 61829 61830 61831 61832 61834 61835 61837 61838 61839 61841 61842 61843 61844 61845 61846 61847 61860 61886 61901 61907 61908 61910 61911 61916 61921 61922 61923 61924 61926 61927 61928 61931 61932 61933 61934 61935 61937 61938 61939 61940 61941 61942 61956 61966 61972 61974 61977 61981 61985 61991 61992 61993 61998 62000 62003 62004 62005 62006 62007 62008 62009 62011 62012 62014 62015 62016 62017 62018 62021 62024 62025 62026 62027 62028 62029 62030 62032 62059 62062 62067 62141 62219 62228 62261 62391 62502 62569 62645 62714 62726 62727 62755 62771 62783 62930 62936 62954 62961 63007 63092 63123 63184 65032 65155

jobs/import_swissmedic fails with undefined method `sequences' for "format_type":String

Defaulting to UTF-8, set your own if this is incorrect.
/var/www/oddb.org/log/job.pid is deleted
(druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:119:in `comparables': undefined method `sequences' for "format_type":String (NoMethodError)
        from (druby://localhost:10005) /var/www/oddb.org/src/model/package.rb:214:in `comparables'
        from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:96:in `block in export_generics'
        from (druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each'
        from (druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each_package'
        from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
        from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:149:in `block in each_package'
        from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_value'
        from (druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_package'
        from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
        from (druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:664:in `block in each_package'
        from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `each_value'
        from (druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
        from (druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:663:in `each_package'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:94:in `export_generics'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:113:in `block in export_generics_xls'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `call'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `block in safe_export'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/tempfile.rb:316:in `open'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:285:in `safe_export'
        from (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:111:in `export_generics_xls'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
        from (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
        from /var/www/oddb.org/src/plugin/xls_export.rb:22:in `export_generics'
        from /var/www/oddb.org/src/util/exporter.rb:154:in `export_generics_xls'
        from /var/www/oddb.org/src/util/updater.rb:416:in `update_swissmedic_followers'
        from jobs/import_swissmedic:15:in `block in <module:Util>'
        from /var/www/oddb.org/src/util/job.rb:40:in `call'
        from /var/www/oddb.org/src/util/job.rb:40:in `run'
        from jobs/import_swissmedic:12:in `<module:Util>'
        from jobs/import_swissmedic:11:in `<module:ODDB>'
        from jobs/import_swissmedic:10:in `<main>'

Looking at log/oddb/debug/2014/06.log I see, that the swissmedic_import did correctly find the latest Packungen, e.g

2014-06-11 09:29:27 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 276 updated download.size is 2672682./var/www/oddb.org/data/xls/Packungen-2014.06.11.xlsx now 2672682 bytes != /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2674915
2014-06-11 09:29:27 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 53 update target "/var/www/oddb.org/data/xls/Packungen-2014.06.11.xlsx"2672682 bytes. Latest /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2674915 bytes  
2014-06-11 09:29:28 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 281 skip writing /var/www/oddb.org/data/xls/Präparateliste-2014.06.11.xlsx as /var/www/oddb.org/data/xls/Präparateliste-latest.xlsx is 2217380 bytes. Returning latest  
2014-06-11 09:37:32 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 67 Found 86 news, 512 updates, 17 replacements and 63 package_deletions  
2014-06-11 09:37:32 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 68 changes: {"00613"=>[:composition], "08909"=>[:production_science], "08911"=>[:production_science, :expiry_date], "10077"=>[:expiry_date

Similar lines were found two hours earlier when running import_daily

2014-06-11 07:19:31 CESTlog notify Fach- und Patienteninfo Updates (swissmedicinfo.ch): sent mail
2014-06-11 07:19:32 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 276 updated download.size is 2672682./var/www/oddb.org/data/xls/Packungen-2014.06.11.xlsx now 2672682 bytes != /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2674915
2014-06-11 07:19:32 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 53 update target "/var/www/oddb.org/data/xls/Packungen-2014.06.11.xlsx"2672682 bytes. Latest /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2674915 bytes
2014-06-11 07:19:32 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 281 skip writing /var/www/oddb.org/data/xls/Präparateliste-2014.06.11.xlsx as /var/www/oddb.org/data/xls/Präparateliste-latest.xlsx is 2217380 bytes. Returning latest
2014-06-11 07:28:39 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 67 Found 86 news, 512 updates, 17 replacements and 63 package_deletions
<...>
2014-06-11 08:04:10 +0200: update_compositions: row[0] 63177 iksnr 63177 02 seq Abilify Maintena 400 mg, Depot-Injektionslösung opts {:create_only=>false, :date=>#<Date: 2014-06-11 ((2456820j,0s,0n),+0s,2299161j)>, :composition=>0, :label=>nil} cell_content praeparatio sicca: aripiprazolum monohydricum 475 mg corresp. aripiprazolum 400 mg, carmellosum natricum, mannitolum, natrii dihydrogenophosphas monohydricus, pro vitro.

No idea however why the import_daily stopped without signalling any error around 08:04:10 this morning. We should run once import_daily in a separate screen to see whether it will crash again or not. Will import thinpowers database as of yesterday to my oddb-ci2 and relaunch import_daily there.