Remarked yesterday that we have about 1500 inactive_agents which have their boolean-field is_active_agent as nil. This leads to wrong number of active agents display in entries like http://i.ch.oddb.org/de/mobile/search/zone/drugs/search_query/Perindopril-Amlodipin-Mepha+5mg%252F5mg/search_type/st_oddb#best_result.
Started yesterday the job jobs/cleanup_active_agents at 21.58. It seems stuck. The last lines displayed are
cleanup_active_agents_in_compositions 2016-11-28 22:02:13 +0100: Skip inactive 41173/03 Array Array active? false cleanup_active_agents_in_compositions 2016-11-28 22:02:13 +0100: Start 41174/04. Skip? false cleanup_one_composition 2016-11-28 22:02:13 +0100: 41174/04 1951 nr_updates cleanup_one_composition 2016-11-28 22:02:13 +0100: 41174/04. Done cleanup_one_composition 2016-11-28 22:02:13 +0100: 41174/04 before [1, Array] after 1/4 2016-11-28 22:02:13 +0100: 9436853 timeout status for (41174/04) cleanup_active_agents_in_compositions 2016-11-28 22:02:13 +0100: Start 41174/01. Skip? false cleanup_one_composition 2016-11-28 22:02:13 +0100: 41174/01 1952 nr_updates
Killing the job at 08:05.
I will try to find a solution to get rid of the explicit field @is_active_agent. Looking at the code I could see why the field can be created with a nil value (If one uses ActiveAgentCommon.new). Added the following unit test to avoid this situation
def test_init_is_comon_agent
name = 'Perindopril'
active_agent = ODDB::ActiveAgentCommon.new(name)
assert_equal(false, active_agent.is_active_agent)
end
ActiveAgentCommon is also used in src/model/user.rb (Which I had not expected).
Added a cleanup method in oddbapp.rb for the bin/admin. It finished now cleanly on oddb-ci2. And running it second time returns an empty array in less than a minute (if the packages are loaded).
Added some debug puts to see, whether some methods are called with is_active_agent when displaying the packages.
But my fix is terrible. It deleted the active_agent, instead of the reference to it. Why did the (unchanged) composition.delete_active_agent not work correctly? Or did the jobs/update_active_agents create havoc? Reimporting the database then calling in bin/admin again $res=cleanup_active_agents_all_nil. The search for metoprolol spirig returned many articles without an active agent (e.g. 52110 04 165 Beloc Zok 25 mg).
I must rework the delete as againg bin/admin tells me
registration('52110').sequence('04').package('165').compositions.first.active_agents.size
ch.oddb> registration('52110').sequence('04').package('165').compositions.first.active_agents.size
-> 0
Writing a unit test in test_util/oddbapp_3.rb for cleanup_active_agents_nil made me discover my flaw in the intialization of the active agent. Changed iterator to delete active_sequence starting with the sequence.compositions and not package.compositions.
As I iterate over all sequences (not only the active_sequences) I delete a lot of unused active_agents, e.g.
Deleting wrong_agent odba_isolated_store :!registration,52882!sequence,01!composition,9431651!active_agent,219014. is 219014 Biotinum 2.5 mg
On thinpower bin/admin tells me that
ch.oddb> registration('52882').packages.last.active_agents.first.is_active_agent.class
-> NilClass
Should I force all of these to false or true? I think I better leave them and iterate only over the active_sequences. Somehow they get deleted because the ative_agent.chemical_substance has the same oid as the active_agent.oid whose is_active_agent is nil.
What is the desired output for IKSNR 00288 where I have a composition of
globulina equina (immunisé avec tissu intestinal et pulmonaire, foie, reins, pancréas, vésicule biliaire, peau, ganglions lymphatiques de porcin) 8 mg, propylenglycolum, conserv.:
# from thinpower
ch.oddb> registration('00288').sequences.values.first.compositions.first.active_agents.collect{|a| a.to_s}
-> ["Globulina Equina (Immunisé avec Tissu Intestinal et Pulmonaire", "Foie", "Reins", "Pancreas", "Vésicule Biliaire"]
ch.oddb> registration('00288').sequences.values.first.compositions.first.active_agents.collect{|a| a.is_active_agent}
-> [nil, nil, nil, nil, nil]
# from oddb2xml/calc
<SUBSTANCE_NAME>Globulina Equina (immunisé Avec Tissu Intestinal Et Pulmonaire, Foie, Reins, Pancréas, Vésicule Biliaire, Peau, Ganglions Lymphatiques De Porcin)</SUBSTANCE_NAME>
<IS_ACTIVE_AGENT>false</IS_ACTIVE_AGENT>
<QTY>8</QTY>
<UNIT>mg</UNIT>
</SUBSTANCE>
<SUBSTANCE>
<SUBSTANCE_NAME>Propylenglycolum</SUBSTANCE_NAME>
<IS_ACTIVE_AGENT>false</IS_ACTIVE_AGENT>
</SUBSTANCE>
<SUBSTANCE>
<MORE_INFO>conserv.</MORE_INFO>
<SUBSTANCE_NAME>E 216</SUBSTANCE_NAME>
<IS_ACTIVE_AGENT>false</IS_ACTIVE_AGENT>
</SUBSTANCE>
<SUBSTANCE>
<MORE_INFO>conserv.</MORE_INFO>
<SUBSTANCE_NAME>E 218</SUBSTANCE_NAME>
<IS_ACTIVE_AGENT>false</IS_ACTIVE_AGENT>
currently I have 5 active_agents which have is_active_agent == nil.
After a discussion with Zeno we decided that I should just remove all active_agents which have the is_active_agent == nil and use the Swissmedic importer (or a variant) of it to reparse all composition texts. Also I will replace the is_active_agent accessor by the method active_agent? which will respond using the derived class name (eg. self.is_a?(ActiveAgent)). But I think I will later get rid of the method, as I think it is no longer used anywhere (exept in test_model/ative_agent)
Made some fixes in the swissmedic plugin to delete active_agents with is_active_agent == nil before starting the import. Running. Will analyze the output tomorrow.