(Branch medreg)
---
Analysing why I did not rescue correctly when importing doctors. Looked for nr-retries 3 in oddb.log and found
2014-10-29 22:27:50 CET MedregDoctorPlugin store_doctor 7601000041111 update in database. pointer :!doctor,6733.. Have now 34794 doctors. hash {:ean13=>"7601000041111", :exam=>0, :firstname=>"
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041173". nr_retries 3
<..>
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041210". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041227". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041234". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041241". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041258". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041302". nr_retries 3
2014-10-29 22:28:01 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041319". nr_retries 3
2014-10-29 22:28:02 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041401". nr_retries 3
2014-10-29 22:28:02 CET MedregDoctorPlugin rescue Mechanize::ResponseCodeError "7601000041425". nr_retries 3
<..>
2014-10-29 22:35:05 CET MedregDoctorPlugin {:ean13=>"7601000120076", :name=>"Iff", :firstname=>"Ernst", :exam=>"Ärztin/Arzt", :specialities=>["Allgemeine Innere Medizin,1996,Schweiz", "Endokri
2014-10-29 22:35:06 CET MedregDoctorPlugin store_doctor 7601000120076 update in database. pointer :!doctor,14480.. Have now 34794 doctors. hash {:ean13=>"7601000120076", :exam=>0, :firstname=>"
which means, that after 7 minutes the server was ready again. Also I did not find any retry before or after this lapse. Therefore I think it will be sufficient to increase the sleep to 10 minutes and abort the import if I read 3 retries (in this case I retried the next gln, which means that I missed 7.
Pushed commits Made oddbapp doctors_by_gln more robust and Sleep 10 minutes between retries. Really abort if server fails
As soon as my other job finished, I will reload the dump from thinpower.
Medreg-Doctor import failed after 842 imports. When looking by hand at the URL http://www.medregom.admin.ch/ I got
Server Error in '/' Application. Exception of type 'System.OutOfMemoryException' was thrown. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace: [OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.] System.Runtime.CompilerServices.RuntimeHelpers._CompileMethod(IntPtr method) +0 System.Reflection.Emit.DynamicMethod.CreateDelegate(Type delegateType, Object target) +28 System.Linq.Expressions.ExpressionCompiler.CompileDynamicLambda(LambdaExpression lambda) +192 System.Linq.Expressions.LambdaExpression.Compile() +34 System.Data.Linq.SqlClient.Localizer.MakeLocal(Expression e) +242 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +41 System.Data.Linq.SqlClient.ExpressionVisitor.VisitBinary(BinaryExpression b) +33 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +116 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.ExpressionVisitor.VisitBinary(BinaryExpression b) +20 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +116 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.ExpressionVisitor.VisitLambda(LambdaExpression lambda) +13 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +403 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.ExpressionVisitor.VisitUnary(UnaryExpression u) +13 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +75 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.ExpressionVisitor.VisitExpressionList(ReadOnlyCollection`1 original) +70 System.Data.Linq.SqlClient.ExpressionVisitor.VisitMethodCall(MethodCallExpression m) +31 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +362 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.ExpressionVisitor.VisitExpressionList(ReadOnlyCollection`1 original) +70 System.Data.Linq.SqlClient.ExpressionVisitor.VisitMethodCall(MethodCallExpression m) +31 System.Data.Linq.SqlClient.ExpressionVisitor.Visit(Expression exp) +362 System.Data.Linq.SqlClient.Localizer.Visit(Expression exp) +69 System.Data.Linq.SqlClient.Funcletizer.Funcletize(Expression expression) +57 System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Execute(Expression query) +64 System.Data.Linq.DataQuery`1.System.Linq.IQueryProvider.Execute(Expression expression) +49 System.Linq.Queryable.FirstOrDefault(IQueryable`1 source) +269 Bag.MedReg.Core.Managers.TextManager.GetTranslation(String key, String defaultValue, TextType textType) in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Core\Managers\TextManager.cs:74 Bag.MedReg.Web.Helpers.Translator.Translate(RouteValueDictionary routeData, String key, String defaultValue, TextType textType) in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Web\Helpers\Translator.cs:54 Bag.MedReg.Web.Controllers.BaseController.Translate(String key, String defaultValue, TextType textType) in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Web\Controllers\BaseController.cs:31 Bag.MedReg.Web.Controllers.SucheController.GetKantoneList(SucheManager sucheManager) in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Web\Controllers\SucheController.cs:201 Bag.MedReg.Web.Controllers.SucheController.ErstelleIndexModel(SucheManager sucheManager) in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Web\Controllers\SucheController.cs:381 Bag.MedReg.Web.Controllers.SucheController.Index() in d:\bamboo-agent-1\build-dir\MEDREG-MASTERREL-BUILD\Bag.MedReg.Web\Controllers\SucheController.cs:331 lambda_method(ExecutionScope , ControllerBase , Object[] ) +40 System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters) +17 System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +178 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +24 System.Web.Mvc.<>c__DisplayClassd.<InvokeActionMethodWithFilters>b__a() +52 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +254 System.Web.Mvc.<>c__DisplayClassf.<InvokeActionMethodWithFilters>b__c() +19 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +254 System.Web.Mvc.<>c__DisplayClassf.<InvokeActionMethodWithFilters>b__c() +19 System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +192 System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +314 System.Web.Mvc.Controller.ExecuteCore() +105 System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) +39 System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(RequestContext requestContext) +7 System.Web.Mvc.<>c__DisplayClass8.<BeginProcessRequest>b__4() +34 System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +21 System.Web.Mvc.Async.<>c__DisplayClass8`1.<BeginSynchronous>b__7(IAsyncResult _) +12 System.Web.Mvc.Async.WrappedAsyncResult`1.End() +59 System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult) +44 System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result) +7 System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +8760862 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +155 Version Information: Microsoft .NET Framework Version:2.0.50727.3662; ASP.NET Version:2.0.50727.3658
When trying to restart the import getting https://www.medregbm.admin.ch/Publikation/CreateExcelListMedizinalPersons failed with Timeout::Error
The report for SL-Update comes after my last commit with the attachement but is missing the content. Must be fixed.
Analysing the log output via grep -B20 -ri SL-Update log/oddb/debug/2014/11.log where I find lines like
2014-11-06 07:30:06 CET bsv_xml: entry.name = PR141101.txt 2014-11-06 07:30:06 CET bsv_xml: entry.name = Preparations.xml 2014-11-06 11:09:08 CET bsv_xml: entry.name = Publications.xls 2014-11-06 11:09:08 CET return_value_BsvXmlPlugin.update = "/var/www/oddb.org/data/xml/XMLPublications-2014.11.06.zip" 2014-11-06 11:09:08 CET getin log_notify_bsv 2014-11-06 11:09:08 CET date=#<Date: 2014-11-01 ((2456963j,0s,0n),+0s,2299161j)> 2014-11-06 11:09:08 CET after pointer creating 2014-11-06 11:09:40 CET bsv_xml: attached file Duplicate_Registrations_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Package_Data_was_completed_from_SL.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file SMeX_SL_Differences__Registrations__06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file SMeX_SL_Differences__Packages__06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Critical_Pharmacodes_BAG_XML_06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Missing_Swissmedic_Codes_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Missing_Pharmacodes_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Missing_Swissmedic_Codes_in_SL__out_of_trade__06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Unknown_Packages_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:40 CET bsv_xml: attached file Unknown_Products_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:40 CET after log_info(plug) 2014-11-06 11:09:40 CET before @app.update 2014-11-06 11:09:41 CET after @app.update 2014-11-06 11:09:41 CETlog notify SL-Update (XML): start outgoing process ["oddb_bsv"] 2014-11-06 11:09:41 CET part ["text/plain", "Duplicate_Registrations_in_SL_06.11.2014.txt", "Duplicate Registrations in SL 06.11.2014 0\nZwei oder mehr \"Preparations\" haben den selben 5-stelligen Swissmedic-Code\n\n"] <..> 2014-11-06 11:09:41 CET part ["text/plain", "Unknown_Products_in_SL_06.11.2014.txt", "Unknown Registrations in SL 06.11.2014 <..> 2014-11-06 11:09:46 CETlog notify SL-Update (XML): sent mail 2014-11-06 11:09:46 CET the first log.notify end 2014-11-06 11:09:46 CET return_value_log_notify = #<File:/var/www/oddb.org/log/oddb/debug/2014/11.log (closed)> 2014-11-06 11:09:46 CET bsv_xml: attached file SMeX_SL_Differences__Registrations__06.11.2014.txt is saved 2014-11-06 11:09:46 CET bsv_xml: attached file Missing_Swissmedic_Codes_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:46 CET bsv_xml: attached file Missing_Swissmedic_Codes_in_SL__out_of_trade__06.11.2014.txt is saved 2014-11-06 11:09:46 CET bsv_xml: attached file Unknown_Packages_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:46 CET bsv_xml: attached file Unknown_Registrations_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:46 CET bsv_xml: attached file Unknown_Packages_in_SL__out_of_trade__06.11.2014.txt is saved 2014-11-06 11:09:47 CET bsv_xml: attached file Missing_Pharmacodes_in_SL_06.11.2014.txt is saved 2014-11-06 11:09:47 CET bsv_xml: attached file Duplicate_Registrations_in_SL_06.11.2014.txt is saved
Found the problem. In src/plug/bsv_xml.rb in the procedure log_info_bsv the body was never updated!
Added first a unit test that sending e-mails with a body and attachment works. Pushed commits indent with whitespace and Added test for sending mail with attachments
I think I fixed the problem now. Running usr/local/bin/ruby jobs/import_bsv on oddb-ci2. My fix did not work and threw an error. I really should do a better test of wrap_update with plugin bsv_xml.
Second run had errors againg. Added a test unit for src/util/log.rb. But I produce an error as the one occured when running the import. Will continue tomorrow.
Running bundle exec bin/oddb2xml in my checkout to try to verify Zenos findings. Listing the shows
-rw-r--r-- 1 niklaus niklaus 42852418 Nov 10 08:10 Preparations.xml -rw-r--r-- 1 niklaus niklaus 4770630 Nov 10 08:10 swissindex_Pharma_DE.xml -rw-r--r-- 1 niklaus niklaus 4764078 Nov 10 08:10 swissindex_Pharma_FR.xml
Looking for Tomudex (which was recently changed) I found in the oddb_article.xml the following
<ART DT="">
<REF_DATA>1</REF_DATA>
<PHAR>2059753</PHAR>
<SMCAT>A</SMCAT>
<SMNO>53536018</SMNO>
<PRODNO>535361</PRODNO>
<VAT>2</VAT>
<SALECD>A</SALECD>
<CDBG>N</CDBG>
<BG>N</BG>
<QTY>Vial 1 Stk</QTY>
<DSCRD>TOMUDEX Trockensub 2 mg</DSCRD>
<DSCRF>TOMUDEX subst sèche 2 mg</DSCRF>
<SORTD>TOMUDEX TROCKENSUB 2 MG</SORTD>
<SORTF>TOMUDEX SUBST SèCHE 2 MG</SORTF>
<SYN1D>Tomudex</SYN1D>
<SYN1F>Tomudex</SYN1F>
<SLOPLUS>2</SLOPLUS>
<ARTCOMP>
<COMPNO>7601001383777</COMPNO>
</ARTCOMP>
<ARTBAR>
<CDTYP>E13</CDTYP>
<BC>7680535360188</BC>
<BCSTAT>A</BCSTAT>
</ARTBAR>
<ARTPRI>
<VDAT>01.03.2010</VDAT>
<PTYP>PEXF</PTYP>
<PRICE>220.93</PRICE>
</ARTPRI>
<ARTPRI>
<VDAT>01.01.2011</VDAT>
<PTYP>PPUB</PTYP>
<PRICE>270.05</PRICE>
</ARTPRI>
<ARTPRI>
<VDAT>10.11.2014</VDAT>
<PTYP>ZURROSE</PTYP>
<PRICE>229.73</PRICE>
</ARTPRI>
<ARTPRI>
<VDAT>10.11.2014</VDAT>
<PTYP>ZURROSEPUB</PTYP>
<PRICE>270.05</PRICE>
</ARTPRI>
<ARTINS>
<NINCD>10</NINCD>
</ARTINS>
</ART>
which looks good for me.
Problem was reported by Marco Descher who used in https://srv.elexis.info/jenkins/view/Artikelstamm/job/Artikelstamm%20generate%20oddb2xml/ the command oddb2xml -a nonpharma -p zurrose -e. He has fixed the problem by removing all files in the downloads folder before running the import. Nevertheless trying to reproduce the problem here.
Could not verify this behaviour here. It downloaded a new file (easily verified by looking at the SOAP timestamp) which reports the following <soap:Body><PHARMA CREATION_DATETIME="2014-11-08T06:37:08.6144413+01:00"