20110704-debug-slupdate-xml-oddb_org-rebulid-webalizer
<< Masa.20110705-testcases-oddb_org-view-10-state-16-model-2-util-2-command-1-ext-4 | 2011 | Masa.20110701-testcases-oddb_org-state-24-model-7-view-4-remote-2-plugin-1-command-1-util-1 >>
- Goal/Estimate/Evaluation
- Debug slupdate / 80% / 100%
- Rebuild webalizer / 80% / 80%
- Milestones
- Summary
- Commits
- Added a new method update_g_l__diff__s_b that does nothing because Generics.xml changed to GL_Diff_SB.xml but 'generic_type' is updated by Preparations.xml. (oddb.org)
- Fixed the undefined variable 'server'. (oddb.org)
- Tomorrow
- Rebuild webalizer on de.oddb.org, too
Confirm the bug locally
Plugin: ODDB::BsvXmlPlugin Error: NoMethodError Message: undefined method `update_g_l__diff__s_b' for #<ODDB::BsvXmlPlugin:0x7f2f9487f1e8> Backtrace: /var/www/oddb.org/src/plugin/bsv_xml.rb:640:in `send' /var/www/oddb.org/src/plugin/bsv_xml.rb:640:in `_update' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:761:in `get_input_stream' /var/www/oddb.org/src/plugin/bsv_xml.rb:640:in `_update' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1123:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1123:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1266:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1403:in `foreach' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1382:in `open' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.1/lib/zip/zip.rb:1401:in `foreach' /var/www/oddb.org/src/plugin/bsv_xml.rb:636:in `_update' /var/www/oddb.org/src/plugin/bsv_xml.rb:631:in `update' /var/www/oddb.org/src/util/updater.rb:251:in `update_bsv' /var/www/oddb.org/src/util/updater.rb:473:in `call' /var/www/oddb.org/src/util/updater.rb:473:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:249:in `update_bsv' /var/www/oddb.org/src/util/updater.rb:212:in `run' /var/www/oddb.org/jobs/import_daily:13 /var/www/oddb.org/src/util/job.rb:17:in `call' /var/www/oddb.org/src/util/job.rb:17:in `run' /var/www/oddb.org/jobs/import_daily:12
Run locally
$ bin/admin ch.oddb> Updater.new(self).update_bsv
Result
Plugin: ODDB::BsvXmlPlugin Error: NoMethodError Message: undefined method `update_g_l__diff__s_b' for #<ODDB::BsvXmlPlugin:0x7fbbdc3e7c00> Backtrace: /home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `send' /home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `_update' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:787:in `get_input_stream' /home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:640:in `_update' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1154:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1154:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1297:in `each' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1434:in `foreach' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1413:in `open' /usr/lib64/ruby/gems/1.8/gems/rubyzip-0.9.4/lib/zip/zip.rb:1432:in `foreach' /home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:636:in `_update' /home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:631:in `update' /home/masa/ywesee/oddb.org/src/util/updater.rb:251:in `update_bsv' /home/masa/ywesee/oddb.org/src/util/updater.rb:473:in `call' /home/masa/ywesee/oddb.org/src/util/updater.rb:473:in `wrap_update' /home/masa/ywesee/oddb.org/src/util/updater.rb:249:in `update_bsv' (eval):1:in `_admin' /home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call' /home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe' /home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1426:in `_admin' /home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1424:in `initialize' /home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1424:in `new' /home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1424:in `_admin' /usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__' /usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block' /usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform' /usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop' /usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop' /usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop' /usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start' /usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop' /usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run' /usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start' /usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run' /usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize' /usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new' /usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service' bin/oddbd:38
Note
- It is the same error
Trace error message
Test
masa@masa ~/ywesee/oddb.org/data/xml $ unzip XMLPublications-latest.zip Archive: XMLPublications-latest.zip inflating: GL_Diff_SB.xml inflating: Gestrichene_Packungen_Emballages_radies.xls inflating: ItCodes.xml inflating: PR110701.txt inflating: Preparations.xml inflating: Publications.xls
Note
- The new file, GL_Diff_SB.xml, in the archive is the problem
Compare file list
masa@masa ~/work $ unzip XMLPublications-2011.06.09.zip Archive: XMLPublications-2011.06.09.zip inflating: PR110601.txt inflating: Preparations.xml inflating: Publications.xls inflating: Generics.xml inflating: ItCodes.xml
Note
- Generics.xml is gone,
- Instead of that, GL_Diff_SB.xml and Gestrichene_Packungen_Emballages_radies.xls, are newly included.
New process
- Check 'pharmacode' in the cache
- Then, update 'generics' or 'original' depending on 'CodeOrgGen'
Note
- The 'generic_type', :original or :generic, is set by Preparations.xml, not by Generics.xml.
- The Generics.xml is used to identify the 'generic_group'
- and the problem is thta the pointer of generic group uses the 'GenGroupGen' tag data that is not in the new GL_Diff_SB.xml
Next
Think how to make a 'generic_group' from GL_Diff_SB.xml- Do nothing, just skip the file, because we do not use 'generic_group' anymore.
Commit
- Added a new method update_g_l__diff__s_b that does nothing because Generics.xml changed to GL_Diff_SB.xml but 'generic_type' is updated by Preparations.xml. (oddb.org)
- Fixed the undefined variable 'server'. (oddb.org)
Result
Created SL-Entries 71 Updated SL-Entries 8316 Deleted SL-Entries 26 Created Limitation-Texts 4 Updated Limitation-Texts 1306 Deleted Limitation-Texts 0 Duplicate Registrations in SL 04.07.2011 0 Package-Data was completed from SL 4 SMeX/SL-Differences (Registrations) 04.07.2011 0 SMeX/SL-Differences (Packages) 04.07.2011 6 Critical Pharmacodes BAG-XML 04.07.2011 9 Missing Swissmedic-Codes in SL 04.07.2011 7 Missing Pharmacodes in SL 04.07.2011 198 Missing Swissmedic-Codes in SL (out of trade) 04.07.2011 28 Unknown Packages in SL 04.07.2011 190 Unknown Registrations in SL 04.07.2011 2 Unknown Packages in SL (out of trade) 04.07.2011 52 Packungen in der ODDB Total: 23819 Packungen ohne Pharmacode: 10641 - ausser Handel: 10481 - inaktive Registration: 0 - noch nicht auf MedWin: 160
Confirm webalizer statistics
http://webalizer.ch.oddb.org http://webalizer.de.oddb.org For both the stats for June and July have not been created.
Check webalizer log
online server oddb.org/webalizer/oddb $ ls -al -rw-r--r-- 1 root root 143793 2011-04-01 00:00 usage_201103.html -rw-r--r-- 1 root root 142662 2011-04-30 23:51 usage_201104.html -rw-r--r-- 1 root root 143332 2011-06-01 23:51 usage_201105.html
Note
- There is no log file of June and July.
Check crontab
# update webalizer stats for virtual hosts */10 * * * * root /usr/local/bin/update_vhost_stats
update_vhost_stats
#!/bin/bash
for conf in /etc/webalizer/*.conf
do
webalizer -c $conf &> /dev/null
done
Run online Segmentation fault
thinpower log # webalizer -c /etc/webalizer/oddb.org.conf Webalizer V2.01-10 (Linux 2.6.32) German Webalizer Xtended RB17 patch implemented (3-Jan-2007) Verwende Protokolldatei /var/www/oddb.org/log/oddb/access_log (clf) DNS Abfrage (5): 492 Adressen in 670.50 Sekunden Benutze Datei fur den DNS-Cache /etc/webalizer/dns_cache.db Erzeuge Dateien in /var/www/oddb.org/webalizer/oddb Rechnername fur Bericht ist 'ch.oddb.org' Lese Datei mit historischen Daten... webalizer.hist Lese Daten vom letzten Programmlauf... webalizer.current Segmentation fault
Refer
Set up webalizer locally
Install
sudo emerge webalizer
Install Webalizer Xtended
Download source and patch
- ftp://ftp.mrunix.net/pub/webalizer/old/webalizer-2.01-10-src.tgz
- http://www.patrickfrei.ch/webalizer/rb17/webalizer-2.01-10-RB17-patch.tar.gz
Install
masa@masa ~/work/webalizer-2.01-10 $ tar zxvf webalizer-2.01-10-src.tgz masa@masa ~/work/webalizer-2.01-10 $ tar zxvf webalizer-2.01-10-RB17-patch.tar.gz masa@masa ~/work/webalizer-2.01-10 $ cd webalizer-2.01-10 masa@masa ~/work/webalizer-2.01-10 $ patch -Np1 -i ../webalizer-2.01-10-RB17-patch masa@masa ~/work/webalizer-2.01-10 $ ./configure masa@masa ~/work/webalizer-2.01-10 $ make masa@masa ~/work/webalizer-2.01-10 $ sudo make install /usr/bin/install -c webalizer /usr/local/bin/webalizer /usr/bin/install -c -m 644 webalizer.1 /usr/local/man/man1/webalizer.1 /usr/bin/install: reguläre Datei „/usr/local/man/man1/webalizer.1“ kann nicht angelegt werden: Datei oder Verzeichnis nicht gefunden make: *** [install] Fehler 1
Adjust the configuration in /usr/local/etc/webalizer.conf
masa@masa ~/work/webalizer-2.01-10 $ /usr/local/bin/webalizer Webalizer V2.01-10 (Linux 2.6.34) English Webalizer Xtended RB17 patch implemented (3-Jan-2007) Error: Can't open log file /var/log/apache/access_log
Experiment (oddb.org/log/rebuild-webalizer)
#!/bin/bash
LOCK_FILE="/var/lock/update_vhost_stats"
LOOKANDFEEL=${1%"/"}
CONFIG_FILE=$2
trap 'rm $LOCK_FILE' EXIT
TMPDIR="/tmp/$(basename $0).$$"
LOG_FILE="$LOOKANDFEEL/access_log.bak"
TARGET_DIR="../webalizer/$LOOKANDFEEL"
#DNS_CACHE="/etc/webalizer/dns_cache.db"
mkdir -p $TMPDIR
TMP_FILE="/tmp/oddb.$LOOKANDFEEL.access_log"
#cat $LOOKANDFEEL/*/*/*/access_log > $TMP_FILE
cat $LOOKANDFEEL/2011/*/*/access_log > $TMP_FILE
#fastzolver -s -L100 -H1500 -U1500 -D $DNS_CACHE $TMP_FILE
#webalizer -p -c $CONFIG_FILE -N15 -D $DNS_CACHE -o $TMPDIR $TMP_FILE
/usr/local/bin/webalizer -p -c $CONFIG_FILE -o $TMPDIR $TMP_FILE
rm $TMP_FILE
rm -rf $TARGET_DIR.bak
cp -r $TARGET_DIR $TARGET_DIR.bak
mv $TMPDIR/*.* $TARGET_DIR
rm -rf $TMPDIR
Note
- It is simplified from the original one
Run
masa@masa ~/ywesee/oddb.org/log $ sudo ./rebuild-webalizer oddb oddb.org.conf Passwort: Warning: Invalid keyword 'DNSCache' (oddb.org.conf) Warning: Invalid keyword 'DNSChildren' (oddb.org.conf) Webalizer V2.01-10 (Linux 2.6.34) English Webalizer Xtended RB17 patch implemented (3-Jan-2007) Using logfile /tmp/oddb.oddb.access_log (clf) Creating output in /tmp/rebuild-webalizer.14476 Hostname for reports is 'ch.oddb.org' History file not found... Previous run data not found... Saving current run data... [06/30/2011 23:59:59] Generating report for June 2011 Generating summary report Saving history information... 487683 records (382861 ignored) in 114.00 seconds, 4277/sec rm: Entfernen von „/var/lock/update_vhost_stats“ nicht möglich: Datei oder Verzeichnis nicht gefunden
Result
masa@masa ~/ywesee/oddb.org/log $ ls ../webalizer/oddb/ ctry_usage_201106.png daily_usage_201107.png hourly_usage_201106.png responses_201106.png usage_201107.html webalizer.hist ctry_usage_201107.png error_201106.html hourly_usage_201107.png responses_201107.png usage.png daily_usage_201106.png error_201107.html index.html usage_201106.html webalizer.current
Note
- 'index.html' file is overwritten
rebuild-webalizer (final)
Run online
cd /var/www/de.oddb.org/log
oddb.org/log/oddb $ du -sh 2011 12G 2011 $ df -h Filesystem Size Used Avail Use% Mounted on rootfs 149G 73G 77G 49% / log # ./rebuild-webalizer oddb /etc/webalizer/oddb.org.conf
Note
- 12 GB of disk space will be used