view · edit · sidebar · attach · print · history

Index>

20161221-davaz-com-multithreaded

Summary

Commits

Index

Davaz.com should support heavy workload (multithreading)

Things to complete today:

  • Finish and document choice of multithreading in SBSM
  • Fix displaying paiting for davaz.com.

In SBSM I have now 8 failing tests. The last commit had only 1 in Travis-CI. That is nothing I like to commit.

The first painting is not shown, as the whole javascript is rendered on the first read. Not only the reference to a gif as on davaz.com. Why?

It should easy to write a test for it as the correct title is 'Da Vaz - Creating a Drawing.' Which changes afterward to 'Da Vaz - Abstract Artist from Switzerland | Personal'. Created the unit test for it and it fails as expected.

If I load http://localhost:11090/index.html the drawing is displayed correctly. Somehow the (non-exisitent flavor) leads to visiting /undefined. Debugging it. When visiting '/' I see in the

I, [2016-12-21T10:30:58.089813 #3980]  INFO -- : session.rb:112:in `initialize' initialize th #<DaVaz::Util::TransHandler:0x0055b86d293c60> validator #<DaVaz::Util::Validator:0x0055b86d293eb8> app DaVaz::Util::App
D, [2016-12-21T10:30:58.094838 #3980] DEBUG -- : session.rb:137:in `initialize' session initialized #<DaVaz::Util::Session:0x0055b86c12f488> with @cgi #<CGI:0x0055b86c12d750> multi_threaded false app 47125296879940
D, [2016-12-21T10:30:58.094971 #3980] DEBUG -- : session.rb:50:in `initialize' session DaVaz::Util::Validator @app DaVaz::Util::App app DaVaz::Util::App @session 8 validator DaVaz::Util::Validator
D, [2016-12-21T10:30:58.095157 #3980] DEBUG -- : app.rb:120:in `call' starting session_id 21efa435d8c90c297abe7eed35f  session DaVaz::Util::Session /: cookies  are {} @cgi NilClass
D, [2016-12-21T10:30:58.095737 #3980] DEBUG -- : session.rb:127:in `logout' logout SBSM::UnknownUser with persistent_user_input(remember) nil
I, [2016-12-21T10:30:58.095855 #3980]  INFO -- : app.rb:376:in `logout' @yus_server #<DRb::DRbObject:0x0055b86d292c98 @uri="druby://localhost:14000", @ref=nil> Class 47125289197260 
D, [2016-12-21T10:30:58.097172 #3980] DEBUG -- : session.rb:370:in `logout' logout "/" setting @state 47125283781900 DaVaz::State::Personal::Init remember nil
I, [2016-12-21T10:30:58.097305 #3980]  INFO -- : app.rb:263:in `load_movies_ticker' @db_manager is #<DaVaz::Stub::DbManager:0x0055b86d43d700>
D, [2016-12-21T10:30:58.099441 #3980] DEBUG -- : session.rb:107:in `login_token' login_token name nil token 
D, [2016-12-21T10:30:58.099607 #3980] DEBUG -- : session.rb:213:in `block in process_rack' active_state.trigger state 47125283719940 remember nil
D, [2016-12-21T10:30:58.099708 #3980] DEBUG -- : session.rb:222:in `block in process_rack' Changing from 47125283781900 to state DaVaz::State::Personal::Global 47125283719940 remember nil
D, [2016-12-21T10:30:58.103691 #3980] DEBUG -- : app.rb:136:in `call' finish session_id.1 21efa435d8c90c297abe7eed35f: matches response.headers['Set-Cookie']

Checking whether the first state visited when going to '/' is src/view/personal/init.rb. Yes. But it returns already the text for the en/personal/home page.

Decided to revert SBSM & davaz.com to the state where they worked with DRb. Will test, then just turn multithreading off in SBSM. Pushed the corresponding commits for SBSM.

Will add the changes to davaz.com to make it thread safe. Davaz.com work as soon as the mysql2 gem is installed in version 0.4.5 and not 0.4.4. (wrk reports 14.68 request/secondds). Checking whether undoing the change in sbsm is needed, yes it is needed. Also remarked that while wrk -t12 -c40000 -d30s --script=test/wrk_davaz.lua http://davaz.ngiger.ch davaz.ngiger.ch does not respond.

With sbsm 1.3.6-rc1 (no DRb + single-threaded) + davaz.com with mysql2 0.4.5 I have (when working in parallel browser) still a few non 2xx/3xx answers as shown

Running 30s test @ http://davaz.ngiger.ch
  12 threads and 40000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    18.66s     6.45s   29.77s    75.22%
    Req/Sec     2.02      1.47     7.00     82.27%
  634 requests in 30.08s, 7.90MB read
  Socket errors: connect 24011, read 0, write 0, timeout 573567
  Non-2xx or 3xx responses: 4
Requests/sec:     21.07
Transfer/sec:    269.01KB

The problem with the initial drawing not show stems from the fact that under mod_ruby the apache directive DirectoryIndex index.html index.rbx was honoroud, whereas with Webrick it isn't.

This can be changed in sbsm/app.rb by replacing then line file_name = File.expand_path(File.join('doc', request.path)) with

      if '/'.eql?(request.path)
        file_name = File.expand_path(File.join('doc', 'index.html'))
      else
        file_name = File.expand_path(File.join('doc', request.path))
      end

This patch works for the '/' in davaz.com but does replace the Apache behaviour if sub-directories would be concerned and/or the index.html would be placed after index.rbx. But as far as I see we do not have such occurrences in the apps distributed by ywesee.

Therefore still asking myself whether I should replace the DRb calls. In favor are:

  • Without DRb we serve around 30 request/second with DRb only about 15
  • davaz.com needs mysql2 gem 0.4.5 in both cases

against are:

  • 8 failures in unit test of sbsm
  • Davaz-Unit-Test reports 31 runs, 138 assertions, 4 failures, 1 errors, 8 skips
  • run wrk with test/wrk_davaz.lua viting the pages {"/de/page/feedback", "/de/page/about", "/", 'en/works/movies'} reports
wrk -t12 -c40000 -d30s --script=test/wrk_davaz.lua  http://davaz.ngiger.ch
Running 30s test @ http://davaz.ngiger.ch
  12 threads and 40000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    15.93s     1.70s   17.28s    67.88%
    Req/Sec     5.90      3.78    23.00     62.78%
  1758 requests in 30.07s, 18.52MB read
  Socket errors: connect 24011, read 0, write 0, timeout 559728
  Non-2xx or 3xx responses: 497
Requests/sec:     58.45
Transfer/sec:    630.50KB

Pushed commits Create RackInterface. Global Mutex possible and Updated to use SBSM with RackInterface

Installed the changes to fastpower. Changed Gemfile to point SBSM to gem 'sbsm', :git => 'https://github.com/ngiger/sbsm.git' and called sudo -u apache /usr/local/bin/bundle install. Changed apache2/vhosts.d/davaz.com.conf to rack. svc -d /service/davaz.com; svc -u /service/davaz.com. Changed owner of /var/www/new.davaz.com/log/2016/12/ to apache. Now my manual tests with davaz.com work fine.

Running (a more realistic) wrk test with 3 thread/connection gave no errors and I was able to visit other sides too. wrk reported

wrk -t3 -c3 -d30s --script=test/wrk_davaz.lua  http://davaz.com
Running 30s test @ http://davaz.com
  3 threads and 3 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   613.85ms  284.39ms 810.42ms   76.01%
    Req/Sec     4.54      8.33    43.00     84.91%
  405 requests in 30.00s, 29.14MB read
Requests/sec:     13.50
Transfer/sec:      0.97MB

Pushed the lua script with commit Added test/wrk_davaz.lua

The URL /en/communication/links/ chokes, because the model does not have the required fields title and url. As seen by this pry session

[1] pry(#<DaVaz::View::TextBlock>)> @model
=> #<DaVaz::Model::Link:0x000000015ab850
 @artobject_id=946,
 @artobjects=
  [#<DaVaz::Model::ArtObject:0x000000015ab7b0
    @artgroup_id="PHO",
    @artobject_id=783,
    @author="",
    @charset="",
    @country_id="J",
    @date=#<Date: 2006-05-04 ((2453860j,0s,0n),+0s,2299161j)>,
    @language="",
    @links=[],
    @location="Kanazawa",
    @material_id=40,
    @movie_type="none",
    @price=0,
    @public=1,
    @serie_id="AEN",
    @serie_position="1",
    @size="20KB",
    @tags=[],
    @text="",
    @title="Kanazawa, Cherry Blossoms, Mai 2006",
    @tool_id=69,
    @url="",
    @wordpress_url="">],
 @link_id=107,
 @word="Hunting">

Okay. I think I found the problem. The method load_links is defined twice in src/util/app.rb (line 229 and 331), but does not return the same elements. Same for the methods delete_link(link_id), but here the implementation is the same. Deleting one. Renamed one occurrence of load_links to load_artobjects_links.

Pushed a commit Cleanup app. Removed duplicated methods I am unsure whether we need somewhere the old implementation of load_links or not.

Pushed commit Removed possibility to add guestbook to prevent SPAM

view · edit · sidebar · attach · print · history
Page last modified on December 21, 2016, at 06:24 PM