Justin Francis Self-Portrait

Saturday, July 26, 2008

Persistence of an In-Memory Application Server

My philosophy is that the database should be used primarily as dumb storage. Keeping as much as possible in the middle layer of an application affords us programmers the most power; most of the changes we will need to make will be in the language we are most comfortable in.

For an In-Memory application like ours (one that loads all its data at startup and does not read from the database thereafter), the database takes on a totally different role from that of the holder of application state to a glorified file. On each modification of the system, of course, we immediately write the change to the database so if the system crashes, it will boot back in the same state.

Recently, the largest single cause of wait-times in our system was contention for these database writes. But with an In-Memory system, the state of the application is not maintained in the database. Therefore the correct functioning of the system is not dependent on the database. In fact, if we ignored system crashes, we could just persist the entire domain model of the system right before we shut it down and everything would work fine.

After becoming comfortable with the idea that writing changes to the database is only so the system can reboot properly, the most profound revelation occurred to me. What does the user care if the system persisted the change they just made? Regardless, when they make their next request (assuming no system crash) the changes they made are still there because the state of the application is in memory, not in the database. So why not skip the save (from the user's perspective) and schedule it to be done after the response is returned to the user?

Doing saves asynchronously increased the response time of all system modifications by factor of 10 on average. Not bad for a day or two of programming. Oh, and 5 years of building software.

The implementation was relatively simple. We created a class on which domain object saves would be queued. In fact, we made it more general and used a decorator that would queue any function or method on this queue, which would then call the function at some point in the near future.

The advantages are obviously the speed boost, but as a bonus we save writes when there are multiple sequential modifications of the same object, which as you might expect are very common. We also reduce the load both on the database server and our system because only a single thread (the queue thread) is trying to modify the database at any given time.

There are some disadvantages. Probably the biggest your are thinking of is a failure during a save. Some other things that come to mind are data integrity errors and too many modifications such that we cannot keep up. One final potential problem I'll discuss is a system crash, which could result in some lost writes.

Foreign keys, data-type and other data-related errors that could happen during the save are not a problem. First of all, they should never happen because you have coded these constraints into your system (or they exist naturally -- how can you have a broken FK in your object model? The worst you could have is an unlinked object). Second, if they do happen, you can notify the programmers (because if the system does not prevent this kind of problem, the user certainly won't be able to solve it from some cryptic error message) or simply re-queue the save (it's probably a temporary problem anyway).

The problem of potentially having saves queue up faster than they can be cleared is unlikely. I have never found that spawning multiple threads to handle many writes to a database speed things up much (which is what happens when each request synchronously writes to the database). It is simple, however, to create multiple threads to save a backlogged queue. So far, we have not seen this in our system. The average size of the backlog in the queue is 0.

Finally, what happens if the system crashes? I should point out that our system does not crash more than once a month, and that is just because we have been going through a rough patch. The real question is how many writes are normally waiting to be completed? It will probably be close to zero. No matter what, writes are getting lost in a crash (the ones in progress or starting). If you tune your queue properly, you can minimize the risk and impact of a crash. The top priority, however, should be finding out why the system is crashing and fixing the source of this problem.

The conclusion here is that users don't care about persistence, so don't make them wait for it unless you have to. If you have to, examine your assumptions, and if you are still using the database as your authority on application state, consider an in-memory system. It might solve a lot of problems you are having, database contention probably chief among them.

21 comments:

Anonymous said...

The most controversial acne product on any market, Accutane is arguably also the most powerful

[url=http://www.clearskin.net/viewtopic.php?t=1637&highlight=accutane ]accutane moisturizer [/url]
http://www.clearskin.net/viewtopic.php?t=1934&highlight=accutane
http://www.clearskin.net/viewtopic.php?t=2421&highlight=accutane
http://www.clearskin.net/viewtopic.php?t=2391&highlight=accutane

[url=http://www.clearskin.net/viewtopic.php?t=3083&highlight=accutane ]facial flushing and accutane [/url]

Anonymous said...

Elise, wtf???

Yours truly
Anderson

Anonymous said...

ромны знакомства интим знакомства в минске и беларуси знакомства волгоградская обл серафимович знакомства в каховке сайты знакомства в тольятти мужчины узбекистана знакомства знакомства только для занятия ом знакомства для секса пустошка афоризмы о целях знакомства знакомства за деньги сургут

Anonymous said...

buy facebook likes
facebook likes

http://drupalsn.com/learn-drupal/drupal-tutorials/converting-drupal-themes-drupal-6-drupal-7-my-favourite-additions-chan http://www.crowdedhouse.com/news/vote-best-crowded-house-live
1000 facebook likes buy facebook likes get facebook likes
i have kaspersky virus protection 2010 and under "my protection" it says it has detected 20 trojans. I did a full scan which took about 2 and a half hours, and nothing happened. it still says i have 20 trojans. What do i do to get rid of them?

buy facebook likes 1000 facebook likes [url=http://1000fbfans.info]facebook likes [/url] 1000 facebook likes

Anonymous said...

[url=http://levitranowdirect.com/#zzxph]levitra online[/url] - levitra 10 mg , http://levitranowdirect.com/#zxeyp buy cheap levitra

Anonymous said...

[url=http://viagranowdirect.com/#yyrlz]cheap generic viagra[/url] - viagra without prescription , http://viagranowdirect.com/#opbwv viagra 120 mg

Anonymous said...

[url=http://viagraboutiqueone.com/#yiggz]viagra 50 mg[/url] - viagra 120 mg , http://viagraboutiqueone.com/#mnain buy generic viagra

Anonymous said...

[url=http://orderviagradirectlyonline.com/#agqut]viagra without prescription[/url] - cheap viagra , http://orderviagradirectlyonline.com/#etqpt viagra online

Anonymous said...

[url=http://www.frsacslongchampsoldes.com/]Longchamp Pliage[/url] a la mode et d'actualisation 2013 Longchamp Le Pliage Sac de voyage noir, avec Longchamp Sacs 100% authentique de vente a ne pas rater I use only Hessian and organic cotton bags. I have six large Hessian shopping bags for food shopping only, I take them to farmers markets etc. The smaller organic cotton ones I use because I can fold them up small and put them in my coat pocket or bag whenever I go out.
http://www.frsacslongchampsoldes.com/ To get the word out, use your poster printing budget to create posters that tout a free product if someone shows up at your store. You could also mail out or hang advertising flyers in supermarkets touting a free product if people will go to your Web site and request it. This drives traffic to your Web site and you also get customer contact information to use later..

Anonymous said...

Coal-black Rhinoceros - a large and resilient animal. he did not as sturdy as the bloodless rhinoceros, but but powerful - reaches the weight 2-2, 2 m, lengths of up to 3, 15 m in height shoulders of 150-160 cm.

Anonymous said...

portion the students who Bear the Fashion Design appetite. Multi-colored bleached fabrics are extraordinarily complex and personal freedom o'er political and economic interventions in many styles -- from low-rise to good outfits, from boho way Fashion too. www.kasper-suits.net http://kasper-suits.net [url=http://kasper-suits.net/]kasper suits[/url] many of the fashion, you testament Appear, and more vulgar for men, women sexual love. kasper dresses kasper pant suits functional with jcpenney has allowed me to use a fashion disaster.

John said...

abercrombie and fitch
oakley sunglasses
ed hardy clothing
beats by dre
ray ban outlet
louis vuitton outlet stores
instyler curling iron
louis vuitton outlet
christian louboutin outlet
michael kors outlet
uggs on sale
abercrombie and fitch
michael kors outlet
abercrombie
canada goose
nike air max
nike roshe run
michael kors bag
cheap oakley sunglasses
supra shoes
louis vuitton bags
replica watches
uggs for men
jordan retro
canada goose outlet
ugg boots
coach factory outlet
ugg boots for men
hollister jeans
michael kors handbags
canada goose jackets
hollister uk
canada goose outlet
air max 90
lebron 12
canada goose jackets
louis vuitton handbags
marc jacobs
ray-ban sunglasses
toms shoes
20151219yuanyuan

sourabh gupta said...
This comment has been removed by the author.
sourabh gupta said...
This comment has been removed by the author.
sourabh gupta said...
This comment has been removed by the author.
sourabh gupta said...
This comment has been removed by the author.
chenmeinv0 said...

canada goose outlet
adidas nmd runner
cheap ray ban sunglasses
cheap oakley sunglasses
cheap nfl jerseys
louis vuitton outlet
canada goose jackets
cheap uggs
coach factory outlet
nike roshe run shoes
201612.27chenjinyan

Unknown said...

timberland outlet
toms shoes
ugg outlet
mont blanc pens
timberland boots
ray ban sunglasses
coach outlet store online
moncler uk
oakley sunglasses
michael kors handbags
20172.7wengdongdong

Anonymous said...

louboutin shoes
oakley sunglasses outlet
christian louboutin sale
air jordan retro
pandora charms outlet
michael kors outlet online
tory burch outlet online
moncler
cheap nike shoes
ralph lauren outlet online
mt1225

Unknown said...

qzz0428
fitflops
coach factory outlet
true religion jeans
ralph lauren uk
nike factory store
oakley sunglasses
nike outlet
mbt shoes outlet
mulberry handbags
air max 90

Obat Gatal Kudis/Gudik said...


This article is interesting and useful. Thank you for sharing. And let me share an article about health that God willing will be very useful. Thank you :)

Walatra Gamat Emas Kapsul
Vitamin Untuk Kesehatan Anak
Penyebab sering mimisan
Cara Mengatasi Cacar Air
Cara Menghilangkan Kantung mata
Obat Telinga Berkerak dan Berair