OVERVIEW
After installing Magento on hosting you probably have a desire to optimize its work.
Google offers solutions, which consist on using one of the following options: installing higher-performance hardware, correction for mysql/php configuration, using PHP accelerators, accelerate downloads JS and CSS, the correction assignment expires date for content, customization Magento core files to fit your needs.
Varien released a book titled “Enterprise Edition Whitepaper High Performance eCommerce” which explains the above methods of Magento optimization. I recommend reading this book and use the described methods of optimization.
I propose another new approach for the optimization of Magento, which was not described in the Internet before. I propose to use Varnish for caching pages.
What is Varnish you can read here. Varnish handles the request, looks in the cache this page and, if such page is found, returns it. If the page is not found in the cache – a request sent to Apache. This approach allows you to create any number of dynamic pages.
EXAMPLE
On the main page you can see poll module. Assume that we have 4 different polls. I propose randomly create 100 different copies of the main page. When user loads main page Varnish gives random copy of the main page. The probability of withdrawal of one of the 4 polls close to 25%.
This caching will give us the static pages to users who simply browse the site and did not do such actions as a vote, adding product to cart, login, etc. Majority of these users, so using this approach can significantly reduce the load on Apache. Once a user has voted, logged, added product to cart, etc. Varnish cache turn off.
INSTALLATION
Download the latest version of Magento and install it.
Create folder app/code/local/Varnish.
Create file app/etc/modules/Varnish.xml :
<?xml version="1.0"?>
<config>
<modules>
<Varnish>
<active>true</active>
<codePool>local</codePool>
</Varnish>
</modules>
</config>
app/code/local/Varnish/controllers/IndexController.php :
<?php
class Varnish_IndexController extends Mage_Core_Controller_Front_Action
{
public function purgeallAction() { }
}
app/code/local/Varnish/etc/config.xml :
<?xml version="1.0"?>
<config>
<modules>
<Varnish>
<version>0.0.1</version>
</Varnish>
</modules>
<global>
<models>
<varnish>
<class>Varnish_Model</class>
<resourceModel>varnish_mysql4</resourceModel>
</varnish>
</models>
<events>
<http_response_send_before>
<observers>
<varnish>
<type>singleton</type>
<class>varnish/observer</class>
<method>varnish</method>
</varnish>
</observers>
</http_response_send_before>
<application_clean_cache>
<observers>
<varnish>
<type>singleton</type>
<class>varnish/observer</class>
<method>purgeAll</method>
</varnish>
</observers>
</application_clean_cache>
</events>
</global>
<frontend>
<routers>
<varnish>
<use>standard</use>
<args>
<module>Varnish</module>
<frontName>varnish</frontName>
</args>
</varnish>
</routers>
</frontend>
<default>
<varnish>
<purgeall_key>
<key>#Fj1nzljh</key>
</purgeall_key>
</varnish>
</default>
</config>
app/code/local/Varnish/Model/Observer.php :
/**
 * Varnish Observer model
 *
 * @category   Varnish
 * @package    Varnish
*/
class Varnish_Model_Observer
{
    private $_request = null;
    public function __construct()
    {
    }
    private function getSecureKey()
    {
        return Mage::getStoreConfig('varnish/purgeall_key/key');
    }
    public function getCookie()
    {
        return Mage::app()->getCookie();
    }
    public function varnish($observer)
    {
        if ($this->isSetNoCacheStable()) {
            return false;
        }
        if ($this->pollVerification()) {
            $this->setNoCacheStable();
            return false;
        }
        if ($this->quoteHasItems()) {
            $this->turnOffVarnishCache();
            return false;
        }
        if ($this->customerIsLogged()) {
            $this->turnOffVarnishCache();
            return false;
        }
        $this->turnOnVarnishCache();
    }
    public function turnOffVarnishCache()
    {
        $this->getCookie()->set('nocache', 1);
    }
    public function turnOnVarnishCache()
    {
        $this->getCookie()->delete('nocache');
    }
    public function quoteHasItems()
    {
        $quote = Mage::getSingleton('checkout/session')->getQuote();
        if ($quote instanceof Mage_Sales_Model_Quote && $quote->hasItems()) {
            return true;
        }
    }
    public function customerIsLogged()
    {
        $customerSession = Mage::getSingleton('customer/session');
        if ($customerSession instanceof Mage_Customer_Model_Session  &&
            $customerSession->isLoggedIn()) {
            return true;
        }
    }
    public function pollVerification()
    {
        $justVotedPollId = (int) Mage::getSingleton('core/session')->getJustVotedPoll();
        if ($justVotedPollId) {
            return true;
        }
    }
    public function setNoCacheStable()
    {
        $this->getCookie()->set('nocache_stable', 1, 0);
    }
    public function isSetNoCacheStable()
    {
        return $this->getCookie()->get('nocache_stable') === 1;
    }
    public function purgeAll()
    {
        try {
            $url = Mage::getBaseUrl().'varnish/index/purgeall/key/'.$this->getSecureKey().'/';
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PURGE');
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
            $responseBody = curl_exec($ch);
            curl_close ($ch);
        } catch (Exception $e) {
            Mage::log($e->getFile().' '.$e->getLine().' '.$e->getMessage());
        }
    }
}
sudo vim /etc/varnish.vcl :
backend apache {
  .host = "127.0.0.1";
  .port = "8088";
  .max_connections = 30;
}
acl purge {
  "localhost";
}
sub vcl_recv {
  #purge all
  if (req.request == "PURGE") {
    if (!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    if (req.url ~ "varnish/index/purgeall/key/#Fj1nzljh") {
      purge_hash( ".*" );
    }
  }
  #described below need to create random "DynamicPage" value; this functionallity we use
  #to create several caches for 1 dynamic page.
  #for example, "int i = rand() % 2 + 1;" means that we have 2 different copies of one dynamic page
  C{
   #include
   #include
   char buffer [33];
   int i = rand() % 10 + 1;
   sprintf(buffer, "%d", i);
   VRT_SetHdr(sp, HDR_REQ, "\014DynamicPage:", buffer, vrt_magic_string_end);
  }C
  if (req.request != "GET" &&
  req.request != "HEAD" &&
  req.request != "PUT" &&
  req.request != "POST" &&
  req.request != "TRACE" &&
  req.request != "OPTIONS" &&
  req.request != "DELETE") {
    return (pipe);
  }
  # do not cache POST requests
  if (req.request == "POST") {
    return (pipe);
  }
  #we should not cache any page for Magento backend
  if (req.request == "GET" && (req.url ~ "^/admin") || req.url ~ "^/index.php/admin") {
    return (pass);
  }
  #we should not cache any page for checkout and customer modules
  if (req.request == "GET" && (req.url ~ "^/checkout" || req.url ~ "^/customer")) {
    return (pass);
  }
  #do not cache till session end
  if (req.http.cookie ~ "nocache_stable") {
    return (pass);
  }
  #unique identifier witch tell Varnish use cache or not
  if (req.http.cookie ~ "nocache") {
    return (pass);
  }
  if (req.request == "GET" && (req.url ~ "\.(png|jpg|jpeg|gif)$" || req.url ~ "print.css")) {
    lookup;
  }
  #Even though there are few possible values for Accept-Encoding, Varnish treats
  #them literally rather than semantically, so even a small difference which makes
  #no difference to the backend can reduce cache efficiency by making Varnish cache
  #too many different versions of an object.
  #http://varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;
    }
  }
  return (lookup);
}
sub vcl_pipe {
  # Note that only the first request to the backend will have
  # X-Forwarded-For set.  If you use X-Forwarded-For and want to
  # have it set for all requests, make sure to have:
  # set req.http.connection = "close";
  # here.  It is not set by default as it might break some broken web
  # applications, like IIS with NTLM authentication.
  return (pipe);
}
sub vcl_pass {
  return (pass);
}
sub vcl_hit {
  if (!obj.cacheable) {
    return (pass);
  }
  return (deliver);
}
sub vcl_miss {
  return (fetch);
}
sub vcl_fetch {
  return (deliver);
}
sub vcl_deliver {
  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
}
sub vcl_error {
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>"} obj.status " " obj.response {"</title>
</head>
<body>
<h1>Error "} obj.status " " obj.response {"</h1>
<p>"} obj.response {"</p>
<h3>Guru Meditation:</h3>
<p>URL: "} obj.http.bhash {"</p>
<p>XID: "} req.xid {"</p>
<hr>
<address>
<a href="http://www.varnish-cache.org/">Varnish cache server</a>
</address>
</body>
</html>
"};
return (deliver);
}
This is the place for you to tell a litle bit about your self. Just write your name, age, hobbies or anything else.
Horaayy..there are 44 comment(s) for me so far ;)
very interesting article. I didn’t try the code myself yet but I hope to try it soon.
do you have any benchmarks available?
Thank you for sharing this info. It looks very promising.
I was wondering if you have any production sites running this way that you would mind sharing?
Perhaps some benchmark data?
Hello Filipe and Shiloh!
Thank you for your interest in Varnish.
At the moment I do not have a dedicated server on the Internet with Varnish.
When we use native cache, Magento’s home page is loaded in about 1.4-1.6 seconds.
I thoroughly tested it on the local machine and got the result: page is loaded in about 0.3-0.4 seconds.
All the code in the article checked on a real project.
Hi I run into this problem.
root@server [/usr/local/sbin]# ./varnishd -a 127.0.0.1:8088 -f /etc/varnish.vcl -s file,/var/cache/varnish.cache,512M
storage_file: filename: /var/cache/varnish.cache size 512 MB.
Message from VCC-compiler:
No backends or directors found in VCL program, at least one is necessary.
Running VCC-compiler failed, exit 1
VCL compilation failed
I solved the problem but another one came up
storage_file: filename: /var/cache/varnish.cache size 512 MB.
Message from C-compiler:
./vcl.1P9zoqAU.c:481:12: error: #include expects “FILENAME” or
./vcl.1P9zoqAU.c:482:12: error: #include expects “FILENAME” or
./vcl.1P9zoqAU.c: In function âVGC_function_vcl_recvâ:
./vcl.1P9zoqAU.c:485: warning: incompatible implicit declaration of built-in function âsprintfâ
@luke just delete the two #include in the configuration and it will work.
I’m using your configuration and it’s working quite well. the only problem is when someone changes a url in magento (eg: base url, link url, new cms page etc), the server doesn’t return the page until I reload varnish and the http server (nginx). does anyone anyone know how to change the configuration to avoid this problem?
tried and fail again :9
wish I could debug the weird letters.
Message from C-compiler:
./vcl.1P9zoqAU.c: In function âVGC_function_vcl_recvâ:
./vcl.1P9zoqAU.c:483: warning: incompatible implicit declaration of built-in function âsprintfâ
Using old SHMFILE
@Luke, you may have figured this out: but sometime when coping from a web page invisible characters can also be copied, check the code in a good editor and insure your file encoding is say UTF8 and set the editor to ‘show invisible’ or something like that (or count the characters, you will see the count jump without the cursor move), in the case above you can see where the rouge characters are, hope this helps (I have wasted way to much time on this in the past)
Ta, Carl.
i’ll say “WOW”,
i’m looking for a varnish/magento solution since 1 year.
i’ll try your code tonight
doesn’t seem to work with magento 1.4 … (magento crash when i activated the varnish magento module).
can somebody help me ?
Hello Alexandre,
I will check it’s functionality with Magento 1.4 and let you know about the issue.
Hi
I am testing this and am always getting some cross session issues. I can replicate this by clearing all cache data and cookies. In one firefox browser navigate to a page to get the frontend cookie, navigate to another page to get it cached, add something to cart, navigate to another page.
in second instance firefox browser (different profile not just a new window) navigate to page cached in previous instructions, this should be the first hit to the site, you should see the same cookie value for the frontend cookie the other browser and thus have items in your cart.
Have you had any experience of this?
cheers
Hey guys, great information here!
Any news about running Varnish with Magento 1.4 ?
I got a strange error on the Observer.php module… any idea of what it could be?
Look at the error:
/** * Varnish Observer model * * @category Varnish * @package Varnish */ class Varnish_Model_Observer { private $_request = null; public function __construct() { } private function getSecureKey() { return Mage::getStoreConfig(‘varnish/purgeall_key/key’); } public function getCookie() { return Mage::app()->getCookie(); } public function varnish($observer) { if ($this->isSetNoCacheStable()) { return false; } if ($this->pollVerification()) { $this->setNoCacheStable(); return false; } if ($this->quoteHasItems()) { $this->turnOffVarnishCache(); return false; } if ($this->customerIsLogged()) { $this->turnOffVarnishCache(); return false; } $this->turnOnVarnishCache(); } public function turnOffVarnishCache() { $this->getCookie()->set(‘nocache’, 1); } public function turnOnVarnishCache() { $this->getCookie()->delete(‘nocache’); } public function quoteHasItems() { $quote = Mage::getSingleton(‘checkout/session’)->getQuote(); if ($quote instanceof Mage_Sales_Model_Quote && $quote->hasItems()) { return true; } } public function customerIsLogged() { $customerSession = Mage::getSingleton(‘customer/session’); if ($customerSession instanceof Mage_Customer_Model_Session && $customerSession->isLoggedIn()) { return true; } } public function pollVerification() { $justVotedPollId = (int) Mage::getSingleton(‘core/session’)->getJustVotedPoll(); if ($justVotedPollId) { return true; } } public function setNoCacheStable() { $this->getCookie()->set(‘nocache_stable’, 1, 0); } public function isSetNoCacheStable() { return $this->getCookie()->get(‘nocache_stable’) === 1; } public function purgeAll() { try { $url = Mage::getBaseUrl().’varnish/index/purgeall/key/’.$this->getSecureKey().’/'; $ch = curl_init(); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, ‘PURGE’); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); $responseBody = curl_exec($ch); curl_close ($ch); } catch (Exception $e) { Mage::log($e->getFile().’ ‘.$e->getLine().’ ‘.$e->getMessage()); } } }
Fatal error: Call to a member function varnish() on a non-object in /var/www/httpdocs/app/code/core/Mage/Core/Model/App.php on line 1239
good project, it can help magento became faster use cache; how about the magento 1.4.0.1 work with the method
Awesomes, thanks for putting this together
#15: add <?php to the top of it.
Verification and Varnish might fail becuase of this Varnish known issue: http://www.varnish-cache.org/trac/ticket/432
Use with care…
Hi
I have this error:
Warning: include(Varnish/Model/Observer.php) [function.include]: failed to open stream: No such file or directory in /usr/home/bebecor.es/web/lib/Varien/Autoload.php on line 93.
Hi.
I’m installed Varnish in magento.
I’ve done every step of the installation, but i don’t know where is “sudo vim / etc / varnish.vcl” nor i’ve to do with.
#20 Solved
how did you solve #20?
This is working on magento 1.4.1.1?
I guess here, observer isn’t working, cause I can’t see the nocache cookie seted. Even if I modify the code and put garbage on it, magento runs fine, what makes me think the code isnt’ called at all.
Any clues someone?
How is possible to run it? I made changes in Magento and install Varnish properly. There were few errors in varnish.vcl file. I have to remove
# purge_hash(“.*”);
And
#include
#include
After that varnish started properly. And is listening on port. I am running nginx. How can I start nginx with Varnish support?
Some ideas how to extend this this here: http://moprea.ro/2011/feb/16/magento-performance-optimization-varnish-cache-2/
Purges only urls that needs to be refreshed
I am trying to get this up and running and I have a few issues. It looks like in my headers I have a few cookies and then there are some cache control headers.
Can anyone tell me how to get Varnish to ignore these things?
Hi Josh,
Be careful with Varnish because it is standard compliant and it will listen to your cache control headers.
If you really want to instruct Varnish to ignore headers or cookies, use something like this in your Varnish configuration:
sub vcl_recv {
//…
// you decide to cache
# Remove cookie
unset req.http.Cookie;
return (lookup);
}
Here you can manipulate response headers (modify it to your needs):
sub vcl_fetch {
set obj.grace = 300s;
set obj.http.X-Cache-Set = “NO:Not-Cacheable”;
if (obj.status >= 300) {
// if the request from backend is not 2xx, don’t cache
set obj.http.X-Cache-Set = “NO: Non 200 Code”;
pass;
}
## Cache static content
if (req.request == “GET” && (
req.url ~ “\.(gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf)$”
|| req.url ~ “\.(svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$”
|| req.url ~ “\.(js|css|xml|txt|pdf|pls|torrent|gz|zip|rar|bz2|tgz|tbz)$”
|| req.url ~ “^/store/(skin|js|media)/.*”
)) {
set obj.ttl = 1800s;
set obj.http.Cache-Control = “max-age = 1800″;
set obj.cacheable = true;
set obj.http.X-Cache-Set = “YES: Static”;
# remove cache headers set by Magento
unset obj.http.Pragma;
unset obj.http.Set-Cookie;
}
I recommend you to enable this and check X-Cache header:
## Called before a cached object is delivered to the client
#
sub vcl_deliver {
set resp.http.X-Served-By = “add FE name here”;
if (obj.hits > 0) {
set resp.http.X-Cache = “HIT”;
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = “MISS”;
}
}
New version of Varnish expose only some specific objects in these functions. Please check the docs, http://www.varnish-cache.org/trac/wiki/VCL.
Hope this is a good start to hack your Varnish configuration.
Good luck
Thank you for your code ~~~
You’re great !
[...] I am following the example at: http://www.kalenyuk.com.ua/magento-performance-optimization-with-varnish-cache-47.html [...]
Is possible to install it on Magento CE 1.6?
Hi
I am running magento v1.6.2 on Apache in a shared hosting environment. Varnish 3 is installed on the server. Because it is a shared environment I have no access to the server level Varnish .vcl files. Right now my site is talking with Varnish but Varnish is not serving any cached pages. Will your setup work in a shared hosting environment and if I create your /etc/varnish.vcl in the etc folder under my /public_html folder will this suffice?
Many Thanks
Summer thanks…
Hey like your efforts have a look of mine…
What was the fix for #20. Jesus claims that he fixed it, but I don’t see what the solution was.
Thanks. Actually, it removes _authorize_me inalotinntely in the example; the idea was to ask for e.g. /authorized_content/foobar.flv_authorize_me, process the authorization and then request /authorized_content/foobar.flv from the real backend. Obviously this is just a silly example.
fajne/smart. we’ve been doing this with esi so far, but this one require way less efofrt from webapp. if i understand this correctly it should remove authorized_content prefix in line 8 and not _authorize_me.
[...] to present some ideas that extend the work of Kalenyuk’s presented in a blog post called Magento performance optimization with Varnish cache: granular Varnish purging, [...]
I’ll immediately clutch your rss feed as I can not in finding your email subscription link or newsletter service. Do you’ve any? Kindly allow me realize in order that I could subscribe. Thanks.
It’s in reality a nice and helpful piece of information. I am satisfied that you simply shared this useful information with us. Please stay us informed like this. Thanks for sharing.
Hello there, I found your web site by means of Google even as looking for a related topic, your website came up, it seems to be good. I’ve bookmarked to favourites|added to my bookmarks.
of course like your web site but you have to take a look at the spelling on several of your posts. Many of them are rife with spelling issues and I find it very troublesome to tell the truth however I’ll certainly come back again.
I like the helpful info you provide for your articles. I will bookmark your weblog and check again here regularly. I’m somewhat certain I will be told lots of new stuff proper here! Good luck for the following!
i have been spent a lot of time try in my site but it doesnt work. i finally using wordpress http://webrada.com/
Thanks , I have recently been searching for information approximately this topic for a long time and yours is the best I’ve discovered so far. But, what about the conclusion? Are you certain in regards to the source?|What i do not understood is in reality how you’re no longer really much more well-liked than you may be now. You’re very intelligent.