Magento performance optimization with Varnish cache

OVERVIEW

After installing Magento on hosting you probably have a desire to optimize its work.

Google offers solutions, which consist on using one of the following options: installing higher-performance hardware, correction for mysql/php configuration, using PHP accelerators, accelerate downloads JS and CSS, the correction assignment expires date for content, customization Magento core files to fit your needs.

Varien released a book titled “Enterprise Edition Whitepaper High Performance eCommerce” which explains the above methods of  Magento optimization. I recommend reading this book and use the described methods of optimization.

I propose another new approach for the optimization of Magento, which was not described in the Internet before. I propose to use Varnish for caching pages.

What is Varnish you can read here. Varnish handles the request, looks in the cache this page and, if such page is found, returns it. If the page is not found in the cache – a request sent to Apache. This approach allows you to create any number of dynamic pages.

EXAMPLE

On the main page you can see poll module. Assume that we have 4 different polls. I propose randomly create 100 different copies of the main page. When user loads main page Varnish gives random copy of the main page. The probability of withdrawal of one of the 4 polls close to 25%.

This caching will give us the static pages to users who simply browse the site and did not do such actions as a vote, adding product to cart, login, etc. Majority of these users, so using this approach can significantly reduce the load on Apache. Once a user has voted, logged, added product to cart, etc. Varnish cache turn off.

INSTALLATION

Download the latest version of Magento and install it.

Create folder app/code/local/Varnish.

Create file app/etc/modules/Varnish.xml :

<?xml version="1.0"?>
<config>
  <modules>
    <Varnish>
      <active>true</active>
      <codePool>local</codePool>
    </Varnish>
  </modules>
</config>

app/code/local/Varnish/controllers/IndexController.php :

<?php
class Varnish_IndexController extends Mage_Core_Controller_Front_Action
{
 public function purgeallAction() { }
}

app/code/local/Varnish/etc/config.xml :

<?xml version="1.0"?>
<config>
    <modules>
       <Varnish>
         <version>0.0.1</version>
       </Varnish>
    </modules>
    <global>
        <models>
            <varnish>
              <class>Varnish_Model</class>
              <resourceModel>varnish_mysql4</resourceModel>
            </varnish>
        </models>
        <events>
            <http_response_send_before>
                <observers>
                    <varnish>
                        <type>singleton</type>
                        <class>varnish/observer</class>
                        <method>varnish</method>
                    </varnish>
                </observers>
            </http_response_send_before>
                <application_clean_cache>
                    <observers>
                        <varnish>
                            <type>singleton</type>
                            <class>varnish/observer</class>
                            <method>purgeAll</method>
                        </varnish>
                    </observers>
                </application_clean_cache>
        </events>
    </global>
    <frontend>
       <routers>
           <varnish>
               <use>standard</use>
               <args>
                   <module>Varnish</module>
                   <frontName>varnish</frontName>
               </args>
           </varnish>
       </routers>
    </frontend>
    <default>
       <varnish>
           <purgeall_key>
               <key>#Fj1nzljh</key>
           </purgeall_key>
       </varnish>
    </default>
</config>

app/code/local/Varnish/Model/Observer.php :

/**
 * Varnish Observer model
 *
 * @category   Varnish
 * @package    Varnish
 */
class Varnish_Model_Observer
{
    private $_request = null;

    public function __construct()
    {

    }

    private function getSecureKey()
    {
        return Mage::getStoreConfig('varnish/purgeall_key/key');
    }

    public function getCookie()
    {
        return Mage::app()->getCookie();
    }

    public function varnish($observer)
    {
        if ($this->isSetNoCacheStable()) {
            return false;
        }

        if ($this->pollVerification()) {
            $this->setNoCacheStable();
            return false;
        }

        if ($this->quoteHasItems()) {
            $this->turnOffVarnishCache();
            return false;
        }

        if ($this->customerIsLogged()) {
            $this->turnOffVarnishCache();
            return false;
        }

        $this->turnOnVarnishCache();
    }

    public function turnOffVarnishCache()
    {
        $this->getCookie()->set('nocache', 1);
    }

    public function turnOnVarnishCache()
    {
        $this->getCookie()->delete('nocache');
    }

    public function quoteHasItems()
    {
        $quote = Mage::getSingleton('checkout/session')->getQuote();
        if ($quote instanceof Mage_Sales_Model_Quote && $quote->hasItems()) {
            return true;
        }
    }

    public function customerIsLogged()
    {
        $customerSession = Mage::getSingleton('customer/session');
        if ($customerSession instanceof Mage_Customer_Model_Session  &&
            $customerSession->isLoggedIn()) {
            return true;
        }
    }

    public function pollVerification()
    {
        $justVotedPollId = (int) Mage::getSingleton('core/session')->getJustVotedPoll();
        if ($justVotedPollId) {
            return true;
        }
    }

    public function setNoCacheStable()
    {
        $this->getCookie()->set('nocache_stable', 1, 0);
    }

    public function isSetNoCacheStable()
    {
        return $this->getCookie()->get('nocache_stable') === 1;
    }

    public function purgeAll()
    {
        try {
            $url = Mage::getBaseUrl().'varnish/index/purgeall/key/'.$this->getSecureKey().'/';
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PURGE');
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
            $responseBody = curl_exec($ch);
            curl_close ($ch);
        } catch (Exception $e) {
            Mage::log($e->getFile().' '.$e->getLine().' '.$e->getMessage());
        }
    }
}

sudo vim /etc/varnish.vcl :

backend apache {
  .host = "127.0.0.1";
  .port = "8088";
  .max_connections = 30;
}

acl purge {
  "localhost";
}

sub vcl_recv {
  #purge all
  if (req.request == "PURGE") {
    if (!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    if (req.url ~ "varnish/index/purgeall/key/#Fj1nzljh") {
      purge_hash( ".*" );
    }
  }

  #described below need to create random "DynamicPage" value; this functionallity we use
  #to create several caches for 1 dynamic page.
  #for example, "int i = rand() % 2 + 1;" means that we have 2 different copies of one dynamic page
  C{
   #include
   #include
   char buffer [33];
   int i = rand() % 10 + 1;
   sprintf(buffer, "%d", i);
   VRT_SetHdr(sp, HDR_REQ, "\014DynamicPage:", buffer, vrt_magic_string_end);
  }C

  if (req.request != "GET" &&
  req.request != "HEAD" &&
  req.request != "PUT" &&
  req.request != "POST" &&
  req.request != "TRACE" &&
  req.request != "OPTIONS" &&
  req.request != "DELETE") {
    return (pipe);
  }

  # do not cache POST requests
  if (req.request == "POST") {
    return (pipe);
  }

  #we should not cache any page for Magento backend
  if (req.request == "GET" && (req.url ~ "^/admin") || req.url ~ "^/index.php/admin") {
    return (pass);
  }

  #we should not cache any page for checkout and customer modules
  if (req.request == "GET" && (req.url ~ "^/checkout" || req.url ~ "^/customer")) {
    return (pass);
  }

  #do not cache till session end
  if (req.http.cookie ~ "nocache_stable") {
    return (pass);
  }

  #unique identifier witch tell Varnish use cache or not
  if (req.http.cookie ~ "nocache") {
    return (pass);
  }

  if (req.request == "GET" && (req.url ~ "\.(png|jpg|jpeg|gif)$" || req.url ~ "print.css")) {
    lookup;
  }

  #Even though there are few possible values for Accept-Encoding, Varnish treats
  #them literally rather than semantically, so even a small difference which makes
  #no difference to the backend can reduce cache efficiency by making Varnish cache
  #too many different versions of an object.
  #http://varnish.projects.linpro.no/wiki/FAQ/Compression
  if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
      # No point in compressing these
      remove req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
      set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate") {
      set req.http.Accept-Encoding = "deflate";
    } else {
      # unkown algorithm
      remove req.http.Accept-Encoding;
    }
  }

  return (lookup);
}

sub vcl_pipe {
  # Note that only the first request to the backend will have
  # X-Forwarded-For set.  If you use X-Forwarded-For and want to
  # have it set for all requests, make sure to have:
  # set req.http.connection = "close";
  # here.  It is not set by default as it might break some broken web
  # applications, like IIS with NTLM authentication.
  return (pipe);
}

sub vcl_pass {
  return (pass);
}

sub vcl_hit {
  if (!obj.cacheable) {
    return (pass);
  }
  return (deliver);
}

sub vcl_miss {
  return (fetch);
}

sub vcl_fetch {
  return (deliver);
}

sub vcl_deliver {

  if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
}

sub vcl_error {
 set obj.http.Content-Type = "text/html; charset=utf-8";
 synthetic {"
 <?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
 <head>
 <title>"} obj.status " " obj.response {"</title>
 </head>
 <body>
 <h1>Error "} obj.status " " obj.response {"</h1>
 <p>"} obj.response {"</p>
 <h3>Guru Meditation:</h3>
 <p>URL: "} obj.http.bhash {"</p>
 <p>XID: "} req.xid {"</p>
 <hr>
 <address>
 <a href="http://www.varnish-cache.org/">Varnish cache server</a>
 </address>
 </body>
 </html>
 "};
 return (deliver);
}

59 Comments

  • Filipe |

    very interesting article. I didn’t try the code myself yet but I hope to try it soon.
    do you have any benchmarks available?

  • Shiloh |

    Thank you for sharing this info. It looks very promising.

    I was wondering if you have any production sites running this way that you would mind sharing?
    Perhaps some benchmark data?

  • admin |

    Hello Filipe and Shiloh!

    Thank you for your interest in Varnish.

    At the moment I do not have a dedicated server on the Internet with Varnish.

    When we use native cache, Magento’s home page is loaded in about 1.4-1.6 seconds.
    I thoroughly tested it on the local machine and got the result: page is loaded in about 0.3-0.4 seconds.

    All the code in the article checked on a real project.

  • Luke |

    Hi I run into this problem.

    root@server [/usr/local/sbin]# ./varnishd -a 127.0.0.1:8088 -f /etc/varnish.vcl -s file,/var/cache/varnish.cache,512M
    storage_file: filename: /var/cache/varnish.cache size 512 MB.
    Message from VCC-compiler:
    No backends or directors found in VCL program, at least one is necessary.
    Running VCC-compiler failed, exit 1
    VCL compilation failed

  • Luke |

    I solved the problem but another one came up

    storage_file: filename: /var/cache/varnish.cache size 512 MB.
    Message from C-compiler:
    ./vcl.1P9zoqAU.c:481:12: error: #include expects “FILENAME” or
    ./vcl.1P9zoqAU.c:482:12: error: #include expects “FILENAME” or
    ./vcl.1P9zoqAU.c: In function ?VGC_function_vcl_recv?:
    ./vcl.1P9zoqAU.c:485: warning: incompatible implicit declaration of built-in function ?sprintf?

  • Filipe |

    @luke just delete the two #include in the configuration and it will work.

    I’m using your configuration and it’s working quite well. the only problem is when someone changes a url in magento (eg: base url, link url, new cms page etc), the server doesn’t return the page until I reload varnish and the http server (nginx). does anyone anyone know how to change the configuration to avoid this problem?

  • Luke |

    tried and fail again :9
    wish I could debug the weird letters.

    Message from C-compiler:
    ./vcl.1P9zoqAU.c: In function ?VGC_function_vcl_recv?:
    ./vcl.1P9zoqAU.c:483: warning: incompatible implicit declaration of built-in function ?sprintf?
    Using old SHMFILE

  • Carl |

    @Luke, you may have figured this out: but sometime when coping from a web page invisible characters can also be copied, check the code in a good editor and insure your file encoding is say UTF8 and set the editor to ‘show invisible’ or something like that (or count the characters, you will see the count jump without the cursor move), in the case above you can see where the rouge characters are, hope this helps (I have wasted way to much time on this in the past)

    Ta, Carl.

  • Alexandre Derumier |

    i’ll say “WOW”,

    i’m looking for a varnish/magento solution since 1 year.

    i’ll try your code tonight 😉

  • Alexandre Derumier |

    doesn’t seem to work with magento 1.4 … (magento crash when i activated the varnish magento module).
    can somebody help me ?

  • admin |

    Hello Alexandre,

    I will check it’s functionality with Magento 1.4 and let you know about the issue.

  • shaun |

    Hi

    I am testing this and am always getting some cross session issues. I can replicate this by clearing all cache data and cookies. In one firefox browser navigate to a page to get the frontend cookie, navigate to another page to get it cached, add something to cart, navigate to another page.

    in second instance firefox browser (different profile not just a new window) navigate to page cached in previous instructions, this should be the first hit to the site, you should see the same cookie value for the frontend cookie the other browser and thus have items in your cart.

    Have you had any experience of this?

    cheers

  • Thiago |

    Hey guys, great information here!

    Any news about running Varnish with Magento 1.4 ?

  • Thiago |

    I got a strange error on the Observer.php module… any idea of what it could be?

  • Thiago |

    Look at the error:

    /** * Varnish Observer model * * @category Varnish * @package Varnish */ class Varnish_Model_Observer { private $_request = null; public function __construct() { } private function getSecureKey() { return Mage::getStoreConfig(‘varnish/purgeall_key/key’); } public function getCookie() { return Mage::app()->getCookie(); } public function varnish($observer) { if ($this->isSetNoCacheStable()) { return false; } if ($this->pollVerification()) { $this->setNoCacheStable(); return false; } if ($this->quoteHasItems()) { $this->turnOffVarnishCache(); return false; } if ($this->customerIsLogged()) { $this->turnOffVarnishCache(); return false; } $this->turnOnVarnishCache(); } public function turnOffVarnishCache() { $this->getCookie()->set(‘nocache’, 1); } public function turnOnVarnishCache() { $this->getCookie()->delete(‘nocache’); } public function quoteHasItems() { $quote = Mage::getSingleton(‘checkout/session’)->getQuote(); if ($quote instanceof Mage_Sales_Model_Quote && $quote->hasItems()) { return true; } } public function customerIsLogged() { $customerSession = Mage::getSingleton(‘customer/session’); if ($customerSession instanceof Mage_Customer_Model_Session && $customerSession->isLoggedIn()) { return true; } } public function pollVerification() { $justVotedPollId = (int) Mage::getSingleton(‘core/session’)->getJustVotedPoll(); if ($justVotedPollId) { return true; } } public function setNoCacheStable() { $this->getCookie()->set(‘nocache_stable’, 1, 0); } public function isSetNoCacheStable() { return $this->getCookie()->get(‘nocache_stable’) === 1; } public function purgeAll() { try { $url = Mage::getBaseUrl().’varnish/index/purgeall/key/’.$this->getSecureKey().’/’; $ch = curl_init(); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, ‘PURGE’); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); $responseBody = curl_exec($ch); curl_close ($ch); } catch (Exception $e) { Mage::log($e->getFile().’ ‘.$e->getLine().’ ‘.$e->getMessage()); } } }
    Fatal error: Call to a member function varnish() on a non-object in /var/www/httpdocs/app/code/core/Mage/Core/Model/App.php on line 1239

  • max |

    good project, it can help magento became faster use cache; how about the magento 1.4.0.1 work with the method

  • jesus |

    Hi
    I have this error:
    Warning: include(Varnish/Model/Observer.php) [function.include]: failed to open stream: No such file or directory in /usr/home/bebecor.es/web/lib/Varien/Autoload.php on line 93.

  • Jes?s |

    Hi.
    I’m installed Varnish in magento.
    I’ve done every step of the installation, but i don’t know where is “sudo vim / etc / varnish.vcl” nor i’ve to do with.

  • Andr? Zaiats |

    This is working on magento 1.4.1.1?

    I guess here, observer isn’t working, cause I can’t see the nocache cookie seted. Even if I modify the code and put garbage on it, magento runs fine, what makes me think the code isnt’ called at all.

    Any clues someone?

  • Jan |

    How is possible to run it? I made changes in Magento and install Varnish properly. There were few errors in varnish.vcl file. I have to remove

    # purge_hash(“.*”);
    And
    #include
    #include

    After that varnish started properly. And is listening on port. I am running nginx. How can I start nginx with Varnish support?

  • Josh Pennington |

    I am trying to get this up and running and I have a few issues. It looks like in my headers I have a few cookies and then there are some cache control headers.

    Can anyone tell me how to get Varnish to ignore these things?

  • Mario |

    Hi Josh,

    Be careful with Varnish because it is standard compliant and it will listen to your cache control headers.

    If you really want to instruct Varnish to ignore headers or cookies, use something like this in your Varnish configuration:

    sub vcl_recv {
    //…
    // you decide to cache
    # Remove cookie
    unset req.http.Cookie;

    return (lookup);
    }

    Here you can manipulate response headers (modify it to your needs):
    sub vcl_fetch {
    set obj.grace = 300s;
    set obj.http.X-Cache-Set = “NO:Not-Cacheable”;

    if (obj.status >= 300) {
    // if the request from backend is not 2xx, don’t cache
    set obj.http.X-Cache-Set = “NO: Non 200 Code”;
    pass;
    }

    ## Cache static content
    if (req.request == “GET” && (
    req.url ~ “\.(gif|jpg|jpeg|bmp|png|tiff|tif|ico|img|tga|wmf)$”
    || req.url ~ “\.(svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$”
    || req.url ~ “\.(js|css|xml|txt|pdf|pls|torrent|gz|zip|rar|bz2|tgz|tbz)$”
    || req.url ~ “^/store/(skin|js|media)/.*”
    )) {
    set obj.ttl = 1800s;
    set obj.http.Cache-Control = “max-age = 1800”;
    set obj.cacheable = true;
    set obj.http.X-Cache-Set = “YES: Static”;
    # remove cache headers set by Magento
    unset obj.http.Pragma;
    unset obj.http.Set-Cookie;
    }

    I recommend you to enable this and check X-Cache header:
    ## Called before a cached object is delivered to the client
    #
    sub vcl_deliver {

    set resp.http.X-Served-By = “add FE name here”;

    if (obj.hits > 0) {
    set resp.http.X-Cache = “HIT”;
    set resp.http.X-Cache-Hits = obj.hits;
    } else {
    set resp.http.X-Cache = “MISS”;
    }
    }

    New version of Varnish expose only some specific objects in these functions. Please check the docs, http://www.varnish-cache.org/trac/wiki/VCL.

    Hope this is a good start to hack your Varnish configuration.

    Good luck 😉

  • Ian Ryan |

    Hi

    I am running magento v1.6.2 on Apache in a shared hosting environment. Varnish 3 is installed on the server. Because it is a shared environment I have no access to the server level Varnish .vcl files. Right now my site is talking with Varnish but Varnish is not serving any cached pages. Will your setup work in a shared hosting environment and if I create your /etc/varnish.vcl in the etc folder under my /public_html folder will this suffice?

    Many Thanks

  • Trevor |

    What was the fix for #20. Jesus claims that he fixed it, but I don’t see what the solution was.

  • Bruno |

    Thanks. Actually, it removes _authorize_me inalotinntely in the example; the idea was to ask for e.g. /authorized_content/foobar.flv_authorize_me, process the authorization and then request /authorized_content/foobar.flv from the real backend. Obviously this is just a silly example. :-)

  • Bill |

    fajne/smart. we’ve been doing this with esi so far, but this one require way less efofrt from webapp. if i understand this correctly it should remove authorized_content prefix in line 8 and not _authorize_me.

  • large rugs clearance |

    I’ll immediately clutch your rss feed as I can not in finding your email subscription link or newsletter service. Do you’ve any? Kindly allow me realize in order that I could subscribe. Thanks.

  • hall runners rugs |

    It’s in reality a nice and helpful piece of information. I am satisfied that you simply shared this useful information with us. Please stay us informed like this. Thanks for sharing.

  • Fastest Automated Blog Commenter |

    Hello there, I found your web site by means of Google even as looking for a related topic, your website came up, it seems to be good. I’ve bookmarked to favourites|added to my bookmarks.

  • motorhomes |

    of course like your web site but you have to take a look at the spelling on several of your posts. Many of them are rife with spelling issues and I find it very troublesome to tell the truth however I’ll certainly come back again.

  • Best Hacks, Cracks, Keygens, Mutlipacks on the Web ! |

    I like the helpful info you provide for your articles. I will bookmark your weblog and check again here regularly. I’m somewhat certain I will be told lots of new stuff proper here! Good luck for the following!

  • Pizza Woodland Hills |

    Thanks , I have recently been searching for information approximately this topic for a long time and yours is the best I’ve discovered so far. But, what about the conclusion? Are you certain in regards to the source?|What i do not understood is in reality how you’re no longer really much more well-liked than you may be now. You’re very intelligent.

  • acne |

    My pal advisable I’ll this way blog site. This individual has been completely correct. The following distribute in fact built my morning. You can not think about the amount of occasion I had created expended for this data! Thanks!

  • Testerone |

    Hi my family member! I wish to say that this post is awesome, nice written and come with approximately all important infos. I’d like to peer extra posts like this .

  • Web Design Agencies Shrewsbury |

    You are truly a excellent webmaster. The web site loading velocity is amazing. It seems that you’re doing any unique trick. Moreover, The contents are masterwork. you’ve performed a magnificent task on this matter!

  • mekanlar istanbul |

    Woah this weblog is great i like studying your articles. Stay up the good paintings! You recognize, a lot of people are searching around for this info, you can help them greatly.

  • ????????????????????????? ?? - ????? |

    I have been surfing on-line more than three hours these days, but I by no means discovered any fascinating article like yours. It’s lovely worth enough for me. In my view, if all website owners and bloggers made excellent content material as you probably did, the net will be much more useful than ever before.

  • carpet , Carpets , Persian Carpet |

    Somebody essentially lend a hand to make critically posts I might state. This is the first time I frequented your website page and so far? I amazed with the analysis you made to create this particular publish amazing. Great job!

So, what do you think ?