How we overcame Magento 2 challenges and built a superfast solution

In ecommerce, speed is important; there’s nothing new about that. As a rule of thumb, you should not keep the user waiting for more than 3 secs, otherwise, there is a good chance she will move on to a more responsive competitor.

A two-second delay in web page load time increase bounce rates by 103 percent.

Akamai (source)

While cached pages are served in literally no time, all pages cannot be cached. A good example are category pages with dynamic filtering systems. Moreover, managing categories and product data by store managers will purge the cache. The difference in loading times of cached and uncached pages confuses the user and tends to grow exponentially as the number of page view requests rises.

For a smooth user experience, it is vital to keep response time of uncached pages as low as possible. However, as more and more features are added to the store in the name of that very same user experience, the speed is usually one of the first victims. Wouldn’t it be great to have both speed and features?

First, the results

I am happy to say that our team has delivered one of the fastest Magento 2 enterprise shops out there; squeezing the server response time of uncached pages to less than 0.7 sec.  

Leveraging www.webpagetest.org, we have compared the loading times of “our” solution for Merkur—Magento 2 Enterprise edition https://www.merkur.si/

with two other recently designed M2 shops:

• Intersport—Magento 2 Community Edition: https://www.intersport.si/  and
• Helly Hansen—Magento 2 Enterprise Edition: https://www.hellyhansen.com/

While all the shops have TTFB (Time To First Byte) response times of uncached sites roughly in the acceptable time frames, their speed still varies significantly. Here are the results (less is better):

The testing method: Right-click anywhere on the page and select “inspect” from the menu. To make sure you are testing an uncached page, type a question mark followed by a few random characters (eg. “?test1” or “?test2”) at the end of the url string being tested. Alter this parameter in every test. Example: "https://www.merkur.si/?test3".

This comparison makes me proud of my team, really. But the path to these results was not an easy one. Here is our story.

The Challenge

Creatim was assigned with building a new online store for Merkur (www.merkur.si), an established DIY retailer in SE Europe. The company struggled with its existing, over-customized Magento store. Based on Magento 2 Enterprise edition, the new solution should aim for a better user experience and more room for innovative merchandising initiatives.

We approached the project with due respect since it was the first M2 solution on our agenda. However, the development environment was similar to M1, we were moving on according to plan, everything seemed to be working fine. Perhaps too fine. We clearly lost a bit of the initial edge, skipping some of the testing in order to catch up with the deadlines. True, there were some warning signs indicating the performance was rather slow. We assumed it was about the constraints of the testing environment and that once we reached the production environment these issues will just fade out.

However, in the production environment it didn’t get any better, in fact, it got worse.

The response times were 5 seconds and more. Simulating user peaks made the solution simply stall. Something down there went terribly wrong and we had no idea where the crux was. It was an embarrassing situation.

It looked like the problem was bigger than us. We contacted all Magento experts we could think of, hired Magento consulting service and, of course, Magento enterprise support. I guess we unrealistically expected that some Magento magician out there will just pull all the answers out of the hat. Yet no one could sort out the problem. Considering the feedback, the solution seemed to be in accordance with the Magento guidelines and recommendations and should be working fine.

Well, it wasn’t.

We were on our own. We went to the client and put the cards on the table. We needed more time. There was simply no easy way out, but we were determined to get solution in check, so they agreed to give us a second chance.

Luckily, we sorted out the main culprit soon after. The facet navigation module appeared to be the main cause of many performance issues we were struggling with.

I am not going to do bad advertising here, I am sure the guys are already working hard on improving their module to bring its performance up to standard.

However, I certainly want to wow the alternative.

We replaced the initial module with a new solution based on Elasticsearch (ES). Everyone who is familiar with the ES knows it does much more than just providing a “search” function. In M2, it can play a pivotal role in the catalogue, relieving the database and speeding up the page view response times. And while the front-end part of its multi-filter component may not look as fancy as some of the competitors’, its speed is persuasive.

So we redesigned much of the solution around Elasticsearch; performing many improvements along the dataroad while rigorously testing all aspects of solution development. Now, the test results were telling a different story.

Here’s what we have learned:

Magento Performance Rule No. 1: Do not slow it down.

Despite a barrage of complaints on the web regarding its performance, Magento 2 is actually not that slow. What really thwarts its speed are the modules that are being added to the core platform in order to improve the customer experience. The quality of these modules can vary significantly. What works well in M1 does not necessary perform in M2. Since the modules come from various vendors, they may not get along with each other smoothly just out of the box.

Hence, limiting 3rd party modules to the smallest number possible is a well-known first step in the battle for milliseconds. For many features cases, custom development can be a more viable alternative. Still, there are certain components like Layered navigation or Advanced search that would require too much effort to build from scratch. But as long as the number of such components are kept in check, there is no appealing reason for doing so anyway.

During the solution development, it is vital to continuously monitor how much the 3rd-party-added modules as well as custom-tailored components affect the time to first byte (TTFB). Test every feature added, and if you find out it is compromising the overall performance, fix it, redesign it, or simply replace it. Fight for every millisecond! Redesign the dataflow, test every line of the code—just get to the bottom of the problem while still on track. Having to reach back later in the process can cause a domino effect, forcing you to revamp much of the application logic already in place.

Magento Performance Rule Nr 2: Make it faster.

Luckily, in Magento 2 EE, the implementation of Elasticsearch is much better supported than in Magento 1. Still, making substantial changes to the solution once already in the finish line is always a daunting task, but in our case, it was worth it. We were hoping to alleviate, at least, some of the most notorious performance issues. But we refused to stop there. Learning many hidden M2 ropes along the way, we have turned a flat-tire pick-up into a racing car. How?

In addition to choosing the right components, there are many decisive factors. Here is what worked for us.

The development team

People matter. Considering the scale of adjustments, we lost just two months to get the solution back on track. Yes, people make mistakes. However, having experienced developers on board means they will not get stuck in escalated situation. They will find a way out eventually.

Testing

Encourage rigorous testing in every step of the process. Plan enough testing time. Test everything, even processes and components you think should work fine (because they always have). Test them, optimize them, then test them again.

The infrastructure

Magento enterprise is a hungry beast, so don’t scrimp on server capacity. But it is not just about capacity; go deeper.

After the initial failure, we decided to rethink everything, including the technical infrastructure. Our administrators compared various types of bare metal hosting offerings. The final infrastructure was based on processors that proved to be twice as fast as the initial ones. Furthermore, they implemented a spider that crawls the servers and moves all the product data into cache every morning, thus reducing the TTFB as the first users knock on the door.

Conclusion

There are a lot of articles out there that aim to give advice on how to boost Magento 2 performance. Use Varnish, GZIP compression, reduce picture sizes, etc, etc.

All that matters, no doubt. However, an effective ecommerce solution is more than just a sum of its components.

Just as food ingredients alone do not yet make a recipe, merely choosing the right components to meet the solution requirements is not enough. Even if you picked up only high quality ones (good luck with that) their interaction may not run seamlessly.  Knowing how to streamline the data through these components effectively will not only preserve the overall performance but also substantially speed up the system as a whole.

In ecommerce, speed is important; there’s nothing new about that. As a rule of thumb, you should not keep the user waiting for more than 3 secs, otherwise, there is a good chance she will move on to a more responsive competitor.