Scaling Day-By-Day

As a consultant, I deal with companies of all sizes. From new startups all the way up to large conglomerates. And I am constantly amazed at the old-world view to application scalability and performance.

When I ask how they will scale to thousands of users, or how to achieve millions or billions of transactions; I end up hearing statements such as the following:

We will tune it at the end.

We added another tier in our architecture so we could add more hardware, so we aren’t worried about it.

We will wait until we get to that point as we can’t afford to scale now.

We will buy 64bit machines.

What? I thought you said 100 users, not 100 million!

They seem unaware that scalability has many facets, and some haunt them from the first draft of an architecture all the way through each milestone of coding. And many can be resolved without spending money on hardware or expensive software, or even without derailing the project schedule. Here are some of my scalability tips that should be applied at all times of a project’s life.

Build Your Technical Team From The Top Down

Bring someone into the team that has experience with scalability. Give them time to get an architecture, design work and best practices in place before you flood them with less experienced engineers that they need to caretake. If they are buried in work, they cannot feed the rest of your team and keep an eye on the overall project. And yes, your leads should caretake the project at all times for scalability, security, robustness, structure, test coverage, and general health of code.

And while you are at it, have frequent team code inspections

Scalability can break at minute levels. Be careful that developers are not wandering off the path of scalability with bad transaction handling, non-performant code, not understanding the best use of a database, or other developer-level mistakes. The whole team can make each other better by watching out for anti-patterns and then educating each other on better patterns to follow.

Develop With Realistically Sized Data

Your development data should be significant enough that inefficient processing or careless SQL is noticed at the time it is written. If you are planning on having 30 million products in your database, do not test daily with 400. Find a number that matters, along with appropriate cardinality between other data entities so that your set pushes the code in a way that proves it scales. Your favorite search engine will probably show many available data generators and you can always create one if you have special needs not served by that market.

Besides pumping up data on the developer’s machine, you might want to have a reasonable testing environment

Some days you need to test big. Really big. Large data. Large numbers of concurrent users. Large hit rates. Large machine counts. You name it. And you can create, grow and shrink your lab as needed.

If you don’t have the hardware, look up these items in Google:

1. Amazon EC2
2. GoGrid
3. VMWare
4. Xen
5. sorry all you other vendors, but you can add yourselves in the comments below)

And when your testing is done you can toss the hardware in the dump heap (virtually) so that it only costs when you are using it for valuable testing. Ooops, I might have introduced something with cost here in this list but I’m sure you will see that the low cost here is more than paid back by the insight the testing can provide.

Prove Your Architecture and Designs

So you have a new shiny architecture that will scale like the wind? Prove it. Most things can be modeled, rigged, contrived, and coerced into a test that shows if it will do what you expected. If you haven’t already seen it working elsewhere or as common knowledge, then test it. Prove it rather than betting your business on hope and conjecture.

Proof of concepts for scalability are your friend. Embrace them.

Design For A Single-Slice Cluster

Build your application towards a very small clustered environment. The smallest cluster that can run your application and still be a cluster. 2 application servers with a master/slave database setup really changes the picture of what your code might need to accomplish. If you can add the few days of work to develop in that model you will really open the application up to its first great step in scaling.

Why does this help? By clustering early, you learn that you need a few things to take that first step into a cluster. That you might need to share sessions. That shared cache might be useful. That having a master/slave setup means you have to consider writing to one database and reading from a list of possible contenders. That you have to worry about replication speed and patterns such as the “recent write” pattern for master/slave setups.

Yes it is more work, but it is minimal when tackled on a daily basis and fades quickly over time to only be a benefit. Most clustering can be tested on a single machine. If not, look at virtual machines (VMWare, Xen) or grid computing options.

Do Not Make Users Wait for Things They Do Not Care About

Some scaling, or really performance, comes from avoiding unnecessary work while the user is paying attention. If you were to do the following:

1. Receive a user request
2. Log it to a user access log to the database
3. Check the database for a cached search query
4. Call the slower remote search query API
5. Save the search query in the database to cache the results
6. Return the results to the user

How many of those steps does the user care about? They really care about:

1. Receive a user request
2. Check the database for a cached search query
3. Call the slower remote search query API
4. Return the results to the user

The two steps of logging the user access and caching the search results are of no interest to the user so why make them wait? Take them out of the main line of processing and get back to your user quickly without them paying a tax for system related tasks that do not impact their request.

When you multiply server requests by millions these things tend to matter quickly.

Use Experts for Short Spurts

If your team hasn’t proven their application to you in the past few months, bring in someone to audit the architecture, design, code, database, and other infrastructure and render an opinion. 8-12 hours of an expert’s time should allow them to give you a reasonable idea of how your application is constructed and areas of concern.

It just may be that your team thinks they know what they are doing, and mostly does, but that there is a gap in their thinking. Or it could be that they are great PHP programmers, but just do not know how to best utilize Oracle. It doesn’t hurt to have more opinions and more minds on solving your problem. And using experts in this burst model keeps them from breaking the bank.

Avoid Over-Using Shared Resources

Most applications see a database and think: “I’ll dump everything in that database and have a nice big heap-o-data!”

Um, that isn’t always a good idea. Some times you want 2, 3 or more heaps-o-data. It is best to divide unrelated data into separate databases (even if they are on the same physical database) so that you can easily separate them physically later. You may not always have this case, or might not think you do; but there are times when you can find splits in data and should take advantage of that when you can.

Your Transaction Database Is Not Your Reporting Database

Heavy transactions do not happily co-mingle with long running reads in a database. This causes contention, causes more database resource use, causes people to add too many indexes to their transactional database, and probably causes world hunger.

This goes hand-in-hand with “Avoid Over-Using Shared Resources” and is a great place to do a little bit of extra work and create a separate database, or separate set of tables to handle those crafty queries that everyone wants. It is important to note that tables designed for fast transactions may not look much like tables designed for fast or difficult querying. So keep that in mind as you design your schemas and keep in mind that it is sometimes better to duplicate data for better performance than to … well … follow some normalization rules like a robot and die a horrible death.

A good helpful tip when it comes to reporting on transactional tables: When someone asks you to add an index to your high volume transaction table so that they can query by some alternative field, ask them if they know the total log write increase that extra index will add to your archive logs? If you don’t know, talk to a DBA and look at the topic “Use Experts for Short Spurts” above. Hopefully they can tell you why that is important.

Conclusion

I ran out of time, but not topics. There are hundreds of daily things to keep in mind for scaling and I’ve only made but a dent with this article.

Topics like how to avoid contention on heavily contended transaction tables. Or why batch processes that commit every 50 loop iterations are doomed. Or why you should always develop with clear sizing and performance targets. Or how to use the database in ways it was meant to be used. Or in using messaging to avoid storing transient events as long-term-persistent data. Or “top 10 things to do asynchronously.” Or why you should be using the Zend Platform. Or when do master/slave database setups top out and fail. Or top mistakes junior programmers hide in your code so that you are constantly haunted by mystery bugs and ghostly performance issues.

I could also talk about how to solve some of these issues, and go into case studies and detail about anti-patterns an patterns for scalability. But my fingers are tired and I’m wishing Cal were here to write some nice quip about my article.

Maybe I’ll write about them next time. Until then, go out there and SCALE!