Mirroring the Zend Server Repositories

      Comments Off on Mirroring the Zend Server Repositories

Zend Server, Zend Server CE and Zend Server Cluster Manager for Linux are
provided through DEB And RPM repositories. This has multiple benefits; one of
them is the ability to easily create an internal mirror of the Zend Server
repositories, using common tools available on virtually all Linux machines.

Why Mirror?

There are several good reasons to mirror a Linux repository, be it the
Zend repositories or any other repository. The most simple reason is sheer
performance: if you need to install Zend Server on a number of machines, you can
save a lot of valuable time by creating a local mirror of the repositories, and
using that mirror as your installation source for all servers. You will then be
able to download packages at LAN speed, which is blazing fast – even
faster than Zend’s Content Distribution Network.

Another good reason is bandwidth conservation – if you work in a
large office with limited bandwidth, or if you are charged for excess bandwidth,
downloading the repositories once and then distributing packages internally by
pointing your package manager to your internal mirror could save huge amounts of
bandwidth (for you and hey, for Zend too!).

Last but not least, in many environments it is important to control
exactly what software version is installed on which servers. Many prefer to
create a profile for their servers, test it, and ensure that additional servers
are installed from the exact same source. Even a few months after a new version
of Zend Server has been released, new production servers would still be
installed based on the same set of packages. This can be controlled by creating
an internal mirror, pointing your servers to that mirror and only installing
from it, and not from the public Zend repositories. Once you do that, you have
full control over which packages are available to your Linux installations.

So how do you achieve that? Well, mirroring the Zend repositories,
whether you use aptitude, yum, zypper or any other package manager, can
be done using very simple, standard and common tools, and doesn’t take
more time than it would take to download and install a full version of Zend
Server.

Step 1: Copy the Zend Server repository to a local machine

The first step of mirroring any kind of Linux repository is creating a
local copy of the repository contents. It is recommended to create this copy on
a local server available to all other machines in your LAN so they can later
install directly from it. This doesn’t need to be a particularly capable
machine, and will not be doing much more than serving files every now and then,
so unless you run thousands of Zend Server instances and update them very
frequently, any Linux machine would do fine.

To do the work, we will use the mighty wget
– one of the more well known Linux powertools, and will rely on its
slightly less known capabilities to yank entire sites into local storage.

The contents of all Zend Server repositories is listed in a single file,
which is updated on each release), and is available at <a
href=”http://repos.zend.com/zend-server/files.lst” target=”_blank”>http://repos.zend.com/zend-server/files.lst.
This is a plain text files with lines listing the relative path to all files in
the Zend repositories. In order to create a local mirror of one repository, we
will grab this file, filer it based on our needs, and feed it to wget as the
list of URLs to download:

These commands use code the DEB repository as an example – replace
deb‘ in the first line with
rpm‘ or ‘sles‘ in order to mirror one of the
other repositores. You may also drop the ‘export’ line and
‘grep’ line all together if you plan to mirror all Zend Server
repositories – but this is rarely required.

While wget is working (you will be downloading a few hundreds of
megabytes now, which may take a few minutes), let’s explain these few
commands:

The first command exports an environment variable containing the
repository type. This variable is later used by grep to filter the files.lst
file so that wget only fetches the files we need. Using an environment variable
is not really required – we gave done it here for the sake of readability.

The second command simply creates a directory for the repository. You
should replace /home/zend-repo-mirror with a path of your choice;
personally I recommend going for /home, /var or a dedicated location on
which typically large amounts of data are stored. Additionally, I have created a
directory for the current Zend Server version (6.1.38 at the time of writing) –
this is a recommended practice if you may need to mirror different versions of
Zend Server side by side. The 3rd line obviously changes directory into the
directory we have just created.

The last command, which is in fact a series of 3 separate commands piped
together, is the really interesting part. Let’s explain that part piece by
piece.

First, we pull the files.lst file from the Zend CDN. The
‘-nv’ flag simply means wget will not produce verbose output (less
clutter on screen unless there’s an actual error). The ‘-O
-‘ part tells wget to write the output to the shell’s Standard
Output stream instead of writing it to a file – this allows us to chain
the next command to read the data fetched by wget without using an intermediate
file.

The grep command filters out the contents of our fetched files.lst,
only allowing lines beginning with the value of $REPO_TYPE
(‘deb’ in our example) to pass through. Again, the remaining lines
are piped on to the next command.

The last wget command is what does most of the work: ‘-i
-‘ means “read the list of files to download from the shell’s
Standard Input” – this is in fact the output of the first wget
command, filtered through grep. Each relative URL passed in is preceded
by a base URL – http://repos.zend.com/zend-server/, as defined by the
‘-B’ flag.

‘-xnH’ and ‘–cut-dirs=1’ means wget will create the same
directory structure locally (starting from the repository directory) as on the
server.

Finally, the ‘-N’ flag makes sure that if you re-run the synchronization process,
existing files will only be fetched if the server has a newer version of them. This way,
you can periodically update your repository without having to download everything all
over again.

Once wget is done, you should have a new deb/ (or rpm/
or sles/) directory created under your mirror directory, with a full
copy of the Zend Server repository inside. The content of this directory is
different based on the repository type, but this is of no concern to us. For
now, all you need to remember is the full path to this local directory – in our
example’s case, /home/zend-repo-mirror/6.1.38/deb.

Step 2: Local Repository Test

Before we make our local repository available to other machines in our
LAN, let’s demonstrate how we can directly access our repository and even
install from it using local file access.

To do so, configure your package manager to access the new repository
locally (remove any other Zend Server repositories you may have set up first),
using the file:// scheme instead of http:// or ftp://
schemes. For example, for Ubuntu and Debian, add the following line to your
/etc/apt/sources.list
file:

And for yum-based distributions (RHEL, Fedora, CentOS, Oracle Linux),
create the following file in /etc/yum.repos.d/zend.repo:

Make sure to replace the paths with the path on which your local
repository resides. Take note that this is exactly identical to the setup
described in the <a
href=”http://files.zend.com/help/Zend-Server/zend-server.htm#installation_guide.htm”
target=”_blank” title=”Zend Server Installation Guide”>Zend Server
Installation Guide; the only difference is that instead of using the public Zend
repository URLs (http://repos.zend.com/...) we use ‘file:///...
with a direct reference to the full local path of the repository. This is true
for both DEB and RPM repository setup.

Now, to test if the local repository was copied correctly, update your
package manager’s cache and try to search for the Zend Server packages, to
see if they are available: For Ubuntu and Debian, you can run:

And for Redhat and friends, try:

If you get some Zend Server packages listed, and don’t have the
public Zend repositories set up, you are on the way to success.

Step 3: Enable network access to your local repository

Now that your local mirror at least somewhat works, it is time to expose
it to other local machines. Most package managers support installing from local
files as well as over HTTP and FTP, and sometimes over additional protocols.
This means you have multiple options to expose your local mirror:

  • Using a web server over HTTP
  • Using an FTP server
  • Using a network shared files system – such as NFS or Samba
    (CIFS)

You can even expose the same repository over more than one protocol, if
you wish to do so. For the sake of this tutorial, we will demonstrate how to
expose your files over HTTP using the Apache Web Server, which is common and
easy to set up (hey, if you use Zend Server you probably know it well anyway,
right?).

First, make sure Apache is installed on your repository mirror machine.
On Debian and Ubuntu, simply install the ‘apache2‘ package; On
the RedHat family install the ‘httpd‘ package.

Once Apache is installed, set it to expose your repository directory for
remote access. The easiest way to do that is to place the following code into a
file named ‘zend-repo-mirror.conf‘, and save this file to your
Apache’s conf.d directory, which is /etc/apache2/conf.d/ on Debian
and Ubuntu, and /etc/httpd/conf.d/ on other distributions:

Make sure the paths are correct, and restart your Apache.

To test that your local mirror is now properly accessible through HTTP,
direct your browser to http://your-mirror-machine.lan/zend-server/
(yes, replace the host name with the right host name or IP address) – you should
be able to see your repository files listed under the ‘deb’ (or
‘rpm’ or ‘sles’) directory.

Step 4: Configure clients to use your local repository

Once your mirror is set up, it is time to use it! To install Zend Server,
Zend Server CE or Zend Server Cluster Manager from your local mirror, simply
follow the manual installation instructions for your specific distribution (e.g.
http://files.zend.com/help/Zend-Server/deb_installation.htm
for Debian based distributions), but make sure to replace the repository URLs
set in your package manager configuration file from http://repos.zend.com/zend-server
to your local repository URL – in the example above, http://your-mirror-machine.lan/zend-server

For example, to install Zend Server with PHP 5.4 on a Debian machine,
assuming the local mirror URL is as in the example above, you can run the
following code:

And that’s all there is to it! From now on, all your internal
machines will not need to directly access the Zend repositories in order to
install Zend Server. You will save some bandwidth, make installations faster,
and be able to control exactly when a new Zend Server version is made available
to your servers.

More Mirroring Fun

Controlling the availability of a new Zend Server version

Once you have a local mirror set up, you have full control on how your
machines receive an updated version of Zend Server and when. Since all your
clients are set to look at a specific local URL (e.g. http://your-mirror-machine.lan/zend-server/deb),
you can control which version of Zend Server is available for installation
simply by modifying your repository setup.

For example, if a new version (say 6.2) of Zend Server is released, you
can do the following on your mirror server to download the new version to a
separate directory:

Does that look familiar? That’s because these are the exact same
commands from above, the only difference being that we’re now creating a
parallel directory with the 6.2 repository.

Once you’ve done this, exposing this new version to all your servers
is easy, and you are free to do so at a time you decide is right: simply edit
your Apache configuration file (the one we saved as ‘zend-repo-mirror.conf‘,
and change the Alias and Directory directives to point to the
6.2 directory instead of the 6.1.38 one. Restart your Apache and Voilà –
your servers will now have access to the new version.

Even Better: Making multiple versions available locally

As a final treat, you may want to make multiple versions (including older
ones) available locally. This is useful for example if you need to run one
version in production but do tests on a newer version. The best way to achieve
this is to slightly modify the setup above in the following manner:

  1. Decide which version of Zend Server you want to expose by default.
    For our example, let’s assume you want to expose 6.1.38 as the default
    version, but still make 6.2 available to servers who may need it.
  2. Create a symbolic link from the actual directory containing the
    repository (this is named either ‘deb‘, ‘rpm
    or ‘sles‘) under the default version you have selected, to a
    directory of the same name under your main repository directory. For example:

  1. Modify the Apache configuration file we have created above to expose
    your main repository directory (that is /home/zend-repo-mirror in our
    example) instead of a specific version:

  1. Restart Apache

Now, your default repository URL is still http://your-mirror-machine.lan/zend-server/deb
– any existing machines configured to use this URL will not require any
modification. However, specific versions can now also be accessed by machines
that need to do so – for example, the 6.2 version can be accessed by pointing
the package manager to http://your-mirror-machine.lan/zend-server/6.2/deb
instead.

Once you are ready to make the next version available by default to all
machines, simply replace the symbolic link to point to the next version, for
example:

Keep in mind however, that it is recommended to use a single default URL
for most of your machines – otherwise you will need to manually modify the URL
on all machines using the repository (e.g. all your production servers) at more
or less the same time in order to upgrade your Zend Server installation.

Final Thoughts

One of the benefits of Zend Server for Linux users, which may sound
trivial but is a big plus if you ask most savvy system administrators, is the
fact that it is installed and updated through standard Linux repositories. This
makes it very easy to perform common tasks, one of which is the ability to keep
a local mirror of the repository. Doing so will allow you to fully control what
is installed across your production, development and testing machines, as well
as make installations much faster, and save a lot perfectly good bandwidth.