 |
|
|
Urchin Rocks!
or why we use Urchin to analyze our PSW web logs
By Ernie Black, PSW Webmaster
Do you maintain a busy web site and wonder which web analysis software is the best tool? Our webmaster presents the case for Urchin 4, the web log analysis program which best fits the needs of the pro audio industry's leading web sites.
Those of you who are long acquainted with ProSoundWeb.com and its subsidiary sites Live-Audio.com, recpit.prosoundweb.com, and links.prosoundweb.com may know that we average over 180,000 unique visitors turning over 3.5 million pages each month. This traffic generates multiple web logs on multiple servers which quickly reach 2 gigabytes in size. For over two years, we used WebTrends Reporting Server to crunch our logs to generate detailed reports of site activity, but our logs were getting so large that the otherwise excellent WebTrends software sometimes required several hours simply for processing. So we decided to look for an alternative log analysis program which would churn our logs much more quickly, and, with luck, offer other improvements in the bargain. That program turned out to be Urchin 4.
Over two years ago I wrote "Know Your Visitors", a comparative look at three leading web traffic analysis programs (Web Trends Enterprise, Funnel Web Analyzer, and Urchin Pro 3) for what was then Web Techniques and is now New Architect magazine. For that article, I distinguished between Enterprise (single corporation) and Service Provider (data center) solutions. Due to both the volume of ProSoundWeb traffic and the multiplicity of sites we manage (including a few outside the Pro Audio realm and not related to ProSoundWeb), I reasoned that Urchin merited a good look. I knew from my previous testing that it was fast—it can crunch a 2 gigabyte web log in under 20 minutes—and also much less expensive than WebTrends (see their sites for current pricing; as of this writing, WebTrends' website currently lists $3,500 for the eBusiness Edition and $10,500 for the Enterprise Edition, whereas Urchin 4 is currently $695 per 100-site license). What surprised and delighted me was the innovation of the UTM (Urchin Tracking Module) in the new Urchin 4 release, which is hands-down the most accurate visitor tracking system I have encountered.
I e-mailed my contact at Urchin, who promptly set me up with the newest version to test. For testing, I selected our then-current PSW logfile, 1948 MB in size, which would have taken our WebTrends software several hours to process. Urchin processed the log in exactly 18 minutes (elapsed time: 21:20:09 - 21:38:09), including reverse DNS lookups—probably the slowest stage of any log processing, and the most vulnerable to outside Internet latency factors. (For reference, our site is running Apache web server on Linux, with Urchin installed on its own Linux box using NFS mounting to access logs on other servers.) This demonstrated a substantial increase in speed over the 3.3 version I had tested two years ago, which took 50 minutes, 14 seconds to process a 1.1 GB log with 80% DNS resolution: in fact, almost 5 times faster. I hasten to add that this is scarcely a controlled test, as the machines used for the processing were totally different, but it gives an indication of improved performance; Urchin's own testing shows Urchin 4 to match Urchin 3's speed while delivering more (most notably the UTM detailed below). Urchin also claims the faster reverse-DNS in the industry, and everything I've seen over the past several years, on PSW and elsewhere, supports this claim.
The reports were what I expected from Urchin: fast, flexible, thorough, accurate, but (compared with WebTrends' colorful graphics) rather flat from a presentation perspective [fig. 1]. While the WebTrends graphics might be more impressive at a corporate board meeting with their bar and pie charts and all, this is of little or no importance to us. What is significant (and is one reason for WebTrends' slower performance) is that, whereas WebTrends generates static reports with hundreds of charts and pages of data, all stored in a separate directory, Urchin instead creates three lean data files which are queried on the fly (with amazing speed) by JavaScript based on the user's input [fig. 2]. So the resulting reports are much smaller, faster, more flexible, and far less resource-intensive. You can query the data by any data range you wish, whereas WebTrends Reporting Server limited us to daily, weekly, monthly, quarterly and annual reports.
Earlier I mentioned Urchin's UTM (Urchin Tracking Module). This is a little wonder: a bit of JavaScript and a tiny 1x1 pixel gif which are stored at your web root, a minor modification to your web server config file (a breeze with Apache), and some fine-tuning in Urchin itself and you have the most accurate unique visitor tracking mechanism available [fig 3]. One way in which this excels is that it forces a query to the web server, by-passing proxy servers, to ascertain that a request has actually been made to your server (many ISPs, including big boys like AOL and EarthLink but also smaller or corporate providers cache frequently-requested pages for up to a day to speed delivery; the problem is that the page may be outdated when you receive it, and your web server will never know that a request was made). Moreover, many ISPs (again, like AOL) dynamically assign IP addresses to their users, so that one user may come to your site in the same session and be seen as several IP addresses (thus, unique visitors). Instead, UTM sets a first-party cookie (between your site and your visitor) which is recognized by the system and is tracked in lieu of IP address for visitor tracking.
Set up, profile and report configuration, and filtering are all easy; Urchin's admin is entirely web-based, so you can access and administer Urchin from any browser. And if you're running multiple sites (as we are), you can monitor bandwidth consumption by all the sites to see how they are performing and which are responsible for bringing the most traffic to your servers. In fact, we run a whole range of reports beyond our actual sites, because we need to see the traffic activity for several clearly defined subsections of PSW: Live, Install, Recording, WebExpo, Study Hall, etc. With Urchin's speed, it's a breeze to generate as many tailored reports as we need. When all is said and done, I am extremely happy with Urchin's performance, and the fact that its price point is far lower than WebTrends is a big plus. I highly recommend Urchin to anyone looking for not only the best value but the best performance in web traffic analysis.
To learn more about Urchin, visit the Urchin website.
|
|
|
