ben robison

First Glimpse of Site Catalyst

I got my first glimpse of site catalyst which came after an evening of playing with AWStats preceeded by an evening of playing with Google Analytics. I publicly proclaim that I am not an expert in any of these areas, I’ve simply been able to give them a test drive and here are my initial reactions to the three.

AWStats
You’d be absolutely amazed if you knew the extent to which our online activities are being tracked and recorded. The basic install of AWStats (Advanced Web Statistics) uses only information obtainable in server logs. These headers get passed around every time your web browser wants to load a page, download an image, etc. Just from that information, AWStats is able to generate an impressive array of statistics and information. The kinds of things that are available there are:

• IP Address
• UserID
• Time = the time that the request was completed
• HTTP request type (Get, Post, etc.)
• Size (in bytes) of the response
• Referer = if you follow a link from my site, I’m the referer
• User-Agent = tells the browser and platform type (Firefox 2.0.0.1, Mac OS X 10.4.8)

From this basic set of statistics, it’s able to use a reverse DNS lookup with relative success to tell what countries all the hits are coming from. Some statistical analysis are run as well, to produce reports showing statistics by month for how many hits, how many unique visitors (this calculation can be problematic), how many page views, browser stats, bandwidth usage, and any myriad of other things. The reports are done in standard HTML and gives a pretty good Dashboard view of all sorts of things going on on your site. In particular, I liked the reports showing what search terms and phrases that others were using to land on my site. Not a bad tool, but again, it’s limited to what gets passed around in HTTP headers. By extending AWStats a bit and including some javascript, you start to approach the type of functionality offered by Google Analytics, so we’ll move on to that now.

Google Analytics
Google Analytics is a straight javascript based solution to recording your web statistics. Trust me when I use the term “javascript voodoo” to describe what’s going on. When you visit my website, your browser will download a little javascript file. Then when the page loads, it calls a javascript function that will send a request to a google server somewhere and request to download a little file. But that request to download sends all sorts of information about your browser above and beyond that’s available in HTTP headers. It can tell me how what screen resolution that you’re using, for example.

Google Analytics shines however in the number and quality of reports that it generated. I can go to Google Analytics and see just about everything related to traffic on my website. It’ll plot visitors on a world map, it’ll build nice graphs and charts, etc. The types of information available in Google Analytics lets you do things like, customize your page to be usable on the platform and browser that most of your users are using. You can see if people hit one page and leave again (bounce rate) or if they stay around for while, it’ll show you the duration of the visit. It’ll tell you the most common entry point for your site, whether it’s the homepage or some other page on your site. Shows you which is the last page your visitors looked at before the left.

It seems that the major difference between Google Analytics and Omniture however is the type of statisticsthat each system is capable of gathering. After a very brief tour of Site Catalyst yesterday, I stand in awe of the types of things you can learn about your visitors.

Site Catalyst
Site Catalyst collects statistics about traffic, but what it does that no other product can do is tell you about the eCommerce implications of your site. You can tell how many people landed on your page as a result of a particular add campaign, or an add placed on another website, or an email that you sent out to your customers. You can tell how much revenue you generated from a particular product or product line, with all sorts of relevant details (profit margin, cost, etc).

In a way, Site Catalyst can make you nearly omniscient in the realm of your online eCommerce presence. Omniture is currently developing a new product called Discover that we can expect to be rolled out in the first half of this year - don’t ask, I don’t know. Discover refines the granularity of control down even further. You want to blast an email out to everybody that added a MacBook Pro to their cart and abandoned the cart before checkout? It’s done.

Now, Site Catalyst is not for the faint of heart, the level of detail available in the reporting tools is amazing, and from what I could tell, you could spend days on the backend analyzing the reports. Omniture has a team dedicated to looking for new ways to analyze data to provide valuable information to its clients.

Site Catalyst is not free, or even inexpensive. It’s the Cadillac of web analytics products, and you’ll pay top-dollar for it. Based on what I’ve seen though, it is entirely within the realm of possibility to recoup your yearly cost with a single marketing campaign (of course, this would be based on the popularity of your site). I certainly couldn’t recover the cost, but think of the possibilities for Newegg or Buy.com.

Bottom line is that the websites you visit probably know a lot more about your computer than you thought they did. I know that I was certainly surprised by the level of detail available in these reports. I don’t mean to turn anyone into a paranoid, but these are things you should be aware of when you go online. After all, anyone who uses these tools would be the first to tell you:

Knowledge is Power.

Howto: Put WSDL on Rails

Why do I love Rails so much? Because there is so much to love! Who ever dreamed that creating web services could be so simple? Where other frameworks can consume them very simply, Rails defines simplicity when it comes to creating them. Consuming them is easy as well, but it’s on the creation side that Rails really stands out from the competition. So how do you do it?

First off, let me say that my inexperience in web services means that I’m far from a subject matter expert. I can however follow simple instructions, and I found a beautiful set over at the SOA Ranch. This is a 3 part series, though more are supposedly coming. I hope these articles continue to be published, but I have my doubts as the most recent articles are nearly 6 months old now. The first one I found was part 3 part 2.5 in the series, and you can follow the links back to part 1 & part 2.

So let’s get down to business. I’ll make the assumption that you’ve got a working Rails environment. For Mac users, the latest version of Locomotive has everything you need for creating web services. For everyone else you’ll have to make your Rails environment the old fashioned way, although you can look forward to Rubuntu soon. We’re going to create a very simple web service that accepts a string, changes it a bit and returns a string.

First create yourself a new application

rails WebServiceProvider

Then you’ll want to open your application in your favorite Rails editor. Then we need to generate your web service.

./script/generate web_service Manipulation manipulate

This will create an API file for us to define our web service’s interface. It also creates a controller for us to use, and then a test file for us to use later (we won’t get to scripted testing in this article, although we will test it manually to prove that it works).

Now open up your manipulation_api.rb file in app/apis. You’ll see the following line of code that is the beginnings of our string manipulating web service.

api_method :manipulate

We’ll need to define parameters and data types for our method like so:

api_method :manipulate, :expects => [:string], :returns => [:string]

There are many data types that can be used here, but we’re keeping things easy for now. Now we’ll go look at the ManipulationController at app/controllers/manipulation_controller.rb. Because we followed the standard Rails naming conventions the mapping of the web service to the controller is already done, but if we hadn’t, this mapping can be done explicitly by adding the following to our controller.

web_service_api ManipulationApi

We can also give our web service a different name with the wsdl_service_name directive

wsdl_service_name 'Manipulation'

Notice also that in the controller is where we define all our logic for what happens with the parameters being passed in by our web service consumer. the manipulate method has already been created for us, so we simply need to tell our method to accept a string, perform our logic, and return another string.

Again we’ll keep things simple.

def manipulate(terms)
return "Yeah! " + terms + "!"
end


Now for all intents and purposes, our web service has been coded. We’ll add one more line to our controller that will generate a web front-end for the service so we can go see how it looks on the browser end. Just add the following line to your controller.

web_service_scaffold :invoke

Now let’s turn on the server and see how things look. Back at the terminal type

./script/server

Open your browser and point it to http://localhost:3000/Manipulation/invoke. This shows us the web view of our API. Follow the manipulate link to see the auto-generated web form that will show us what our service looks like. Type your message into Param0 box and click the invoke button to see all your beautiful web service XML. Another important point for those wishing to consume your web service is that Rails is automatically generating your wsdl for you. You can see how this works by checking the routes.rb file in app/config but just know that visiting http://localhost:3000/Manipulation/service.wsdl you can see the WSDL file that others will need to use your web service. You can use something besides service.wsdl by editing your routes.rb file appropriately.

Now this is obviously a simple example, but it illustrates the basic steps and important points. You probably wouldn’t have much use for a web service that adds the word “Yeah” and a few exclamation points to some user-input, but if we were doing this in the context of a real web application, you can start to imagine the possibilities available in your controller if you’ve got a dynamic database-driven application to work with. You can add other methods to your API, define the param and return types there, then define the logic in the controller in the appropriate method. Once you get the basics down, you’re on your way and the sky is the limit. Good luck developing your first piece of the Web 2.0.

Web Services [Part 2]

I wrote a few days ago about web services. Those writings were based on some preliminary research and class discussions. Since then I’ve had a little spare time to sit down and play them a bit, and even was able to whip up my own little web service in Rails. I’ll provide a HowTo as another article, but wanted to give my definition a bit of a refinement after some personal experience.

Any programmer is familiar with the concept of invoking a method on an object. For the non-programmers in the crowd, an object is simply something that us programmers use to represent real-life objects. When you register for my web site, give your email address, name, and password, I create a user object that represents you. When you come back to my website and login, I talk to your user object and ask it if the password you typed in is the right password. If it says yes, then I let you in, and if it says no, I tell you to try again.

In a traditional computer program or web application, we always have to create our own objects from our own data. Web services are changing that traditional model and changing the way we think about programming. In a larger sense this realignment of thought is changing the underlying structure of the web. It is changing the way that companies think about eCommerce. It is changing the fundamental make-up of the web itself. If you’ve heard about the Web 2.0, this is a very simplified description of that concept. For more about the Web 2.0 you should look up what O’Reilly wrote about the subject.

Web Services Redefined
So now that I’ve got the preliminaries out of the way, I’m going to explain web services much more simply than I did a few days ago. Web services simply provides a way for you to access and talk to objects that other people have written. More importantly, it doesn’t matter what programming language or platform that someone else used to create the service, you can use it with any platform you want to. That’s really all there is to it.

What About Web Services…

No doubt you’ve heard about web services, but what exactly are they? I’ve had teachers and others try to get this into my brain for at least the last three years, but it was only lately in a discussion about service-oriented architecture that it finally began to make sense. Let me lay these two topics out as simply as I can to see if I can make sense of it.

One of the problems with defining web-services is that has many different meanings to many different people. Don’t believe me? Type “define:web services” into your nearest Google search box to see what I mean.

So What Are They?
To get a better sense for what web services actually are, let’s first briefly discuss service-oriented architecture or SOA. The concept of SOA is not new, rather it is the context in which we use it that is new and improved. Very simply, the idea is that a piece of information or a particular process is exposed to consumers as a service. These consumers can be end-users or other services attempting to organize or “orchestrate” individual services into something an end-user can use.

Some people believe that web services is simply a new term for something that we’ve been doing for years. In a way this is true, and I’d bet that you interact with them more often than you realize, but it seems like it’s nothing new. Your computer runs dozens of services, each doing something important. What’s different about web services?

Let’s say that you have a web-service that looks up the weather, one that plots an address on a map, and one that finds the nearest gas station. An end-user can look up each of these individually or they can use an intermediate service that integrates each of these individual services and provides a single point of interface for all three pieces of information.

Abstracting out one more level, this intermediate service can be described as a so-called “portlet.” Think of a portlet as a widget; one piece of information that can be displayed along with many others and can be changed at will. The best example of this I can think of is your customized Google Homepage. You can choose from any number of portlets or widgets that will display many different kinds of information. When you login, requests are submitted to each individual portlet and the proper information comes back and is displayed in your browser.

How Does It Work?
I won’t get into incredible detail here, as the purpose is simply to provide an overview, but as Strong Bad says, “Technology is another word for magic.” Actually, there are some very basic technologies at work here. Each web service must simply describe itself with Web Service Description Language (WSDL) so that service consumers know what the service does and how they can interact with it.

Interaction happens through Simple Object Access Protocol (SOAP) and eXtensible Markup Language (XML). In very simplistic terms, SOAP is simply an extenstion to the HTTP protocol that enables the passing of parameters via a standard request/response interface. These parameters are passed in a standard XML format so that both sides can properly encode and decode the actual data being passed back and forth.

Why Web Services?
This now lets us as answer the question of why? Web services are language and platform agnostic, meaning that because the paths of communication are so stricly defined, that as long as a programming language or platform can speak the communicate in this common way, the two can communicate.

The other important “why” is that it cuts down costs. Before service-oriented architecture came along, if you had five proprietary systems that needed to communicate, each system had to define a custom interface to the other 4 in order to communicate. This means ten different interfaces would need to be programmed if all five want to talk, but if all speak the common language of services, then only five must be written. The business case for service-oriented architecture? Reduced cost.

Welcome to the Web 2.0.

Now Listed on Technorati

So now that I’m really starting to get serious about this whole blogging thing, I’m suddenly interested in driving traffic to my site. I think this is partially a result of some of the things I’m learning in class. In support of my efforts I’m now joining Technorati. When I asked a friend about Technorati, he described it as “Google for Blogs.” With that kind of an endorsement, how can I resist? With this in mind, you can now view my Technorati Profile.