Cannot Map {objType} to SOAP/OM

This was a strange exception that I struggled with for several hours this evening, but I have finally prevailed.

The Background
For my Information Architecture class, we’ve got to create a web service provider and a web service requester, this provided the initial impetus for my recent push into web services and the last few posts dedicated to the topic. Now our professor has published a web service that basically dumps his database out into an array of complex data types.

His database is a weather reporting database that has zip code, temperature, humidity, etc. The databse is small (currently 3 entries) and populated with completely fabricated data, this is a learning experience not an accurate forecaster.

Our task is to subscribe to his web service, and publish our own that will accept a string parameter as the zip code and return the single weather report for just that zip code. I decided that I’d just subscribe to his web service in my controller, find the appropriate weather report, create my own Struct for my web service, and push it out to my subscribers.

The Problem
Once I had received my professor’s SOAP object back, I stepped into it to retrieve the zip codes from each entry. When I found the matching zip code, I mapped his keys and values to my own with a line of code like this:

return => row.cityzip, :temperature => row.temperature, :etc => ad naseum)

There are ten different attributes that my object must return to my subscribers, so it was slightly tedious to put all these things together. I prevailed.

When subscribing to my own web service via WSDL, however, I received the following exception:

Exception: Cannot map WeatherReport to SOAP/OM

WeatherReport is the name of the Struct that I was using as my return type. I began meticulously checking all my code. I had already written code to subscribe directly to my professor’s web service, so I checked the SOAP object coming back from that. I used a copy of the same code to subscribe in my own web service and could not figure out what was going on (hours pass).

I finally ran across this email archive that had a fix, but not an answer.

The Fix
I had to change my code to this:

return => "#{row.cityzip}", :temperature => "#{row.temperature}", :etc => "#{ad naseum}")

The “#{row.cityzip}” means that the object inside the {} should be interpreted as a variable and it’s value should appear in the string instead of the variable name. It’s just a shorthand way of concatenating strings with the values stored in string variables (and in case you were wondering, yes, you must use double quotes or it won’t work).

Now the most interesting thing about this error is that row.cityzip == "#{row.cityzip}", meaning that the expression evalutates to true. For all intents and purposes, these two strings are exactly the same.

Peter (the guy with the fix) speculates on what might be the cause, but he was also using the a Struct (via SOAP4R), so I assume that the problem’s source may lie in that class. Given the dearth of readily available information on the topic, I hope you find this useful.

WSDL, Rails, & Complex Data Types

This could also be considered part three of my HowTo: Put WSDL on Rails series, but it’s not really a HowTo in the strictest sense of the word. Therefore, this article is more of a monologue than a set of instructions.

If you’ve stuck with me this far you’ve probably figured out that I’m discovering things on my own shortly before I spew them all out here. In Howto: Put WSDL on Rails we built a simple web service that would accept a string parameter, modify it, and send the modified version back to the user. In Part 2, we subscribed to (and used) that same web service using only the WSDL. At the time I thought I was done, but as I’ve played around a bit more, I realized that there was another major part of web services. So now we’ll cover dealing with complex data types.

I began by wondering what kinds of things besides [:string] you could pass as parameters to and from a web service. I finally happened upon the Ruby on Rails manual that somehow I had not discovered up to this point. Why don’t these pages come up higher in the Google search results? Rails uses ActionWebService to interact with web services of all kinds and you can find the beginning of the manual here. You can find the specific answers to my questions here and here.

After reading this and mulling it over a bit, everything was pretty clear except for the so-called Structured Type parameters. I set out to gain a better understanding of this, and this is what I accomplished. First off, you’ll want to read up an something called a Struct. In essence an ActionWebService::Struct is a special kind of object that can be used for web services, but doesn’t have any ties to a database like an ActiveRecord::Base object.

I first defined a new struct in app/models/weather_report.rb of my WebServiceProvider.

class WeatherReport < ActionWebService::Struct
member :zip, :int
member :temp, :int
member :wind_speed, :int
member :wind_direction, :string

I then created the proper method in my API at app/apis/weather_api.rb:

api_method :get_weather, :expects => [:string], :returns => [WeatherReport]

And since I was really just curious to see how this works, I defined a very dynamic method in app/controllers/weather_controller.rb:

def get_weather(zip_code)
return => 84097, :temp => 55, :wind_speed => 5, :wind_direction => 'SW')

And that’s really all there is to it. If you’ve got a database driven application you could simply pass in one of the models that corresponds to a table on your database. So instead of an ActionWebService::Struct you could just pass in an ActiveRecord::Base object with the exact same syntax.

Now to see what comes out on the other end. In another application, I subscribed to my web service and in my view put <%= debug @result %> with @result being the result of my GetWeather call to the web service.

The debugged results came out like this.

#<soap::mapping::object:0x14868c6 {}zip="84097" {}wind_direction="SW" {}wind_speed="5" {}temp="55">

Carrying this logical line one step further, if a web service required a complex data type as input, we could build an ActionWebService::Struct object or use an appropriate ActiveRecord::Base object to pass into a third party web service. I see the struct option being much more useful ona day to day basis, but in a B2B environment, you can start to see a place for some real hard-core service interactions passing full-blown database-drive object types around between systems.

As I began to play around with web services, I thought they were pretty neat, but I’m beginning to see the how powerful they could be given the right set of circumstances.

Howto: Put WSDL on Rails [Part 2]

In the first article about WSDL on Rails we created our own web service that accepted a string, modified it, and returned a new string. Now we’ll take a look at creating another application that can consume or subscribe to our web service. This is almost too easy. I figure the best way to demonstrate the simplicity is to keep this article as short as possible.

1. Fire up the old WebServiceProvider application that we created in Part 1

From the application root directory type ./script/server

2. Create a new WebServiceConsumer application

In a new terminal window type rails WebServiceConsumer

3. Create a controller for your service consumer

cd WebServiceConsumer
./script/generate controller consumer

4. Subscribe to your WebServiceProvider using its WSDL

Open up your consumer_controller.rb in app/controllers. It will be empty and it needs to look like this:

class ConsumerController < ApplicationController

require 'soap/wsdlDriver'

def subscribe
if params[:terms] == nil
@service_output = ""
url = "http://localhost:3000/subscription/service.wsdl"
factory =
service = factory.create_rpc_driver
@service_output = service.Subscribe(params[:terms])

This is where the magic happens. The line require 'soap/wsdlDriver' brings in the needed Ruby library. The subscribe method is simply an action like any other in your application. We’re going to create a form that accepts some user input and then we’ll pass them on to our WebServiceProvider. The url, factory, and service lines could all be done in one line, but I’ve broken them out here to demonstrate.

If the box is left blank, we don’t do anything. If the user inputs some data, then we have a little bit of work to do. We create a new service object based on the WebServiceProvider’s description of itself as defined by its WSDL. Then our local service object acts as a proxy and any method calls we make on the local object are simply passed on to it the actual WebServiceProvider.

We can then define our @service_output as a variable we can then call in our view. Now for testing.

5. Create two views for testing

In app/views/consumer create an index.rhtml file as follows:

Enter your terms below. They will be passed on to our web service and you'll be able to see the response.

<%= start_form_tag :action=> 'subscribe' %>
<%= text_field 'terms', '' %></p>
<%= submit_tag "Subscribe" %>
<%= end_form_tag %>

Also in app/views/consumer create subscribe.rhtml as follows:

The web service returned: <br /><br />
<%= @service_output %>

6. Now fire up your WebServiceConsumer app on a different port

Depending on your particular setup this may be different. I use lighttpd, so you’ll need to open app/config/lighttpd.conf and change the server.port = 3000 to say something like server.port = 3100. Then you can run ./script/server to fire up your app on port 3100.

Now point your browser to http://localhost:3100/consumer

Enter your terms into the text box, click Subscribe, and pat yourself on the back. You’ve just subscribed to a web service.

Howto: Make a Rails Environment in Ubuntu

In class today, there was a demonstration on how to get Apache/Axis/Tomcat running to demo some SOAP web services written in Java. As I plug Rails again, I was again amazed at the simplicity with which Rails handles all this. I began to think that there might be some interest in Rails web services for those other members in my class, so in an attempt to help them out, I decided to provide instructions on setting up a Ruby on Rails environment.

Now I’m running mine inside a virtual machine with VMWare Server on my Windows Server 2003 box. I won’t get into all the reasons here, but these intructions would work for any VM or for a real install as a host operating system. There are only one or two differences and I’ll point them out as we go along.

1. Start with Ubuntu Server distro. I’m using Ubuntu 6.10 Edgy Eft.
Find the mirror you want, then click the Other Installation Options and find the server. If you must have a graphical environment, you can get the desktop version, but know that youll have to install apache2 and mysql on your own. If you choose the server option, you’ll be asked what kind of install you want to do. Pick the LAMP option. Do all appropriate setup, naming, networks, etc., and I’ll meet you again at the shell after you login.

2. Edit your sources list.

sudo vi /etc/apt/sources.list

Once you get inside, you’ll want to uncomment 4 lines, which will enable the universe repository along with the security universe repository. I typically comment out the cdrom: repository as well. For those unfamiliar with vi, the letter x will delete a single character. Pushing the letter i will put you in insert mode to comment our the cdrom line. Pushing Esc will get you out of edit mode. Pushing :, then wq, and finally Enter will put you back at shell with an edited sources list.

3. Get SSH running
For me this is essential. The interface through VMWare isn’t the speediest thing, and when you’re running over a Remote Desktop connection, things start to feel like molasses. A few things will fix us right up, because through SSH everything runs beautifully.

sudo apt-get install openssh-server

This will install and start the SSH server. Now you may need to configure port-forwarding through port 22 in order to connect to your Rails/Ubuntu server. In addition if you need to forward traffic to your VM, then at your router point port 22 at the host OS and follow my other instructions on port forwarding to a virtual machine.

If you’re happy where you’re at, then you can skip step 3. If you want to do it, then SSH to your new server and continue to Step 4.

4. Update all your packages

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install saidar

Saidar is optional, it’s a neat little monitoring utility that runs at the command line, but tells you about your CPU usage, memory usage, disk usage, and everything else you’d expect from your Task Manager or Activity Monitor. Now it’s time to get down to business.

5. Install Ruby

sudo apt-get install ruby ruby1.8 ruby1.8-dev ri rdoc irb libmysql-ruby libmysqlclient15-dev

If you didn’t choose the LAMP option earlier (or picked a desktop distro) you’ll want to add Apache2 and mysql-server to this list:

sudo apt-get install ruby ruby1.8 ruby1.8-dev ri rdoc irb libmysql-ruby libmysqlclient15-dev apache2 mysql-server

6. Install RubyGems
RubyGems is like the apt-get utility for strictly Ruby packages. This series of commands at the shell will download, install, and clean up the RubyGems package.

tar -xvzf rubygems-0.9.0.tgz
cd rubygems-0.9.0
sudo ruby setup.rb
cd ~
rm -rf rubygems-0.9.0
rm rubygems-0.9.0.tgz

7. Install Rails

sudo gem install rails --include-dependencies

This will show some errors while installing documentation, but never fear. All is well. We both know that you weren’t planning on reading the documentation anyway.

8. Install Lighttpd with FastCGI

sudo apt-get install lighttpd libfcgi-dev libfcgi-ruby1.8 build-essential

This will show an error that it tried to bind to port 80 and failed. That’s ok, because we have apache running there. What we’ve done is simply install it so that our Rails apps will use Lighttpd rather than the built-in WEBRick.

If like me you really just don’t like errors you can edit the lighttpd.conf file so that it binds to a different port. This is not necessary at this point, but if you plan on running Rails apps in production you’ll have to play with the lighttpd config file at some point anyway.

vi /etc/lighttpd/lighttpd.conf

Find and uncomment the line that says

# server.port = 81

Now we’ll run Stop just to make sure and then Start it up again.

sudo /etc/init.d/lighttpd stop
sudo /etc/init.d/lighttpd start

Now install the fcgi RubyGem:

sudo gem install fcgi

9. Make sure that your FastCGI bindings and MySQL bindings are working properly.
Now to make sure that our fastcgi and mysql libraries are working properly, we’ll fire up IRB. IRB stands for Interactive Ruby. It basically gives us a functional, yet empty ruby environment to play around.

irb(main):001:0> require 'mysql'
=> true
irb(main):002:0> require 'fcgi'
=> true

Now, unless you want to do all your mysql configuration from the command line, you’ll need to be able to connect as root to your mysql database from a remote host. See my instructions on how to do that. You may need to set up port forwarding through port 3306 in order to get this to work.

First Glimpse of Site Catalyst

I got my first glimpse of site catalyst which came after an evening of playing with AWStats preceeded by an evening of playing with Google Analytics. I publicly proclaim that I am not an expert in any of these areas, I’ve simply been able to give them a test drive and here are my initial reactions to the three.

You’d be absolutely amazed if you knew the extent to which our online activities are being tracked and recorded. The basic install of AWStats (Advanced Web Statistics) uses only information obtainable in server logs. These headers get passed around every time your web browser wants to load a page, download an image, etc. Just from that information, AWStats is able to generate an impressive array of statistics and information. The kinds of things that are available there are:

  • IP Address
  • UserID
  • Time = the time that the request was completed
  • HTTP request type (Get, Post, etc.)
  • Size (in bytes) of the response
  • Referer = if you follow a link from my site, I’m the referer
  • User-Agent = tells the browser and platform type (Firefox, Mac OS X 10.4.8)

From this basic set of statistics, it’s able to use a reverse DNS lookup with relative success to tell what countries all the hits are coming from. Some statistical analysis are run as well, to produce reports showing statistics by month for how many hits, how many unique visitors (this calculation can be problematic), how many page views, browser stats, bandwidth usage, and any myriad of other things. The reports are done in standard HTML and gives a pretty good Dashboard view of all sorts of things going on on your site. In particular, I liked the reports showing what search terms and phrases that others were using to land on my site. Not a bad tool, but again, it’s limited to what gets passed around in HTTP headers. By extending AWStats a bit and including some javascript, you start to approach the type of functionality offered by Google Analytics, so we’ll move on to that now.

Google Analytics
Google Analytics is a straight javascript based solution to recording your web statistics. Trust me when I use the term “javascript voodoo” to describe what’s going on. When you visit my website, your browser will download a little javascript file. Then when the page loads, it calls a javascript function that will send a request to a google server somewhere and request to download a little file. But that request to download sends all sorts of information about your browser above and beyond that’s available in HTTP headers. It can tell me how what screen resolution that you’re using, for example.

Google Analytics shines however in the number and quality of reports that it generated. I can go to Google Analytics and see just about everything related to traffic on my website. It’ll plot visitors on a world map, it’ll build nice graphs and charts, etc. The types of information available in Google Analytics lets you do things like, customize your page to be usable on the platform and browser that most of your users are using. You can see if people hit one page and leave again (bounce rate) or if they stay around for while, it’ll show you the duration of the visit. It’ll tell you the most common entry point for your site, whether it’s the homepage or some other page on your site. Shows you which is the last page your visitors looked at before the left.

It seems that the major difference between Google Analytics and Omniture however is the type of statisticsthat each system is capable of gathering. After a very brief tour of Site Catalyst yesterday, I stand in awe of the types of things you can learn about your visitors.

Site Catalyst
Site Catalyst collects statistics about traffic, but what it does that no other product can do is tell you about the eCommerce implications of your site. You can tell how many people landed on your page as a result of a particular add campaign, or an add placed on another website, or an email that you sent out to your customers. You can tell how much revenue you generated from a particular product or product line, with all sorts of relevant details (profit margin, cost, etc).

In a way, Site Catalyst can make you nearly omniscient in the realm of your online eCommerce presence. Omniture is currently developing a new product called Discover that we can expect to be rolled out in the first half of this year – don’t ask, I don’t know. Discover refines the granularity of control down even further. You want to blast an email out to everybody that added a MacBook Pro to their cart and abandoned the cart before checkout? It’s done.

Now, Site Catalyst is not for the faint of heart, the level of detail available in the reporting tools is amazing, and from what I could tell, you could spend days on the backend analyzing the reports. Omniture has a team dedicated to looking for new ways to analyze data to provide valuable information to its clients.

Site Catalyst is not free, or even inexpensive. It’s the Cadillac of web analytics products, and you’ll pay top-dollar for it. Based on what I’ve seen though, it is entirely within the realm of possibility to recoup your yearly cost with a single marketing campaign (of course, this would be based on the popularity of your site). I certainly couldn’t recover the cost, but think of the possibilities for Newegg or

Bottom line is that the websites you visit probably know a lot more about your computer than you thought they did. I know that I was certainly surprised by the level of detail available in these reports. I don’t mean to turn anyone into a paranoid, but these are things you should be aware of when you go online. After all, anyone who uses these tools would be the first to tell you:

Knowledge is Power.