I’m hoping that this being the 4th of July holiday that neither Mike, Scott nor Jon will be paying enough attention to the blog to be alarmed by the appearance of a random bit of Postgres hackery.  This came about as the result of my wanting to get a feel for some of the initial rental data that we’ve begun collecting. And before you ask: No, sorry, I don’t have a date when we’ll begin incorporating this new set of data into our product lineup.

As I was browsing the data we collected last week, I began thinking about what the general distribution of rental prices looked like. Of course, a histogram is the obvious visualization for answering questions like that. But I dreaded having to jump through all of the hoops necessary to get the data into a charting tool like Excel – export, transfer, import, format, fiddle, tweak, fiddle, tweak, rinse repeat. I just wanted a quick snapshot, not a publication quality rendering.

I began by looking through the Postgres documentation on string manipulation functions, and came across the SQL ‘lpad’ function. With a bit of clever grouping, counting and scaling, I was able to quickly generate the following histogram of rental prices, grouped into $100 buckets:

 rent |                         frequency
------+------------------------------------------------------------
    0 |
    1 |
    2 |
    3 |
    4 | ###
    5 | ##################
    6 | ######################################
    7 | ######################################################
    8 | ##########################################################
    9 | ####################################################
   10 | ###########################################
   11 | ################################
   12 | ###########################
   13 | ########################
   14 | ###################
   15 | ################
   16 | ############
   17 | ##########
   18 | ########
   19 | #######
   20 | #####
   21 | ####
   22 | ###
   23 | ###
   24 | ##
   25 | ##
   26 | #
   27 | #
   28 | #
   29 |
   30 | 

This particular histogram uses a bucket size of 100 and a unit size of 500 (you’ll see those numbers appear a few times in the query). Here is the exact SQL used:

select (re.highprice/100)::int as "rent",
    lpad('', (count(re.*)/500)::int, '#') as "frequency"
    from rental_entry re
    group by ( re.highprice/100)::int
    order by ( re.highprice/100)::int ;

The unit size of 500 was an arbitrary choice, based on the total number of entries in our sample data set and my desire to fit the output in my terminal screen. The key is to adjust the bucket and unit size constants based on the data to fit the output to a reasonably sized monitor.

Happy 4th of July!

Ever wondered why your flat’s Zestimate bounces around so much?

In high school economics class you might have learned about fungible goods. This strange word refers to things that could be swapped without the owners especially caring. A dollar is almost perfectly fungible, and so is an ounce of pure silver. Paintings and emotional knick knacks are not at all fungible. Fungible stuff is easy to trade on a centralized market, since a buyer should be happy to deal with any seller. This network effect is so important that markets “push back,” and invent protocols to force fungibility. Two arbitrary flatbeds of lumber at Home Depot are probably not worth the same amount of cash. However the CME’s random length lumber contract puts strict guidelines on how that lumber could be delivered to satisfy the obligation of the future contract’s short trader.

Real estate is seriously non-fungible. Even a sterile McMansion in the suburbs can have a leaky roof, quirky kitchen improvements, or emotional value for the house-hunting recent college grads. If we consider many similar homes as a basket, or a portfolio of the loans secured by the homes, then the idiosyncrasies of each home should net out to zero overall. Across those ten thousand McMansions, there should be a few people willing to pay extra for a man cave, but also a few people who would dock the price. This is the foundation of real estate “structured products,” such as the residential mortgage backed securities (RMBS) of recent infamy. Like flatbed trucks delivering a certain sort of wood for a lumber futures contract, a RMBS makes a non-fungible good more fungible.

The Usual Place
The combined idiosyncrasies of non-fungible things rarely net out to exactly zero, especially during a financial crisis. Nonetheless traders and real estate professionals want to think about a hypothetical, “typical” property. We define a local real estate market by city, neighborhood or even zipcode. How do we decide the value of a typical property? There is an entire industry built around answering this question. One simple, clean approach is to sample a bunch of real estate prices in a local market at a certain point in time, and then average the prices. Or maybe use a more robust descriptive statistic like the median price.

The most readily available residential home prices in the U.S. market are “closed” transactions, the price a home buyer actually paid for their new place. Using a closed transaction price is tricky, because it is published several months after a property is sold. Can a typical home price really be representative if it is so stale?

Sampling
Even if we ignore the time lag problem, there is another serious challenge in using transactions to calculate a typical home price. Within any local real estate market worth thinking about, there are very few actual transactions compared with overall listing activity and buzz. Your town may have a hundred single-family-homes listed for sale last week, but only four or five closed purchases. A surprise during the buyer’s final walkthrough could wildly swing the average, “typical” home price. For the statistically inclined, this is a classic sample size problem.

There are plenty of ways to address the sample size problem, such as rolling averages and dropping outliers. Or you could just include transactions from a wider area like the county or state. However the wider the net you cast, the less “typical” the price!

Another approach is to sample from the active real estate market, those properties currently listed for sale. You get an order of magnitude more data and the sample size problem goes away. However everyone knows that listing prices do not have a clear cut relationship with closing price. Some sellers are unrealistic and ask too much, and some ask for too little to start a bidding war. What is the premium or discount between listing price and actual value? We spend a lot of time thinking about this question. Even closed transaction prices are not necessarily the perfect measure of typical “value” since taxes and mortgage specifics can distort the final price. Our solution is to assume that proportional changes in listing prices over time will roughly match proportional changes in the value of a typical house, especially given a larger sample from the active market.

A Picture
Below is a chart of Altos Research‘s real estate prices back through 2009, across about 730 zipcodes. For each week on the horizontal axis, and for each zipcode, I calculate the proportional change in listing price (blue) and in sold price (red) since the previous week. Then I average the absolute value of these proportional changes, for a rough estimate of volatility. The volatility of sold prices is extreme.

I don’t practice Santeria / I ain’t got no crystal ball / Well I had a million dollars but I / I’d spend it all…

- Santeria, Sublime

If you’ve followed Altos Research for any length of time, you’ll have noticed that we’re a bit pedantic about the orientation of housing market data. For years we’ve been talking about the power of the insights in the active market. Understanding real estate is not just about which transactions closed, it’s about how much the sellers are asking, changes in those  ask prices; it’s about inventory and demand metrics. The active market of homes for sale is rich with information and insight about future transaction prices, home values, mortgage defaults.

Today’s Real Estate Market Yields Tomorrow’s Home Prices

The at-a-glance view of a mortgage portfolio or RMBS security in AltosEvaluate Forward Valuation Model.

We’ve noticed more and more lately that our institutional clients understand this active market informational wealth, and they keep asking us the same questions, “How much should we adjust our bids? How much are home values going to change in [random zip code]?” You only have to smack me in the face so many times before I wake up and realize the world has a need. That brings us to today.

Announcing AltosEvaluate: Forward Valuation Model

Today, Altos Research launches a technology we call Forward Valuation Modeling (FVM). Part of our AltosEvaluate Mortgage Analytics Suite, the FVM looks at everything thing we know about the current, local housing market–all the pricing and price change trends, all the supply and demand metrics, plus everything we know from years of tracking this data– and investigates how those characteristics have impacted local transaction prices previously. The FVM uses that insight to forecast price changes in each local market over the next 3, 6, 12 months.

In the capital markets, investors buy residential mortgage bonds or whole loan pools to trade or keep as investments. When you make these investments, you have to investigate every asset in the deal, maybe 1000 or 5000 loans in random neighborhoods around the country. You look at the loan, LTV, the borrower, FICO score, payment history, everything you can find to apply a value to that loan. AltosEvaluate FVM brings the ability to look at the local housing markets underlying every mortgage in the deal and make decisions based on the trend in housing values.

For example: You might realize that the 85% average LTV in the pool today is looking more like 95% LTV as the FVM illustrates how property prices are likely to fall in most of the markets over the next twelve months. These loans are MUCH more likely to default and you’re much less likely to get a return on your investment. Bid accordingly.

There’s so much to tell about this new technology, I’ve struggled to keep this post at a manageable length. If you’re interested in more, just contact us. Here’s a quick FAQ to wrap up. AltosEvaluate Mortgage Analytics Suite brochure is embedded below.

AltosEvaluate FVM FAQ:

What does the FVM do? The FVM forecasts the home price changes coming in local housing markets in the next 3, 6, and 12 months.

How does it work? The FVM platform is what’s known as a machine learning technology, a system where we have a vast amount of data, some of which is useful for forecasting the future, the system “trains” itself on the data to build models on how to process the current market. In the case of real estate, this is critical because you have to have local models (what’s interesting in the San Jose housing market is not useful in the Miami Beach condo market, for example). Altos Research tracks 20,000 zip codes, so the AltosEvaluate FVM literally trains tens of thousands of local forecasting models.

Who uses it? Mortgage and RMBS bond investors, companies with exposure to real estate and who need to understand where the local housing values are heading so they can adjust their bids, or make buying and selling decisions in their portfolio.

Is it an AVM for the future? No. AVMs (automated valuation models) take the recent closed transactions and estimate a price for a given property.  The FVM says, based on everything known about this local market, real estate prices are headed down (or up).

How do I get more info? Email our housing market analytics team for more info, white papers, etc.

Forward Valuation Model (FVM) Brochure

We have been giving our tech monkeys extra bananas lately to upgrade the Altos tools and create some exciting new applications – but we’ve been rather quiet publicizing them.  So I thought I’d take a moment to share some of the latest and bestest.

New WordPress plugin – Sidebar Slideshow Charts Widget

We love our blinky charts here at Altos, so we built an easy plugin for you to quickly add an eye-catching slideshow of a bunch of charts on the sidebar of your blog. Watch it in action in our side bar.  —————————————>

Here’s Carl with a demo video to see how easy it is to install:

Full screen version for people with bad eyes

Tip: Don’t forget to set the click-through link so that readers can request more detail from you!

Email Contact Database Importers

Our AltosConnect product has for a long time been a great way to set up email campaigns with your Altos market reports.  We’ve now made it waaay easier to import your contact list.  AltosConnect now has a no-brainer import feature for your contact data from Outlook, LinkedIn, Gmail, or Top Producer. Import and start your email campaign in just a few minutes!

Carl again:

Larger Screen Version Here

For in-depth individual screen casts explaining how to download your database from your Outlook, Linked In, Gmail and/or Top Producer accounts, log into your Altos account and go to the My Contacts page. Click on the orange links to watch “how to” screen casts for the database you would like to connect with and upload your contacts.

These are just a couple of new features we’ve rolled out lately for our Realtor clients. Stay tuned for many more!

I love love love this brand new presentation of housing trends by MacroMarkets. For those of you unaware, MacroMarkets is the firm founded by housing economist Robert Shiller, he of the Case-Shiller Index. They’ve included Altos Research data to round out the application so you can see market trends beyond just prices.

They call it the Gap Gauge web application. You ever wonder how far housing will have to fall to get back on it’s natural trendline? Are we there yet? The Gap Gauge illustrates the “gap” between where we are and where we might expect to be sans bubble. Is it a good time to buy? Mind the Gap.

MacroMarkets Gap Gauge shows national and regional housing trends using the Case Shiller Index and real estate market data supplied by Altos Research

Home Prices Compared to their Pre-Bubble Baseline

Specifically, the Gap Gauge is a free tool that quantifies the difference between the current level of a home price index and its corresponding baseline level.  The baseline level is a simple extrapolation of the average rate of home price growth as measured by S&P/Case-Shiller Home Price Indices before the year 2000, after which prices in many residential real estate markets began appreciating at unprecedented and unsustainable rates.

Says Terry Loebs, managing director at MacroMarkets, “We hope that this simple tool provides additional context for constructive debate regarding the future outlook for U.S. housing markets. And we turned to Altos Research to provide our Gap Gauge users access to housing supply and demand data that are important, interesting, and timely.”

Music to my ears.

It’s time for a nerd break here on the Altos Research blog to help balance Scott’s bout of analytic analysis – so those of you who don’t write code for a living, it is probably best to avert your eyes.

While working on a new feature to make it easier for our users to send email copies of reports to their contacts, we ran into a few difficulties trying to debug errors in our email address search function. Our plan was to use the jQuery autocomplete plugin along with the standard jQuery JSON support to build this new feature.

But in our initial implementation of the code, we were seeing some rather cryptic error messages in both the Google Chrome JavaScript debugger (“Unexpected Token : “) as well as in the Firebug JavaScript console (“invalid label”). Read on for a detailed description of our solution, along with some sample code for others to use and improve.

The jQuery code snippet we’d borrowed from the example code section on the jQuery.getJSON() documentation page is fairly straightforward, and uses the ‘callback’ approach in order to allow the request originating from the browser to be sent directly to the server. In this example, we’re just logging the output to the Firebug Console:

$.getJSON('http://server.hostname.com/dostuff?format=json&jsoncallback=?',
function(data){
console.log(data);
});

On the server side, we are using the wonderful Gson library for converting Java objects into JSON-formatted output. A typical example, using a List of Strings, looks like this:

Gson g = new Gson();
List results = new ArrayList();
results.add("test@domain.com");
results.add("test2@domain.com");
results.add("test3@domain.com");
String jsonOutput = g.toJson(results);

Using this simple list of strings, the JSON output will look like this:

["test@domain.com","test2@domain.com","test3@domain.com"]

Unfortunately, if you feed that JSON output to the callback form of the jQuery.getJSON() function, it will cause the JavaScript interpreters on both Google Chrome and Firefox to throw an exception. In Google Chrome, the error message will be “Unexpected token : ” and in Firefox “invalid label”.

The cause of the problem is that the callback form of the jQuery.getJSON() function expects the return value from the server to be a JavaScript function, not simply a JSON-formatted string. That is why the last parameter in the URL used to invoke the jQuery.getJSON() method is that mysterious looking ‘jsoncallback=?’:

http://server.hostname.com/dostuff?format=json&jsoncallback=?

This causes jQuery to generate a random JavaScript function name, and pass it along to the server using the ‘jsoncallback’ request parameter. Armed with this information, the server-side solution is fairly simple. You need to check the incoming HTTP request object for a ‘jsoncallback’ URL parameter:

String jsonCallbackParam = httpRequest.getParameter("jsoncallback");

NOTE: In the interest of good secure coding practices, friend and crazy-smart engineer Nathan Schrenk has kindly pointed out to me that the blind acceptance and use of this parameter represents a very exploitable security hole. So instead of just getting/using the request parameter, you need to make absolutely sure that the value is not being used to inject evil JavaScript. Along with the Apache Commons Lang library for Java, I recommend something like the following for sanitizing the value. This seems to well for the random method names that are generated by the jQuery JSON library:

public static String sanitizeJsonpParam(String s) {
if ( StringUtils.isEmpty(s)) return null;
if ( !StringUtils.startsWithIgnoreCase(s,"jsonp")) return null;
if ( StringUtils.length(s) > 128 ) return null;
if ( !s.matches("^jsonp\\d+$")) return null;
return s;
}

If that parameter is not null or empty, you know that your service has been invoked by a client expecting JavaScript-callback-formatted output, and not a simple JSON-formatted reply. Assuming you’re JSON string has been generated (see above), this simple conditional statement will take care of properly formatting the output:

if ( jsonCallbackParam != null ) response = jsonCallbackParam + "(" + jsonString + ");";
else response = jsonString;
outputStream.write(response.getBytes());

With this in place, you now have a server-side Java web service that will automatically support both jsonp callback requests as well as plain JSON formatted output.

If I’ve made any syntax errors in the examples above, please comment below. Keep in mind that this is not meant to demonstrate a fully production-ready implementation, but a solution to a rather cryptic error message.

In a software update to the AltosResearch website that went live last night, we’ve added in a whole bunch of new sharing options for AltosCharts users. We have included support for sharing by Email, Facebook, Twitter, Posterous, Tumblr, WordPress Blogs and TypePad Blogs. The buttons for each sharing destination are displayed beneath each chart (screen captures shown below the fold).

For the impatient, here is a short video showing you how to use the new feature:

If you’re in need of other sharing destinations, just let us know and we’ll see if we can get it added.

City/ZIP Charts:

Sharing City/ZIP Charts

Regional Charts:

Sharing Regional Charts

Sharing from AltosXplorer:

Sharing Charts from AltosXplorer

We like to think we have better and more insightful real estate data than you can get anywhere else. For our real estate agent and broker clients, we also work hard to make it super easy for you to use that data.  AltosCharts are the leading real estate market data widgets and have been since we launched them in 2006. AltosCharts are installed on thousands of websites now.

Recently though, more and more of our subscribers are blogging (and almost all of them on WordPress). With the flexibility, search-engine-friendliness, and communication powers of WordPress, many have abandoned their “traditional” website altogether. Rock on.

Four WordPress Plugins for Real Estate Data

In an effort support this trend, today we’re launching four (count ‘em, four!) WordPress plugins for our subscribers. Here they are in order of coolness:

1. Market Data Post – With just a few clicks and create an entire Real Estate Market Data blog post. Sometimes, when you have to get a post done and just don’t have the time, that’s where this plugin comes in.*
   

   Add a complete Real Estate Market Data Post for your local market

2. WYSIWYG Toolbar AltosCharts – Often though, you’re writing your regular blog post, you want pictures or other media, different formatting, and you want to add an AltosChart to comment on the market conditions. This plugin adds four buttons to your toolbar so you can instantly add a local market data AltosChart, regional/county-wide AltosChart, AltosStats table, or cool new AltosCharts Flash into any blog post you’re writing. No cutting and pasting! Works just like adding an image to your blog post. I used this plugin to add the charts to this National Report post last week.
   

   Plugin adds four easy buttons right to your blog post toolbar.

3. AltosCharts and AltosStats Sidebar – so even when you’re blogging about everything else, your visitors still see that you have market data. Set your click through links to your market data detail pages if you like.
4. AltosConnect Sidebar – Easy to include the AltosConnect lead conversion and auto-drip forms. No more cutting and pasting.

All the plugins have a few features in common:

• They use your Altos Research account and are automatically set up for your local markets.
• With just a few clicks you can easily tap all the features of AltosCharts. Show single family homes vs. condos, weekly vs. smoothed data, price range quartiles (only Altos lets you show how the market moves in price ranges), stats tables and or charts.
• Set charts to static date – when you’re blogging you want to create a post where you talk about how the market is right now. Simply set the date in the AltosCharts and they’ll always accompany your post with the right data.
• Or set the charts to always display the latest data – if you want to create a page for, for example, your town so that visitors always see the latest data then you set the charts to be dynamic.
• Set your own links. This is an important distinction between AltosCharts and other “free” market data widgets you might see around. When someone gets to your website, you want to inform them with the data, but you also want to keep them on your site. Remember that freebie widgets are designed to lure your visitors and search-engines to someone else’s site, away from you. With AltosCharts, you direct visitors to your landing page, market data detail page, IDX site.  You get the content, the leads, the SEO juice – not us!

How to get these awesomely cool plugins for your blog

If you’re an Altos subscriber and at all familiar with using AltosCharts then these four plugins will be intuitive. Log in to your Altos account and select My WebTools | WordPress Widgets to download.

Get the Widgets from your Altos Research Account

If you’re new or thinking about subscribing, give us a buzz and ask for a demo. Or check out our complete AltosIntelligence service for real estate professionals.

*A quick shout-out to Eric Stegemann, who helped us spec out the Market Data Post plugin. Eric’s firm Tribus has a technology platform for brokers that includes, among other things, a really cool implementation of AltosCharts on a WordPress site.

“Information wants to be free.”

In 1984 Stewart Brand proclaimed these words to a roomful of hackers. In the 25 years since, the observation has become the justification for an entire internet economy. Everything’s Free! w00t!

What’s less known is that Brand was not referring to price nor to the value of information, but merely to mobility. That because digital data can move and replicate instantly, whole new worlds of understanding open up. This is critical and in the same sentence, he added “the right information at the right time can just change your life.” In other words, “Information wants to be free, but analysis wants money. Maybe lots of money.”

I bring this up because this week real estate information took a step towards Brand’s vision of free. The National Association of Realtors announced the Realtors Property Resource (in a dramatic leap of branding they named it “RPR”) where they’ll offer Fidelity’s LPS property data for basically all real estate in the US.  For Realtors, this is data which previously they’d have had to purchase from LPS, now for “free” (if $12 million of realtor dues is considered free.)  RPR will also consolidate data from MLSes into one place. NAR/RPR also acquired some assets of Fidelity’s Cyberhomes.com, the stealthily successful competitor to Zillow, and are launching Houselogic.com, a consumer website.

For good details about the whole announcement see this post by Brian Boero. Let’s talk about implications.

Run Altos Run!

The project is a TON of real estate data. Not surprisingly I suppose, since the announcement, literally dozens of people have asked me what this means for Altos Research. The loquacious Rob Hahn predicts doom. Others are more sanguine.

While I admit to a pang of fear when NAR CEO Dale Swinton first pre-announced RPR at the Real Estate Barcamp this summer in Chicago, I’ll now argue that relatively little changes for our humble firm. Nor does it change for any of our well-funded co-opetitors.

Surely some of the data applications unleashed by RPR will overlap with what Altos Research provides. My guess is that RPR won’t be concerned with personalizing local market reports for your brand, or building custom data to analyze your unique niche, or providing you with market data chart widgets, and lead conversion, and email marketing systems, or blogging tools and facebook applications, and on and on. The fact is we already worry every day that other applications, sites, alliances, products will attack our business. So we’re going to keep running as fast as we can to get you to the analysis and build the applications that let people actually use the information. (We have our latest product announcement coming this Thursday! Stay Tuned!)

RPR is a big player, going after big applications. Real Estate Data is a massive market with tons of niche applications inside.

Let’s look at some of those applications.

Bring on the NARstimate

In a shot across the bow of Zillow, NAR wants to add home valuation models that it wants to be the “gold standard”. But will the NARstimate be measurably better than say a Zestimate? The answer is no, and here’s why.

Bell Curve

The math behind all the automated valuation models is called a “multi-variable regression” where you take oune property’s characteristics, and compare them to the characteristics and prices of all the other properties, mush them together and find out the probably value of your place.  It’s pretty straightforward, you can do it in Excel. The output of that math is essentially a bell-curve of possible values for the house with the peak of the curve being the most likely value. (If it smells funny it’ll move left in the bell curve, a good paint job will move it to the right, etc.)  Zillow’s marketing innovation was to publish the peak value, and call it the Zestimate – previous AVM companies were truer to the math and published the range of the bell curve.

To improve the output, you need more variables or better data on the homes. RPR sounds like it’ll work diligently to get both. Fabulous. But the improved results merely shift the bell curve a tiny bit or eliminate the occasional egregious error. It’s a lot of work for not a lot of gain. Your existing AVM, Zillow, or whatever, will have for practical purposes the same range as the NARstimate.

Here’s the other thing to note about why Zillow isn’t materially impacted here. These days, Zillow isn’t about the Zestimate, Zillow is about mortgages.

National MLS Data in a Local World

In some senses having one place for understanding all the active listings is very powerful. For Realtors, this is neither new, they already have access to their MLS data  nor is it useful – a realtor in Miami doesn’t need Dallas MLS data.  Maybe RPR will have better user interface than the local MLS can muster. Maybe not. (Consider local differences that MLSes need to handle: For example, Seattle “water-front” is an important variable about a property listing. In Tucson, having “water-front” as an option is laughable at best and credibility destroying at worst.)

Again, the prolix Mr. Hahn predicts doom. Civil War. If Rob is right, then it all falls apart and our teapot can go back to being tempestless. Frankly, I foresee lackadaisical compliance. “Sure, we’ll send data… Oh, yeah, it got buggered this week. We’ll update it as soon as we can…”  But let’s assume successful integration with 900 local partners. With good execution on the part of RPR, there will be lots of useful applications for realtors. And lots more that RPR never gets around to working on.

The Wall Street crowd definitely wants to know about the current market conditions. It sounds like LPS will have the exclusive to resell all the MLS information to Wall Street, etc. Exclusive = expensive. Nuff said. LPS sells a ton of real estate data to Wall Street already and a nice complete data set will be good supplement to existing products.

Houselogic, Cyberhomes, websites et al.

From the consumer-web site standpoint, Houselogic adds a new player that basically supplants Cyberhomes, which already published all the property data. In essence then, HouseLogic is some cool new features on Cyberhomes which already quietly does well in the face of the Zillow juggernaut.

Potential upside: Marty Frame, who ran Cyberhomes, will be president of the new business unit. Marty understands the applications of the data arguably better than anyone from the NAR group did previously.  That augers well for the new business unit.

Big Promises, Big Opportunity.

RPR’s promise is for the first time to bring it all together. He’s the rub: I’ve been selling real estate data to all levels of players in this industry and never has anyone said to me, “what I need is to have all the property data, with all the listings, CMAs, on a website with slider bars and maps and blah blah blah.”

No. What they ask is, “Mike, how can I get more listings?” or “Mike, are the home builders going to make their quarter?”

That, my friends, is the difference between information and value. The beauty is that there are infinite value questions that need to be answered. RPR, when well executed, will answers some.  We’ll mix our unique brew of expertise and technology to answer others. Plenty others.

Onwards!

We know that there is never a good time for maintenance, but according to our traffic logs, late night on Saturday is the least worst time. So we’re going to be doing some hardware maintenance on our primary web cluster (we’re installing more memory in a half dozen of our servers) this Saturday, October 10th starting at 8pm PST and lasting for about two hours.

During this time, www.altosresearch.com and charts.altosresearch.com will be unavailable. Apologies in advance for the inconvenience this will cause, and we appreciate your understanding. We try not to do this more than a few times per year, and we’ll try to stick to that rule as best we can.

Real estate market charts – a.k.a. AltosCharts – is one of the very first features we launched, nearly 4 years ago.  They are among our most widely used and popular features.  However, we figured it was time to cook up the next generation of data visualization tools – resulting in today’s release of Altos FlashCharts.

Chicago, IL – Single Family Properties – DOM vs. Price:

Altos subscribers will find their new Flash Charts by logging in, and selecting ‘My WebTools -> My Widets’. If you’re going to try and use our Flash Charts on a WordPress blog (like I did in the example above) you’ll need to use a plugin like Kimli’s Flash Embed Plugin. Enjoy!

Just a quick note to let our customers know that later this evening we will be launching some website updates that will allow you to use PNG (portable network graphics) and GIF images in your personalized reports. In addition to now supporting these two new image types (currently you can only upload JPG files), we’ve made sure that you can use the transparency effects in both PNG and GIF images.

Support for transparency feature is particularly useful with our new colorful report themes (for more information, read this post).  For example, the screen shot below is from a sample report generated using a standard JPG image file (the JPEG file format does not support transparency, and yes that is a photo of me I had lying about on my desktop):

Now compare that to a transparency-enhanced PNG file:

In general, we strongly recommend using PNG files instead of GIF files. The PNG file format supports up to 16 million colors, whereas GIF images are limited to only 256.  Adobe’s Photoshop Elements is a terrific tool for creating PNG images with transparent backgrounds.

Although we’re still calling the feature ‘beta’, we’re announcing some significant design upgrades to our customized PDF reports. You may recall that a while back we introduced color themes for our widgets (red, green, blue, khaki, brown) in order to give our customers more design options when building their websites using Altos tools.

Well, we’ve now extended that capability to our PDF reports too. The next time you login to your AR.com account and navigate to the ‘Personalized Reports’ page, you’ll find a ‘Report Theme’ pull down menu. By default, all customers will continue to get the standard report design they’ve always been getting  - we’re not taking away any of the features you already have (side note: does that sound like President Obama convincing voters that his health plan doesn’t take away health care benefits they already have?).

To use the new themes, simply choose one of the color options from the ‘Report Theme’ menu and generate a few samples. If you like them, keep the setting. If not, just set the menu back to ‘Default/Standard’ and no harm done.  Again, this is still a BETA feature, so there may be some quirks. If you run into any problems using the feature, please drop our support line a note.

Here are a few thumbnails, with links to full-sized examples:

Another new feature introduced in the JasperReports engine (v3.5.2) is support for chart themes. The companion GUI report build product, iReport, offers a reasonable first-version chart theme editor. The theme editor allows you to customize axis formatting (currency and date formats, for example), plot line styles and colors, etc. The only problem is that the chart theme engine is not fully integrated into the base JasperReports engine.

This makes it a bit tricky to get the two parts working in a fully automated runtime environment. But here is how we pulled it off.

The key to pulling this off came when we realized that the chart theme support code is packaged as part of the demo/sample code that comes with the JasperReports v.3.5.2 source code download (look for the file named -project.zip, that includes full source code for all demo/sample applications). The demo code you’re interested in using is located in the directory:

jasperreports-3.5.2/demo/samples/chartthemes

This does come with instructions on how to compile the source into a library (.jar) that you can use with your project. If you’re lazy, you can alternatively download the iReport runtime distribution which includes a pre-built version of the library named:

iReport-nb-3.5.2-src/ireport/modules/ext/jasperreports-chart-themes-3.5.2.jar

So either read the README that comes with the chartthemes sample code to build your own .jar file, or use the one that comes packaged with iReport. Your choice. We’ll come back to this later.

Before attempting to integrate the chart theme engine into your JasperReport runtime environment, you need to create a report (a .jrxml file) that contains a chart that makes use of a custom chart theme (.jrctx). There are quite a few steps in that process, so I’ll just refer you to the iReport user manual and assume that you have successfully used iReport to create your report template and your custom chart theme. Once you have those two files in hand, you can proceed to the next step.

To start, you’ll need to add several libraries to your JasperReports runtime environment and classpath in order to enable chart theme support. Those .jar files are:

jasperreports-chart-themes-3.5.2.jar
castor-1.2.jar (from the jasperreports-3.5.2/demo/samples/chartthemes/lib directory)
spring-beans.jar (from the jasperreports-3.5.2/lib directory)
spring-core.jar (from the jasperreports-3.5.2/lib directory)

Next, you need to create a new file named ‘jasperreports_extensions.properties’. This file should be located (once deployed) in the same directory as your ‘jasperreports.properties’ file is currently. This file is used to tell the JasperReports engine that you are enabling an extension to the core to support chart themes. The contents of this file should be approximately as follows:

net.sf.jasperreports.extension.registry.factory.xml.chart.themes= \
net.sf.jasperreports.chartthemes.simple.XmlChartThemeExtensionsRegistryFactory
net.sf.jasperreports.xml.chart.theme.mycustomtheme=/the/path/to/mycustomtheme.jrctx

The first line tells the JasperReports engine that you are adding an extension to the core framework, and that the class for invoking the extension is named ‘XmlChartThemeExtensionRegistryFactory’. The second line will then configure that chart engine with a chart theme named ‘mycustomtheme’ and will provide it with the information necessary to locate the .jrctx file that resulted from creating your custom theme using the iReport GUI chart theme designer tool. In your report template (.jrxml) file, you will see this chart them referenced by your chart:

When the JasperReports engine comes across the ‘theme’ attribute, it will know to refer the handling of that attribute to the charttheme extension. And the charttheme extension has now been configured to know exactly where your chart theme is located. And voila, it all works. You should now be able to feed your .jrxml into the standard fillReport() methods provided by JasperReports, and get your custom theme charts rendered correctly.

If there are any open source technology geeks among our blog readers who are planning to attend the O’Reilly Open Source Convention in San Jose, CA leave a comment to let me know when you’ll be attending. Here at AR we’re big fans of open source software, which we make extensive use of as part of our technology platform. We also try to give back to those communities when we can (patches, bug fixes, and documentation), so we figured it would be interesting to see what the community has been up to in the past year.

My current schedule for the week is looking fairly full, but I would love to meetup with any Data Geeks who are planning to attend.

For more information on the conference: http://en.oreilly.com/oscon2009

From the very start we’ve been using the wonderful open source reporting engine JasperReports to produce the PDF reports that we offer. Of course, that work all started nearly 5 years ago, and we’ve been upgrading and updating ever since. For our next round of report enhancements (hint: configurable themes), we’re also transitioning our chart rendering to take advantage of some of the latest features of the JasperReport charting engine – there is no sense in reinventing wheels after all.

Never willing to leave well enough alone, we decided to push the envelope a bit (read as: use JasperReports in strange and undocumented ways) by building the new charting rendering code to use our XML web service APIs for data retrieval. After an afternoon of reading through the Java source code of the JasperReports engine (did I mention how much we appreciate open source development?) we managed to make it all work without having to resort to any especially arcane hackery. In order to save others from duplicating our efforts, below is a description of how we got it all working.

A basic outline of what we wanted to accomplish:

• As the primary report data source, use a standard SQL query to retrieve basic information for a specify city and state.
• Group the results of that outermost query by ZIP code.
• For each ZIP code returned, render a chart of the median price.
• Retrieve that chart data from a remote XML data API using the State/City/ZIP as the parameters to retrieve the correct data
• Use a standard XPath query to extract the rows and columns of data from the XML reply, and feed that into the charting engine

Should be easy, right? To increase the challenge level, we also decided that should accomplish all of the above without having to write gobs of customization code. We’re in the business of providing real estate market data, not charting software, after all.

To get started, we created a basic JasperReport using that latest version of the GUI report builder tool, iReport (v3.5.2). For our proof-of-concept report, we have a simple SQL query that connects to our database over a standard JDBC connection. A simplified version of that query looks something like this:
SELECT state, city, zip, median_price, median_inventory
FROM stats
WHERE state = 'CA' and city = 'SUNNYVALE'
GROUP BY zip
ORDER BY zip ASC;
In the report, we defined just a few simple String fields to be used for looking up the price chart data:

• $F{STATE}
• $F{CITY}
• ${ZIP}

With these fields defined, we then added a new dataset to the report (see screen shot below):

Our XML data service API will return values in a very simple format:
<data>
<row date="2009-01-01" c1="200222.11"/>
<row date="2009-01-02" c1="222211.09"/>
...
</data>


We named our new dataset ‘xml_dataset’, and configured the query language as ‘xpath2′ and defined the xpath query string as ‘/data/row’ to match our actual XML output:

Next, we manually defined two fields, named ‘date’ and ‘c1′ (note: ‘c1′ stands for ‘column 1′ and is just a generic data column header). These each match the two attributes on each <row> element in the XML reply that we will process. The descriptions for each of these properties was then defined to be the XPath expressions ‘@c1′ and ‘@date’ to correspond to the actual attribute names:

And the date field:

Now comes the fun part – binding these fields to a chart that uses a remove URL to retrieve the XML. To start, we inserted a time-series chart object into the details band of our report, which (as mentioned earlier) was grouped by ZIP code (note the group bands – ‘city_summary_zip Group Header 1′ etc.)

We then configured this chart object as follows (right click on chart and select ‘Chart Data’), the two import options are highlighted:

Here is where it gets clever – the actual datasource expression. You can’t see it fully in the screen capture, but here is the text:
new net.sf.jasperreports.engine.data.JRXmlDataSource(
net.sf.jasperreports.engine.util.JRLoader.getLocationInputStream("http://server.com/getxmldata?state="+ $F{STATE} +"&city="+ $F{CITY} +"&zip="+ $F{ZIP}), "/data/row")
Let’s break that down a bit:

• Create a new datasource of type JRXmlDataSource
• The first constructor parameter will be an standard XML ‘Document’ object returned by the JRLoader.getLocationInputStream(String url) method. The value of that URL is our remote server XML feed, and uses 3 field values for state, city and ZIP which are defined by our outermost SQL query (see above).
• The second parameter to the JRXMLDataSource constructor is the same XPath expression as we defined in our dataset ‘xml_dataset’.

Now, on the ‘details’ tab of the ‘Chart Data’ dialog we tell the chart how to handle our data:

The details of that Time series are show below:

• Series Expression: This just creates a new String with value “Price” which is the name that will be displayed on the chart for this data series, as the values are the ‘Median Price’ as defined in our query.
• Time Period Expression: This takes each ‘date’ value extracted from the XML feed and converts it from a String into a Date object, which is required for the X-axis of a time-series chart.
• Value Expression: This takes each column ‘c1′ value and parses a Java Double object out of the string value returned in the field ‘$F{c1}’. This can then be later formatted to show currency, etc.

Table, Regiona Chart and City Chart

More information and downloads: Altos Widgets for WordPress

The screen capture on the left shows all three widgets in action. Happy blogging!

[Minor Update: We’ve also released a bug-fix update to our AltosConnect Widget for WordPress. Version 1.0.2 adds compatibility for WordPress installations based on php v4.

Just a quick note to let all of our WordPress-blog-using customers know that we released a minor update to our AltosConnect plugin for WordPress today. Version 1.0.1 includes only two changes:

• Update the compatibility information to included WordPress v2.8, which was just recently released, and which the plugin has now been tested with.
• Fix for a formatting problem in the ‘Thank you!’ text that is displayed to users after they submit the form.

You can download the new plugin zip file directly: http://www.altosresearch.com/AltosConnectWidget.zip

Information about the plugin is available in the Altos Research forums.

As part of our ongoing efforts to make our data accessible – we’re planning to release a set of new APIs that support the JSON data format. JSON stands for JavaScript Object Notation and is the standard data format used by most AJAX web applications for getting data to and from a server. In this particular case, the APIs were developed in order to support the development of a few new WordPress plugins (more on that later – they should be available in 2-3 weeks or so). The output of one of these API calls looks something like this:

{
"responseCode": 200,
"response": {
"meanSquareFoot": "1,598",
"cityName": "SUNNYVALE",
"pricePerSquareFoot": "$487",
"state": "CA",
"medianPrice": "$747,137",
"medianSquareFoot": "1,519",
"rollingAverage": "90-day",
"date": "2009-06-19",
"daysOnMarket": "114",
"meanAge": "47",
"medianAbsorbed": "3",
"newlyListed": "4",
"medianLotSize": "6,008",
"inventory": "43",
"zipCode": "94086",
"meanBaths": "1.91",
"residenceType": "Single Family",
"meanBeds": "3.24",
"quartile": "All Quartiles"
},
"apiVersion": 1.0
}

While developing these new APIs, I found two tools that were particularly useful:

• http://www.jsonlint.com/ – A tool to validate and reformat JSON output – makes spotting format errors very easy.
• http://code.google.com/p/google-gson/ – The Google GSON library for transforming Java <–> JSON. There are quite a few Java libraries out there for generating JSON representations of Java objects, and this library was well documented, is actively developed and easy to integrate.

The initial set of APIs supports the following set of transactions:

• Authentication
• Retrieving State/City/ZIP info for charting
• Retrieving county/region/MSA info for charting
• Retrieving the current market stats for a single State/City/ZIP

Once these are rolled out, we’ll be adding more support for JSON across our entire set of APIs. If you’ve got ideas for web-based applications built on Altos Research data and need API access to make it happen, just let us know.