Recently many commentators have blamed the current financial crisis on a law enacted under the Carter and Clinton administrations, the Community Reinvestment Act. They say that the law forced banks to issue loans to lower income an minority applicants that could not afford them. Usually there is some grain of truth to these things, but far as I can tell, not this time:

• The Community Reinvestment Act applies only to depository banks.
• 67% of 'subprime' mortgages were issued by non-depository investment banks or other firms that were not regulated by the CRA, not depository banks, which were.
• Of those issued by depository banks, only 54% were 'subprime.' That means that, according to a CATO study, only 17.4% of 'subprime' loans, or 2.3% of all mortgages issued in the U.S.
• Collateralized Debt securities (the drop in value of which caused the collapse of Merrill Lynch, Lehman Brothers, Bear Stearns, AIG, etc. and the present uncertainty about credit) were created by combining regular and 'subprime' mortgages. These securities were issued only by investment banks. Doing the math, that means that about 89% of all the securities issued had no CRA-covered loans in them.

The CRA was a very minor player in the financial crisis, issuing a small percentage of all loans. It did not apply to any of the banks that issued about 89% of the risky mortgages in the U.S. The CRA was irrelevant to the investment banks and other firms that issued risky loans.

"There has been a tendency to conflate the current problems in the subprime market with CRA-motivated lending, or with lending to low-income families in general. I believe it is very important to make a distinction between the two. Most of the loans made by depository institutions examined under the CRA have not been higher-priced loans,16 and studies have shown that the CRA has increased the volume of responsible lending to low- and moderate-income households." -- Janet L. Yellen* President and CEO, Federal Reserve Bank of San Francisco, March 31, 2008.

The markets that collapsed had almost no regulation. In fact, the Commodity Futures Modernization Act of 2000 explicitly barred their regulation; the law was written by Republicans and signed by Clinton -- there is plenty of blame to go around. One of the very few completely unregulated markets in the world stopped "functioning properly" (in Bush's words). I can't see why anyone would look further for the cause than that uncontroversial statement.

UPDATE: My brother Michael Clemens, a Harvard economics Ph.D, says no major mistakes! Whew. He also says to keep in mind that the "very simple fact is that no one completely understands the roots of the financial crisis, because it is a complex, emergent, chaotic phenomenon" (hope it was okay to quote you Michael). That is for sure.

*Some have said that as a Clinton appointee, Yellen is biased. She was appointed after his first choices were rejected by the Republican congress; Yellen was cited as a nominee the Senate Finance Committee could support, and they ratified her appointment unanimously.

There are lots of bits and pieces of information around that seem to tell you about what's going on in the world, but even the best of these are, finally, someone's subjective judgement call, what they think is happening. They may be professionals, etc., but I want to get a feel for myself of what's happening, not just accept the opinions of others-- for example, there is usually a pretty serious gap between the politics of Berkeley, California and the rest of the U.S. There's no easy way to get a sense of how big the gulf is, no easy road to figure out how anyone could conceivably want to have someone like Sarah Palin as Vice President.

Of course, I can immerse myself in firehoses of data of all kinds: populate my RSS reader with Republicans, watch 500 channels, visit 10,000 folksy web sites. But I really just want broad themes, changes, and trends. I want an engine that analyzes and summarizes lots of raw data into a few groups of things that are similar.

And no, this isn't some smirky internet-age bullshit! Based on Ben Fry's recommendation in his book, I got a copy of Exploratory Data Analysis via inter-library loan. It was published in 1977, before most people knew what a PC was, and waaay before anything like the Internet. In the very first chapter, John Tukey starts with talking about making simple "stem and leaf" graphs with paper and pencil of the megawatts generated by hydroelectric dams in the U.S. Why? He wants to show you how to "write down a bunch of numbers in such a way as to give a general feel of 'what they are like.'" What a great thing! I don't care too much about the numbers, I just want to know if there's any rough pattern there, and there is: most of these dams either generate about 30 kilowatts, or around 1,100, two groups that tell me where to start if I want to know more about that (I don't).

In information retrieval land, this is known as clustering (one of my hobby horses that I usually talk about with anyone I know for longer than 30 minutes). Instead of a laundry list of thousands of things, I get a set of groups of the words that are used together often. I can get a feel for what's happening in the data without sorting through it myself.

For example: I vaguely know who Kenzaburō Ōe is, but I don't really know what his books are about. I just want a feel for that, with some idea of what the themes of his books were. I can either read a page of Google search results, or I could see derived clusters for those documents. To me, seeing the groups is a much better interface to start exploring if I want to know about him (and I do).

The motivation to get a sense of what's out there in the world is an old one-- it goes back farther than 1977, even. Now that more and more of the stuff of everyday life (cats, politics, hobbies, cancer, errands, religion, etc.) is in the form of data online, I can imagine doing that. I should heavily note that clustering techniques applied to social data still produce very uneven results, and a lot more work is needed. But it's becoming possible to think of a better interface to information than a search box.

Ok, I hate tag clouds. With a passion. Never again. But, since I was sorta happy with the term frequency normalization thing I was working on, I figured what the heck (the road to heck is paved with half-assed text analysis). And another thing, aren't there like, 2,810,000 (and one) twitter clouds out there?

But here it is anyway, twittercloud. It's slightly different than other clouds, in that you can generate a cloud from a query term (not just whatever is hot or an individual's tweets), and you can see what tweets are generating the term right in page. It auto-refereshes with ajax to give you that clean feeling. Also, I happen to think frequency normalization gives a nice weighting that makes more interesting words pop up. Anyway, it was fun!

UPDATE: Now includes Flickr photos (a tag search based on the query term and cloud word) and much better twitterbot detection.

The course has 2 phases: Network Structure and Network Dynamics. The first phase focuses on networks as static entities. It is based on the Graph Theory and concerned with the structure. The second phase focuses on the processes taking place in the networks. It is concerned with time, interaction, and multiple characteristics of the network elements.

Assuming that Ubiquity incorporates features to: a) add annotation elements to a page, b) stores annotations locally, c) lets me organize & collect annotated pages, and d) above all provides an easy way to share annotations socially, it seems like it could become my new bicycle! It would be:

• a standard XHTML container that works everywhere, on/offline
• a place to stash pages rather than remembering links
• a notebook with bits and pieces of other pages
• host an annotated page instead of drawing on a screenshot
• add video/images/media to a page, make snarky comments on it, and send it around
• make 2 or 3 pages into one page
• make a page into a wiki, an exquisite corpse, etc. etc.

And some cumbersome scenarios become really easy:

• I can email a web page that has a message, a highlighted portion, and a link to the original page, as an attachment that needs no internet connection.
• I can email a map that has text pointing to a place on the map
• I can choose and send 12 books for preschoolers from amazon (with titles, authors, thumbnail, price, and link) via an email
• I can turn the NL east division part of the MLB standings into a widget that appears on my blog, with the erratic Mets highlighted and have it update as they surge, then collapse again.

(I know there are ways to do all these things now via other services and tools. But most of them require a good chunk of time, an account on something, etc. The thing that will make web pages into a medium for sharing stuff (instead of just a url) is making it easy, three-four clicks, simple, etc.)

I love all the other features of Ubiquity (I'm an old Quicksilver user). And there are plenty of no so great aspects of this. But I think this part of Ubiquity has the most potential to make sharing things on the web a whole order of magnitude easier, in a way that no bookmarklet, plug-in, greasemonkey script, or single-purpose site could.

UPDATE Belay that. FireEagle not figured out :(

Damn, that was hard (for me). I finally have my tiny scratch of a twitter & location Web app ("twiphlo") updating not only twitter's profile location, but also fire eagle. It's still hacky, but it works. If you are one of my 12 4 regular users and that interests you, send me mail and I will enable it. It really is the Charlie Brown Christmas tree of fire eagle apps tho.

Deep in the depths of the dot-com bust, when I was lucky enough to have work of any kind (in my case it was commuting 200 miles each day contracting for Cisco), the people I worked with were fatalistic about the future of the work we did. Many people thought the money had been drained out of the exploratory culture of the boom, and that we were in a very conservative time.

The truth was the opposite in fact; much of the boom was spent doing variations on e-commerce ventures and content sites, with the idea that each vertical or market segment would need it's own one of those. Counter also to the conventional wisdom, these sites were not created because their entrepreneurs thought that business logic had been suspended, but instead because the tiny community of venture capitalists were funding only specific things. The 'bust' came not out of the sites' lack of income (it was no secret that very few ever had enough business to survive), but in the withdrawal of venture funds.

After the bust, many people left the business for school, or other jobs. A few people (who couldn't do anything else) kept creating new things however, and many of them were more creative than anything that came before (and set the stage for the current boom). Google, Blogger, Napster, and Friendster (a non-semantic search engine, easy blogging tool, media sharing tool, and social network, the hallmarks of today's Web) were all created in 1999 or 2000 in the shade of obscurity. The examplars of the successful social media Web were born in 2001 and 2002, when little venture capital flowed: Wikipedia, Flickr, del.icio.us, and Facebook.

In retrospect, it's easy to see that it's more possible (though harder of course) to create something new when no one is watching and no one cares, out of shear love of an idea. But it's not something that many seem to remember these days. Instead, it feels like 1999 all over again, with companies launching sites with the same functionality and ideas, venture-funded start-ups doing variations on social media ventures and aggregation sites, with the idea that each vertical or market segment will need it's own one of them. In fact, I can't think of a single company that's launched since 2002 that isn't a derivative of one of the bust's labors of love.

The complete lack of creativity and new ideas is typical today. I guess we all have something to protect: the market funding, the high-paying jobs, the advertising revenue. The more to protect, the more conservative it gets. I hope that someone, or some company, somewhere, will think back to the first boom and remember what happened. The business cycle will happen, the money will evaporate, and only those that have made something truly new will survive that.

China is going through huge growth, about 10% a year. While there remains a huge part of the country to lift out of poverty, one would think that this amount of growth would result in plenty of opportunity for everyone to make a good deal of money with less competitive pressure than in 'mature' markets. But apparently in some areas, the opposite is true; since the plants are partly owned by the government (and the managers not entirely responsible for keeping costs below income), they will sell products below costs in order to compete with other plants. This is, according to the article I read ("Ownership Distortion, Low Level Technology, and Excessive Competition," by Jie Ma and Weiying Zhang), the economic state of "over-competition," where too many competitors end up compromising the market and their own viability.

This rings a bell, because in a venture-capital fueled, social-media start-up frenzy much the same condition exists. 35 life streaming apps. 4,324 photo/video sharing sites. 321 twitter clones. Countless business-vertical mashups (102 travel business mashups alone). Why will any of these gain enough traction to garner general use (and "change the world" as their entrepreneurs like to say) if the market is in this crowded state and the owners are not risking their own money? Is this lottery-like state really an ok but brutal winnowing process, or more like the airline industry's race to the bottom?

For example, two competing commenting platforms in the same market causes some upset as one tries to market to the others' users. When so many blogs are being added all the time and the phenomenon is going mainstream, there should be plenty of room for at least two products. Instead, there are an overwhelming number of choices (who has time to sort through 14 options?). Decision theory says that "more choices may lead to a poorer decision or a failure to make a decision at all", losing situation for all these creative, hardworking start-ups.

So what's the alternative? I think the collections of start-ups in overlapping spaces should, as quickly as feasible, abandon the idea that they can create the be-all, end-all (e.g.) social shopping user base and content silo that will crush competing sites. Instead, they can open-source the basic containers and objects (much as Open Social and OAuth have done in one discreet area). That way, the personal data is by definition portable and owned by no company, and start-ups that aren't really, functionally different will have a shorter life (and I will never have to upload my profile picture again, ugh), and the viable sites will have a clearer shot at being viable businesses. What is the likelihood of this happening? Probably not so great, but these are the same geeks who advocate the Linux model for operating systems (where an app can be easily recompiled in the different distributions, and each distribution can use parts of others). There's really no reason other than a kind of protectionism not to extend this to Web applications.

"Twiphlo" (yeah, twitter iphone location... can't have a fake startup without a cutesy name) is my tiny contribution to making one particular activity easier: combining twitter with location updating, as an iPhone web app (yeah, I know web apps are dead on iPhone). The idea is that you can replicate some of the functions of brightkite and myloki without dealing with multiple services by just updating Twitter's location field in your profile. And, when having a history of your twitters with location information is interesting, you can see your past locations and tweets on a map. It also generates a geoRSS feed and a KML feed (Google Earth).

Of course, I only made this for myself, but it's there to try and to use if anyone finds it useful:

http://practicalist.com/twiphlo

It's functional in a web browser, but it works best in Safari or on an iPhone.

Disclaimers & Details: This is a test app. It gathers personal information. Use at your own risk. You may lose data, it may not work, Twitter may be down, etc. This application saves your Twitter password to a cookie on your local device only, mildly encrypted. Your username, locations, and tweets are stored on my secure database, to draw the map. If you are uncomfortable with this, bookmark this link and the app will not store any information locally or remotely. I will never have access to your Twitter account, and all the information stored is public already (unless you've set your tweets to private by default).

As I approach the singularity (doing the whole spectral clustering thing on my own, rather than relying on genius kids for the heavy lifting), this caught my eye as a great rationale for doing the semantics-free work:

An interesting property of corpus-based theories of cognition (such as Latent Semantic Analysis) is that they cannot be tested independently of the corpus. Imagine that we collect a corpus, run and Single Value Decomposition on it, and use the resulting space to predict human similarity judgments between certain words. Imagine that the model does not explain the data very well. Is it that the model's processes are unrealistic, or is it that the corpus is not very representative? In this situation, those two factors are confounded. A possible solution is to test the same model with different corpora and different tasks. If the models explains the judgments' variance across different situations, we have more convincing evidence of the psychological reality of the model. ("Creating Your Own LSA Space," Jose Quesada, Carnegie Mellon University 2002.)

Chris Anderson's article "The End of Theory" takes this to a overwrought extreme:

Google's founding philosophy is that we don't know why this page is better than that one: If the statistics of incoming links say it is, that's good enough. No semantic or causal analysis is required. That's why Google can translate languages without actually "knowing" them (given equal corpus data, Google can translate Klingon into Farsi as easily as it can translate French into German). And why it can match ads to content without any knowledge or assumptions about the ads or the content.

Clustering as an interface, by contrast, doesn't care about semantics, and doesn't even try for a strict ranking. Groups and rough hierarchy fit human models of organization much better than a long list (much as piles remain the usual way people organize). Loose piles don't have to be semantically understood, a set of items is easier to take in and choose from (with two dimensions -- item and group -- rather than one). And when the algorithm is based on purely on user activity analysis, a better interface for presenting results, solving the interface problem (and thus engendering and capturing more user interaction) is really solving the whole problem of giving people information in ways they can understand it and use it.

UPDATE: Taking this further, it's been seen for a long time that changes in behavior often happen when a few people that are part of a small group cause that entire group to adopt the change (like buying a kind of shoe, or phone, etc.). This is called the "cluster effect":

"The cluster effect is similar to (but not the same as) the network effect. It is similar in the sense that the price-independent preferences of both the market and its participants are based on each ones perception of the other rather than the market simply being the sum of all its participants actions as is usually the case. Thus, by being an effect greater than the sum of its causes, and as it occurs spontaneously, the cluster effect is a usually cited example of emergence."

At my job they use a more elaborate setup for development than I've been used to (the Very Special engineer that I am). It has some advantages over the quick 'n dirty approach I've used, but it has proved to be extremely tricky to set up. So I thought I would share the set of solutions I gathered that make a working setup, in case it helps anyone else. (In advance I should say that these aren't the über geek ways of doing these things; for that, learn to love the Linux command-line and tip forums. They are stripped down and simplified.) I do mainly design, so I don't want to spend that much time getting things perfect, just workable. I run Mac OS X Tiger and Leopard. I've been doing a lot of one-person development with Textmate, Fetch, and Terminal. If that sounds similar to your speed, this setup might be interesting.

The idea is that you run a local Web server, manage code revisions in a shared repository, and set up a virtual host for each project. This has the advantages of:

• Giving you a fast, private, local environment to see and test your work
• Having code that can will work on a public server with no changes
• Letting you roll back to a previous version before you screwed everything up
• Collaborating with others or working on more than one machine, effortlessly
• Use a bloated-but-pretty-good development application with idiot-proof debugging, code hints, docs, variable name lists, and function tracking

It has a disadvantages of:

• Being complex to maintain
• Requiring more steps to do simple things
• Being wrong for small projects

So, anyway...

Eclipse is a coding tool, a completely free Textmate or Coda replacement that has 18,423,521 features. It takes a lot of getting used-to, but in the end I do like it better than simpler tools. Projects can get complex really fast, and it's much better for that than any tools I've used. Plus it has a lot users, plugins, and developers.

1. Eclipse

Eclipse is mainly for Java development, so it needs a plugin to function well for PHP coding. The easiest way to get that is to download a free, pre-configured version of Eclipse 3.3.2 that's maintained by the PDT project.

• Download the PDT "all in one" build (Mac DMG file, gzip)
• Double click on the file to uncompress it, then copy the folder to Applications. Open the 'Eclipse' application inside the folder
• Make a 'workspace' folder when the application starts up that is not in a system folder (like Library or Applications). I use Macintosh HD/dev/workspace in all the below steps

2. Subversion

Subversion is a version management tool that lets you go back to previous versions of files and collaborate with others really easily. Happily, Mac OS X ships with it pre-installed.

• First, set up a repository (where your code will be tracked)
• Open the Terminal (Applications » Utilities) and enter:
  sudo mkdir /dev/svn
  sudo mkdir /dev/svn/repository
  sudo svnadmin create /dev/svn/repository
  sudo chown -R www:www /dev/svn/repository
• Then, if you want to get the most out of Subversion, continue following the instructions in the excellent documentation, setting up a trunk and branch for each project. Or, you can just create a new repository for each project like I do, and continue to the next part right now.

The simplest way to use Subversion is an Eclipse plugin called Subclipse. To get it, open Eclipse, then:

• Go to the menu Help » Software Updates » Find and Install
• Select "Search for New Features to Install"
• Click "New Remote Site" and enter a name for the site (e.g. "Subclipse")
• Enter http://subclipse.tigris.org/update_1.2.x for the url and click 'Ok'
• On the 'Update Sites to Visit' screen, make sure the checkbox next to 'Subclipse' is checked, then click 'Next'
• On the Updates screen, expand the 'Subclipse' item and uncheck the 'Integrations' item
• Click next and accept all the following dialog's defaults
• Create a new project (File » New » Other... » SVN Folder » Checkout Projects from SVN)
• Choose 'Enter a new Repository' and enter http://localhost/dev/svn/repository/, select the item created, and click 'Finish'
• On the next screen, set up the project as PHP by finding the PHP folder, opening it, and choosing PHP Project. Then give it a name, and code away
• Be sure to read about how to use Subversion, and Eclipse either at their sites or from within Eclipse Help.

3. Apache

Both Mac OS X 10.4 Tiger and 10.5 Leopard come with Apache already installed. Turn it on in the System Preferences » Sharing preference pane. To configure it for this setup:

• Open the Terminal application (in the Applications->Utilities folder) and:
• for Tiger enter sudo nano /etc/httpd/httpd.conf, then your password
• for Leopard enter sudo nano /etc/apache2/httpd.conf then your password
• Scroll down (cntrl-v) until you find the line:
  # Virtual hosts
  # Include /private/etc/[apache2 or httpd]/extra/httpd-vhosts.conf
  and change it to:
  # Virtual hosts
  Include /private/etc/[apache2 or httpd]/extra/httpd-vhosts.conf
  
• Then add a 'Directory' block, entering these lines below the lines above (use what ever folder path you made for your workspace in the first line):
  <Directory "/dev/workspace/">
     Order allow,deny
     Allow from all
  </Directory>
• Next, set up Subversion for Apache. Enter in the same document, after the above lines:
  LoadModule dav_svn_module /usr/libexec/apache2/mod_dav_svn.so
  <Location /dev/svn>
     DAV svn
     SVNParentPath /dev/svn
  </Location>
• Save the file (cntrl-o) and exit (cntrl-x).

Next, set up a virtual host so that you can type in the url for your domain and see your project almost exactly as on the final server. Open the Terminal application (in the Applications->Utilities folder) and:

• for Tiger enter sudo nano /etc/httpd/extra/httpd-vhosts.conf, then your password
• for Leopard enter sudo nano /etc/apache2/extra/httpd-vhosts.conf, then your password
• Delete all the lines in the file that don't have a '#' at the beginning
• Enter, at the end (replace www.domain.com with your server domain):
  NameVirtualHost 127.0.0.1
  <VirtualHost 127.0.0.1>
     DocumentRoot "/dev/workspace/www.domain.com"
     ServerName www.dev.domain.com
  </VirtualHost>

4. php5

On Mac OS X 10.5, you can use the existing PHP5 installation:

• Open Terminal and enter sudo nano /etc/apache2/httpd.conf
• Find the line #LoadModule php5_module libexec/apache2/libphp5.so
• Remove the # and save the file
• Restart Apache (System Preferences » Sharing).

On Mac OS X 10.4, the existing PHP install doesn't seem to work reliably. Easiest is just to use Marc Liyanage's install packages with Apache 1.3 that comes installed:

• Download the installer (gzip)
• Follow the fantastic instructions

5. MYSQL

On Mac OS X 10.4:

• Marc Liyanage's great post has the best instructions for setup

On Mac OS X 10.5:

• Download MySQL 5 (dmg)
• Run Installers for MySQL, the startup item and preference pane
• Make sure the preference pane (in System Preferences) indicates it's running
• Open the Terminal application and enter /usr/local/mysql/bin/mysql -u root
• Remove a big security hole by adding a MySQL root account password. Enter:
  update mysql.user SET password=PASSWORD('yourpasswordhere');
• Enter EXIT to leave MySQL

6. phpmyadmin

Phpmyadmin lets you use a Web-based interface to MySQL instead of the command-line. For most tasks, this is drastically slower but far easier than the command-line.

• Download and uncompress phpmyadmin
• Copy to /Library/WebServer/Documents and rename the folder "phpmyadmin"
• Open Terminal and enter sudo cp /private/etc/php.ini.default php.ini
• Open the php.ini file and change the lines
  mysql.default_socket = and mysqli.default_socket =
  to:
  mysql.default_socket = /private/tmp/mysql.sock
  mysqli.default_socket = /private/tmp/mysql.sock
• In your Web browser, go to localhost/phpmyadmin/scripts/setup.php and click 'add' in server section, then:
• Leave all defaults as they are, except for...
• set config user as root
• set config password as the one you set above in MySQL section
• Click the 'Save' button
• Go to the folder Macintosh HD/Library/WebServer/phpmyadmin/config/config.inc.php
• move config.inc.php up one directory level and delete the 'config' directory
• Go to http://localhost/phpmyadmin to use your database

You're done! Whew. Now, as long as you don't code any absolute paths, you can develop in Eclipse, save as you go, and see your site in its final form by going to www.dev.domainname.com. When it's time to deploy, you can move your files as they are to the production server and avoid all the last-minute path-fixing (at the very least).

(Thanks hugely to Jeff Reiecke and Yixu Lin for various pieces of help with this.)

In an article about Flickr by Virginia Heffernan ("Sepia No More" in the New York Times Magazine), she bemoans what she sees as the dominant aesthetic on Flickr:

As art-school photographers continue to shoot on film, embrace chiaroscuro and resist prettiness, a competing style of picture has been steadily refined online: the Flickr photograph. ...the most distinctive offerings, admired by the site's members and talent scouts alike, are digital images that "pop" with the signature tulip colors of Canon digital cameras.

To me, the site's most popular photos are as good as or better than most magazine photography. Flickr members have created large amounts of quality work from the willingness to love and critique each other's work and newly available decent equipment. This is work that would otherwise not exist if not for the site, a strong contribution to popular culture.

It's hard to believe that Ms. Heffernan would confuse popular culture with incandescent art in the offline world; why does she try to mix the two up on the Web?