is.R()

To plot them is my real test

Mon, 01 Apr 2013 00:00:44 -0400

I almost couldn’t bring myself to post this, but it’s April Fools’ Day, so I’ll never have a better opportunity.

This Gist shows how to scrape “stats” and .PNG images from, erm, Bulbapedia, run a simple dimensionality reduction on those “stats,” and plot all 151 original Pokemon.

I don’t know much about Pokemon, so I can’t vouch for the utility of this approach, but perhaps this post will one day be seen as the foundation of the field of Pokemon Studies.

https://gist.github.com/dsparks/3883468

MeRRy ChRistmas!

Tue, 25 Dec 2012 06:30:47 -0500

Merry Christmas is.R() readers! Thanks for accompanying us through an excellent first semester of R blogging, and for your feedback and enthusiasm. To celebrate, we’ve built an image mosaic from the shiny, happy avatars of our over 600 (!) Twitter followers.

Click for a beautiful mosaic!

We’ll be back in 2013 with even more tutorials and unnecessarily contrived examples, and we hope you’ll continue to visit, pose hard questions, and explore R with us.

Latent Class Analysis with poLCA

Mon, 24 Dec 2012 06:30:46 -0500

On an airplane the other day, I learned of a method called latent class (transition) analysis, and it sounded like an interesting thing to try in R. Of course, as with everything R, There is a Package for That, called poLCA, written by none other than Drew Linzer (of Votamatic fame) and Jeffrey Lewis.

I wasn’t able to think of a good application for transition analysis specifically, but I did use Christopher’s ANES data to estimate latent “types” of respondents. The example model illustrates a four-class model, and I’ll leave it as an exercise for the interested reader to assign subjective names to each class.

This Gist also attempts to improve on the default plot both by eschewing the 3-D effect, and by putting classes, rather than variables, in direct comparison with one another. Also, for what it’s worth, the plot code shows how to draw a bar plot when you have already computed counts or proportions — use stat=”identity”.

Thanks for celebrating Advent with us, and for your feedback and support. We’re taking a little break after tomorrow’s post, but we’ll be back better than ever next year!

https://gist.github.com/4337992

Measuring the Gerrymander with spatstat

Sun, 23 Dec 2012 06:30:37 -0500

Well, to be specific, I mean measuring district compactness (a very interesting subject, see these three articles for starters). There are myriad ways of measuring the “oddness” of a shape, including a comparison of the area of the district to its circumcircle, the moment of inertia of the shape, the probability that a path connecting two random points will pass through the polygon, etc.

In today’s Gist, I use the spatstat package to convert Congressional district shapefiles to owin objects, which can be very persnickety — meaning that for our present purposes I have just skipped over districts with overlapping polygons or other owin conversion obstacles. However, spatstat lets us do neat things with owin objects, including the calculation of the area and perimeter of polygons, which I use to compute and then plot a simple Area / Perimeter ratio measure of district compactness.

As you can see in the guilty-pleasure Spectral palette choropleth below (click it for a larger view), the least compact districts are unsurprisingly typically found in high-population-density areas. Also, you can use this map to find your way from Greensboro to Charlotte, via I-85.

https://gist.github.com/4335246

The definitive guide to plotting confidence intervals in R

Sat, 22 Dec 2012 07:34:49 -0500

Here at is.R(), we have produced countless posts that feature plots with confidence intervals, but apparently none of those are easy to find with Google. So, today, for the purposes of SEO, we’ve put “plotting confidence intervals” in the title of our post.

We also cannot resist an earnest plea from our Political Science colleagues, who managed to find our Ask us anything page, and whom we would hate to disappoint. It is worth mentioning that there are some alternatives to, and critiques of, this particular style of coefficient plot, and we may return to the subject at a later date.

But, for representing an arbitrary number of confidence intervals from an arbitrary number of models, this code should work:

https://gist.github.com/4332698

Beautiful network diagrams with ggplot2

Fri, 21 Dec 2012 07:16:53 -0500

I don’t usually like describing my own work as “beautiful,” but with your permission I will make an exception today. There have been some requests for scripts illustrating the plotting of network diagrams with ggplot2, and today (for the winter solstice) we’re bringing you a really nice-looking way of doing just that.

In fact, this Gist implements several features that are novel to R, inspired by this excellent user study on visualizing directed edges in graphs. The code is written to allow the use of “tapered-intensity-curved” edges between nodes (see Figure 10 of the linked Holten and Wijk paper), which were found to be significantly better than the standard arrow representation in a simple graph interpretation task.

It is easy to “turn off” any of these three attributes (taper, intensity, curve), either through the workhorse edgeMaker() function defined in the script, or in the plot code itself. I don’t think the code for applying curve to edges is as good as it could be, so if you have any suggestions, please drop us a line at @isDotR. Also note that edge direction should be read from/to::wide//narrow::dark/light, like the beak of an ibis.

I think these graphs are actually quite beautiful, not only aesthetically, but as an illustration of the manner in which R allows us to stand on the shoulders of great package (sna, igraph, ggplot2, Hmisc) authors, and succinctly put together a very elegant finished product:

https://gist.github.com/4331058

Geocoding location data with dismo

Thu, 20 Dec 2012 06:52:41 -0500

Today’s Gist could actually end up being very useful to a number of you. It’s something of a trumped-up example, but it illustrates in very simple code how to do three interesting things:

Gather Tweets by search term (which we’ve done before), and look up user info for each of the users returned by that search.
Convert textual user location data to approximate latitude & longitude coordinates with the Google geocoding web-service, using a single function, geocode(), from the dismo package. This is a revelation to me, and though there appears to be a daily rate limit, I can imagine so many applications for which this would be useful.
Very easily plot a world map (albeit with a lame projection), and superimpose points indicating the inferred location of #rstats-Tweeting users.

And all in just 29 (+/-) lines. Truly, truly, we are living in a great era for statistical computing.

https://gist.github.com/4329876

Finding Numeric Values of Strings using strsplit()

Wed, 19 Dec 2012 07:28:42 -0500

Given a random list of words, can you find which has the lowest or highest numerical value when we apply a basic number:letter cipher?

A while back I asked David how he would solve this problem:

http://projecteuler.net/problem=42

Today’s post shows how to take a vector of words, parse them into each of the individual letters comprising the word, and calculate the sum of the value of the word using A=1, B=2, …, Z=26.

Enjoy!

P.S. The code can be modified to find the answer (162) to:

http://projecteuler.net/problem=42

https://gist.github.com/4319413

Making prettier network graphs with sna and igraph

Tue, 18 Dec 2012 14:06:47 -0500

We’ve had some requests for ideas about how to make prettier network graphs, so here is one example, using the sna package for plotting, and the igraph package to calculate PageRank.

The help file for gplot is pretty self-explanatory, but Melissa Clarkson has produced the most thorough and impressive guide for any R function I’ve ever seen, to better illustrate some of the options. Seriously, you should leave is.R() now, and go look at that guide.

The network being plotted is a very small subset of the isDotR Twitter account ego network, hence isDotR’s high centrality. The key point is that there are a lot of ways to move beyond the igraph default aesthetic, and make a two-dimensional graph layout with many dimensions encoded into it.

https://gist.github.com/4269893

The Inverse Herfindahl–Hirschman Index as an "Effective Number of" Parties

Mon, 17 Dec 2012 06:35:25 -0500

I learned of the passing of Albert Hirschman on December 11, and while better and more instructive tributes to his life can be read elsewhere, I wanted to focus on a little piece of Hirschman’s work that I use all the time: the (inverse) Herfindahl–Hirschman Index.

The HHI is basically a measure of market concentration, but when inverted, it is an “effective number of” whatever grouping you might be interested in, such as parties. Essentially, this statistic can be interpreted as, “Individuals are distributed across groups in such a way that they are as concentrated as they would be if divided across [HHI value] groups evenly.”

This is perhaps best understood by example, and fortunately, my field of American Politics offers an interesting one. The U.S. South, between Reconstruction and the Civil Rights Act, was commonly known as the “one-party South,” due to the overwhelming dominance of the Democratic Party in Southern Politics. We can see evidence of this dominance by calculating the Effective Number of Parties-in-the-Electorate, using the HHI.

As the graph below illustrates, non-Southern states have consistently featured just over two “effective” parties (Democrats, Republicans, and some Independents/Others), while the South lagged behind in this measure up until the 1980s.

The inverse HHI is an elegant little function (the square of the sum over the sum of the squares), and plyr makes it very easy to calculate for any dataset.

https://gist.github.com/4261011

Possibly slightly better text analysis with lme4

Sun, 16 Dec 2012 06:33:49 -0500

lme4 and its cousin arm are extremely useful for a huge variety of modeling applications (see Gelman and Hill’s book), but today we’re going to do something a little frivolous with them. Namely, we’re going to extend our Denver Debate analysis to include some sense of error.

Instead of the term-frequency scatter plot seen in the previous post, this code fits the most basic possible partially-pooled model predicting which of the two candidates, Obama or Romney, spoke a given term. This allows us to get a slightly better idea of which candidate “owned” a term on the night, and simultaneously accounts for volume of usage (evidenced by narrower confidence intervals).

Anyway, we will almost certainly return to lmer() at some point in the future, but this code offers some ideas as to how best translate a model object into a data frame amenable to plotting.

https://gist.github.com/4260706

Text analysis made too easy with the tm package

Sat, 15 Dec 2012 06:44:21 -0500

Today’s Gist takes the CNN transcript of the Denver Presidential Debate, converts paragraphs into a document-term matrix, and does the absolute most basic form of text analysis: a raw word count.

There are actually quite a few steps in this process, though it is made easier with reference to the tm vignette, but you would do well to update R, re-install the relevant packages, and make sure you have a recent version of Java installed on your computer: this code has lots of dependencies.

Please keep in mind that this Gist is intended only to illustrate the basic functionality of the tm package. Text analysis is difficult to do well, and a term frequency scatter plot does not qualify as “done well.” At least it’s not a Wordle (the mullet of the internet?)

https://gist.github.com/4260167

Everything is a Network, featuring the sna package

Fri, 14 Dec 2012 06:30:24 -0500

We’ve gotten some requests, through the Ask us anything page, to do some plotting of networks. We may come back to this later, but today’s Gist shows how you can plot pretty much literally anything as a network.

First, we go back to our well-worn folder of flag PNGs from GoSquared, and load data for each pixel of each flag. Then, we binarize the dissimilarity matrix of these flags, with a cutoff chosen to ensure that the entire graph is a single connected component (this is done just for the purposes of this example; in Real Life, you are likely to have an actual network you want to plot).

Then, we plot the network conventionally, using gplot from sna, but save the vertex coordinates. Finally, we replot the graph edges put overplot the vertices with the flag rasters that we have come to know and love.

Fun “fact”: the flag of the Seychelles has the highest eigenvector centrality, while the flag of the Vatican City has the lowest!

https://gist.github.com/4258605

Fuzzy clustering with fanny()

Thu, 13 Dec 2012 06:30:32 -0500

This is kind of a fun example, and you might find the fuzzy clustering technique useful, as I have, for exploratory data analysis. In this Gist, I use the unparalleled breakfast dataset from the smacof package, derive dissimilarities from breakfast item preference correlations, and use those dissimilarities to cluster foods.

Fuzzy clustering with fanny() is different from k-means and hierarchical clustering, in that it returns probabilities of membership for each observation in each cluster. Here, I ask for three clusters, so I can represent probabilities in RGB color space, and plot text in boxes with the help of this StackOverflow answer.

The colors and the MDS configuration highlight the three primary clusterings of breakfast items into what we’ll call a muffin group, a bread group, and a sweet group. Of course, cluster identification is a subjective exercise, made even more so by use of probabilistic membership, but I’m pretty happy with this breakfast analysis.

https://gist.github.com/4255895

Multidimensional metric unfolding with SMACOF

Wed, 12 Dec 2012 06:30:32 -0500

SMACOF stands for “Scaling by MAjorizing a COmplicated Function,” and it is a multidimensional scaling algorithm for metric unfolding of, among other things, rectangular ratings matrices.

One neat Political Science application of MDS is inferring ideology from survey thermometer ratings. The 2008 ANES featured 43 different thermometer stimuli, and today’s Gist shows how to use SMACOF to simultaneously scale survey respondents and thermometer stimuli in the same space, and to compare this measure of inferred ideology across partisans.

I’ve also got a little piece of code that replaces numeric axis labels with names of the stimuli, which I think might be better, as the numbers don’t really mean much except in comparison with the stimuli. Let me know what you think!

https://gist.github.com/4252470

US State Maps using map_data()

Tue, 11 Dec 2012 06:30:33 -0500

Today’s short post will show how to make a simple map using map_data().

Let’s assume you have data in a CSV file that may look like this:

Notice the lower case state names; they will make merging the data much easier. The variable of interest we’re going to plot is the relative incarceration rates by race (whites and blacks) across each of the fifty states (we’ll remove DC once we load the data). Using the map_data(“state”) command, we can load a data.frame called “all_states”, shown below:

Merging that data with the data frame we have as a CSV produces:

We can then plot each state and shade it by our variable of interest:

Full code is below:

https://gist.github.com/4252133

Can you please post the R code for making that beautiful Advent CalendarR? Pretty please. I've been trying to get it right but no luck :(

Mon, 10 Dec 2012 22:03:58 -0500

I’m glad you like it! The code is a simple loop, drawing 24 open circles, some filled circles, and plotting numbers inside of those. Stripping out all of the axes and labels leaves us with a white field full of dots.

https://gist.github.com/4255340

"Economics-style" graphs with bezier() from Hmisc

Mon, 10 Dec 2012 06:30:27 -0500

So, I really think this one is pretty cool. We spend much of our time in R making graphs with data, but what if you have a theory that you’d like to express graphically? Something like what I’ll call “economics-style” graphs, illustrating, for example, the Solow growth model, a production–possibility frontier, or an indifference curve?

Well, rest assured that R can produce those, too, and it’s made simple by the bezier() function from Hmisc (Hmisc does a lot of other interesting things, but this is what you got in today’s Advent CalendaR slot). Bézier curves are a workhorse of vector graphics, and if you’re not familiar with them, I encourage you to become so, with this beautiful interactive demo and with this more detailed interactive demo.

The Gist shows you how to use Bézier curves to replicate Wikipedia’s Supply-and-Demand graph, and is pretty heavily commented, but I’ll add a few notes:

Generating a Bézier curve with pre-specified x and y vectors takes some trial-and-error. Fortunately, it is usually a fun puzzle and it’s very quick to test. Just think of each point as “pulling” the curve toward itself.
The script defines a hacky little function called approxIntersection(), which is intended to let you input two (x, y) vectors and will output their approximate intersection. This probably doesn’t work well in a lot of cases, and I would be interested in hearing of anyone’s less hacky solutions.
Earlier drafts of this code required a bit of ggplot2 theme-wrangling, but with the release of ggplot2 0.9.3, theme_classic now produces the exact look I was going for.

https://gist.github.com/4229410

Handling missing data with Amelia

Sun, 09 Dec 2012 06:30:38 -0500

So, what if you have data, but some of the observations are missing? Many statistical techniques assume no missingness, so we might want to “fill in” or rectangularize our data, by replacing missing observations with plausible substitutes. There are many ways of going about this, but one of the most robust and accessible is through the Amelia package.

Today’s Gist applies multiple imputation to some sample ANES survey data, and compares listwise-deleted regression results to results pooled from the same regression run on ten imputed data sets. Amelia makes this imputation, modeling, and recombination straightforward, and I’ve thrown in a nice coefficient plot (using position_dodge!) to illustrate the differences between missing data approaches.

https://gist.github.com/4224887

Evaluating term popularity with twitteR

Sat, 08 Dec 2012 06:30:28 -0500

I really wanted to put something together for this series on the twitteR package. Unfortunately, at the moment the number of interesting things than can be done with twitteR, as opposed to through API calls and RCurl, is limited. Regardless, I have Yet Another Invented Application to illustrate a pretty typical use-case for twitteR: grabbing Tweets by search term.

I’ve done this before, for sentiment analysis of Tweets about Republican presidential primary candidates, and indeed, despite its limitations, the searchTwitter() function can be useful. Since the number of Tweets one can grab appears to be limited to 1000, this Gist attempts to infer term popularity by frequency — with only minor success, as you can see in the plot below.

https://gist.github.com/4219950