This week’s release of Census 2010 redistricting data for Delaware, Kansas, Nebraska, North Carolina and Wyoming brought the number of states out so far to 26. Next week, biggies California, Arizona, Ohio and Pennsylvania are among seven states due. So, if you’re looking for national stories, you’ll soon have more than enough of a national data set to mine.

On to this week’s highlights. USA TODAY added stories on each state released in Week 5, and we updated our interactive map and data profile pages. A quick take on our stories:

Delaware: Mike Chalmers of The News Journal in Wilmington wrote that the state’s two smaller southern counties grew much faster than its more-populous northern county. Asians, he wrote, were the state’s fastest growing racial group, up 75.6%. (Also see Chalmers’ lengthier analysis at DelawareOnline.)

Read more » »

Mar 5, 2011 | Census
No Comments »

I had the privilege this week of speaking on two panels at the 2011 Investigative Reporters and Editors Computer-Assisted Reporting* conference in Raleigh, N.C. Here are the slides my co-presenters and I put together:

– “Free Software: From Spreadsheets to GIS” with Jacob Fenton of the Investigative Reporting Workshop. Here is part 1, and here’s part 2.

– “APIs: Making the Web a Data Medium” with Derek Willis of The New York Times.

* Those of us with a few miles on the tires remember that the conference used to go by the name NICAR — for National Institute for Computer-Assisted Reporting. People still call it that.

Feb 26, 2011 | Journalism, News technology
No Comments »

The week was the busiest so far in the rollout of 2010 Census P.L. 94 data, with the bureau releasing data for eight states. That made for intense times for me and my USA TODAY colleagues — we had to process the files while attending the 2011 Investigative Reporters and Editors computer-assisted reporting conference in Raleigh, N.C. (Thanks to IRE for getting us a quiet room to work.)

For our part, we had stories on Alabama, Colorado, Hawaii, Missouri, Nevada, Oregon, Utah and Washington. We also continued to update data-driven profile pages of each state and an interactive map. And we’re spreading the Census love by sharing the data with IRE members.

Other work I noticed, in no particular order:

Read more » »

Feb 26, 2011 | Census
No Comments »

Last week’s Census 2010 redistricting data releases included two of the most populous states — Texas and Illinois — along with Oklahoma and South Dakota. Highlights in stories and apps:

– The Chicago Tribune’s news apps team launched an interactive map and print graphic that show a dramatic increase in the city’s downtown population in the last 10 years, even while many of the surrounding neighborhoods lost population. As a Tribune story explained:

Hardest hit were the South and West sides, where thousands of African-Americans abandoned neighborhoods beset by crime, foreclosures, bad schools and economic squalor.

Read more » »

Feb 21, 2011 | Census
3 Comments »

This week saw the Census Bureau post 2010 redistricting data from five more states — Arkansas, Indiana, Iowa, Maryland and Vermont — bringing the total so far to nine. As time allows, and because I’ve spent the last two months prepping code for this, I’m chronicling stories and graphics that catch my eye. This week, by state:

Arkansas: My colleague Rick Jervis’ story noted the growth in northwestern Arkansas fueled by employers Wal-Mart and Tysons Foods. This will lead to fairly substantial redistricting:

… Arkansas’ representatives soon will answer to whole new neighborhoods of voters, says Thomas Paradise, a University of Arkansas professor of geosciences. “It’s not hard on the population. It’s going to be hard for the congressmen,” Paradise says. “They’re going to have a radically different constituent.”

– The Arkansas Democrat-Gazette also noted the shift northwest and reports another mayor planning to challenge the count with his city losing 6,000 people. Both stories available to subscribers only.

Indiana: The Indianapolis Star’s Tim Evans (writing for USAT) wrote that the state became more Hispanic and suburban in the last 10 years. Still:

“While Indiana’s racial and ethnic makeup has shifted, the state remains less diverse than the nation,” says Matt Kinghorn, a demographer with the Indiana Business Research Center at Indiana University’s Kelley School of Business. “Compared to the most recent population estimates for the nation, the share of Indiana’s population that is white is well above the U.S. mark of 79.6%.”

– The Star’s graphics team also launched an interactive map that loads data from other states (zoom out to see).

Iowa: The Des Moines Register highlighted the state’s population shift from rural to urban:

Iowa State University economist Liesel Eathington said those population trends reflect a pattern that’s become common throughout the Midwest. One factor is that mechanized agriculture requires increasingly fewer farmers to till ever-larger tracts of land.

(By the way, check out the modal pop-up graphics on that page. Great work, but the one that shows all 99 counties makes me glad I don’t have to memorize their names for a geography bee — I guess the folks who divided up the state’s geography liked things uniform.)

– The Register also launched an interactive map with a bonus: population histories for each county. Really shows the dramatic rise of Dallas County, west of Des Moines.

Read more » »

Feb 12, 2011 | Census
No Comments »

Four states received Census 2010 P.L. 94 redistricting data last Thursday, and just like that Census reporting season was off and running. Our newsroom quickly tackled stories on trends in Louisiana, Mississippi, New Jersey and Virginia and posted an interactive state/county map on our Census page that we’ll keep updating.

I thought it would be fun, at least as time allows, to chronicle here some of the Census work from other newsrooms that catches my eye. So, here are a few stories and visualizations from week 1:

Stories

Given the geography-specific nature of the data, most stories focused on states or cities. Several examined post-Katrina New Orleans, using Census data to measure the effect of the hurricane on demographics:

The Washington Post pointed to the political effect of the city losing 29% of its population between 2000 and 2010:

The city will also probably lose a voice in Washington: Louisiana will end up with six congressional seats instead of seven because of the lost population, and state legislators are expected to eliminate one of the city’s three congressional districts.

The Times-Picayune reported that the broader New Orleans’ regional population drop was less, down 11% in the seven-parish metro area. Its story, as did others, pointed out that New Orleans after the hurricane was more white and Hispanic than before:

Black residents comprised 60 percent of city residents last year, compared with 67 percent in 2000, the data show. Meanwhile, the proportion of white residents grew from 28 percent to 33 percent. The city’s proportion of Hispanic residents, who can be black or white, inched upward, from 3 percent in 2000 to 5.2 percent last year.

Elsewhere in Louisiana, The News-Star in Monroe said the mayor is planning to challenge Census data because his city’s population drop below 50,000 could mean a loss of federal funding:

“We are not going to panic,” [Jamie] Mayo said. “We are going to see what the process is, and we will pursue it. We are not pleased about going below 50,000. Our whole objective is to grow our city. To me, the ideal size would be 60,000 to 65,000.”

Also from my Gannett colleagues: In Mississippi, The Clarion-Ledger wrote about accelerated white flight from Jackson. In Virginia, The News-Leader in Staunton covered the aging population in the central Shenandoah Valley. In New Jersey, the state bureau covered how the population shift to the south could pose an issue for Democrats, and The Daily Record in Morristown covered a rise of vacant homes in the state to near 10%.

Finally, stepping back, The New York Times wrote about a continuation of the long-term trend toward an increasingly diverse America. Examining the data in the first four states, it noted a sharp drop in white youths and called it “a shift that demographers say creates a culture gap with far-reaching political and social consequences.”

Visualizations

I know how hard this work is, so hats off to all the developers and data journalists out there working on interactives.

– Our Juan Thomassie did a really nice front end to the Census storehouse that Paul Overberg and I built.

– The Washington Post’s interactive map of Virginia lets readers zoom down to the block group level for population growth and race/ethnicity counts.

– The New York Times has a detailed map of population changes in New Orleans.

I’m sure that’s just the tip of the iceberg, but the Super Bowl is calling. Add more in the comments. And remember: up to five states coming out this week — Arkansas, Indiana, Iowa, Maryland and Vermont.

Feb 6, 2011 | Census
No Comments »

In a few weeks, the first detailed results of the decennial U.S. Census will start pouring from Census headquarters in Suitland, Md., and a Panic Season will commence in unsuspecting newsrooms. What are these numbers? Where can I download them? Didn’t we just get new Census data? Can you tell me whether Census counts X or Y or Z?

On deadline, that’s a lot of potential headache. I know you want to avoid the pain, so take some advice from a guy who survived reporting on Census 2000: prep is everything.

Here are five steps you can take now:

1. Know your Census products: These days, “Census data” means more than it did a decade ago. The advent of the American Community Survey — a survey of about 3 million households each year that replaced the old Census long form — means we get annual estimates in between the full decennial counts. And the ACS comes in three flavors: single-year data plus three- and five-year aggregates, each providing different levels of geographic granularity.

The regular releases of ACS data make Census seem more routine these days, but the data coming out soon are different. These aren’t estimates from a sample — they’re the complete counts taken in spring 2010 via a short questionnaire sent to every household in America.

This first wave of Census 2010 data, coming state-by-state in February and March, are the Redistricting Data (P.L. 94-171) summary files. They’ll contain the basic counts of population by race for every state, county and place in America, all the way down to the smallest geographies, called blocks. As its name implies, these data will be used to redraw the boundaries of legislative, electoral and other districts in states — a process journalists will want to keep tabs on.

Later, in the summer, Summary File 1 will offer more detailed data on age, sex, households, families, and housing units — again from complete counts. Then, in the fall, we’ll see the next release of ACS data. Got all that?

Read more » »

Jan 19, 2011 | Census
3 Comments »

Update Jan. 17, 2012: The top selling books of 2011 are listed here, and in that table you can view lists back to 2007. The post below refers to the 2010 top-selling titles.

Original post:

Stieg Larsson’s Millennium Trilogy grabbed the top three spots in USA TODAY’s annual list of top-selling books, reflecting a broader move by readers toward fiction this year. The list of 2010′s best-sellers was part of a package published today wrapping up the year’s trends as reflected in USA TODAY’s Best-Selling Books list.

Shepherding the books list is one of those tasks I spend considerable time on and, over the years, it has become one of my favorite opportunities for data journalism. The well never seems to run dry on ideas, and with 17 years of data in our archives, there are plenty of opportunities to see how annual moves on the list stack up against long-term trends. Along with Larsson’s success, our book team’s report on trends highlighted titles and authors reaching No. 1, from Nicholas Sparks to George W. Bush.

Much of our book list data is open for developers. Check out the API for details.

Jan 13, 2011 | Analysis
No Comments »

The web-o-sphere this week brought forth a collection of opinions on the value of data journalism and the skills that go with it. To wit:

• Tim Berners-Lee, he who invented the World Wide Web, told the Guardian that “journalists need to be data-savvy” and that “data-driven journalism is the future.” The story then goes on to question whether data analysis could ever replace traditional reporting.
• The blog 10,000 Words declared that one of the “5 Myths about digital journalism” is that “journalists must have database development skills” and suggested that most journalists should leave high-level hacking to the experts.
• Another site, FleetStreetBlues, opined that “amidst all this hype, earnestness and spreadsheet-geekery, here’s the truth about so-called ‘data journalism’. It’s still about the story, stupid.”

There’s been a bunch of reaction to these posts, including a few people pointing out a 1986 Time story that sounds similar to the one this week from the Guardian. And therein lies the problem with all three pieces: None of them benefits from a big-picture, historical perspective on data journalism — not where it came from, not how it’s changed and especially not the massive amount of ground the label covers these days.

We used to call it CAR

Back  when software came on 5.25-inch floppy disks, or maybe before then, the idea of using a PC to “crunch numbers” was christened “computer-assisted reporting.” These days, we call it data journalism because, along the way, it became obvious the old name was anachronistic. As Phil Meyer once said, we don’t talk about telephone-assisted reporting, do we?

When I got into the game — when Paradox was the desktop database manager of choice — our newsroom had a personal computer designated as the “CAR station.” While others worked on dumb terminals connected to a mainframe, I was surfing the web with Netscape and ringing up Paul Overberg for advice on Census data. I was the newsroom data expert — the guy reporters called when they had a spreadsheet on a disk or an idea to get data from city hall.

In that era — with database-driven web startups like Amazon.com spreading cultural revolution — it was easy to foresee a time when reporters wouldn’t just get the occasional spreadsheet but find themselves inundated with data. Thus was born (at least in my sphere) the drive to evangelize CAR in the newsroom. We taught Excel, we sent people to IRE boot camps, we set up presentations showing the kinds of stories journalists were landing with these skills. The message of CAR was about finding stories and using simple tools to do it: spreadsheets, databases, maps, stats.

Now we call it hacking

Soon enough, though, the craft began to change and so did the talk at IRE CAR conferences — especially in the hands-on classes and demos. In Philadelphia in 2002, the hands-on classes mostly covered Access, Excel, SPSS and, for the adventurous, SQL Server. Just a few years later, in Cleveland and Houston, the offerings included sessions on web scraping, Perl, Python, MySQL and Django.

Read more » »

Nov 26, 2010 | Journalism
17 Comments »

WordPress is a great CMS, but if you’re hosting your own install you’ll want to pay attention to keeping its performance nimble. Here are a few tips I picked up this week after reading this post and thinking about how to apply those ideas to my own site:

1. Optimize the database.

WordPress handily saves every revision of every page and post you create. That’s great for retrieving a lost paragraph, but it can add hundreds of rows to the wp_posts table — especially if, like me, you save revisions often. Eventually, the database grows big enough to affect query performance, which can slow page loads.

There’s a remedy: Delete those unnecessary rows and optimize your tables.

The WordPress plugin directory lists several database optimizers, but they make me nervous. They all claim to optimize but don’t say exactly what changes they make. Instead, I dug straight into my site’s MySQL database using my web host’s phpMyAdmin interface.

Sanity check: Back up your database first.

Once that’s done, run this in the SQL pane:

DELETE FROM wp_posts WHERE post_type = "revision";

Next, use phpMyAdmin to optimize your tables (Structure Pane > Select All > choose Optimize Table from the drop down).

This trick shrank my database from 5.5MB to less than 1MB.

2. Install a caching plugin.

If something I write here goes viral, I’d like to avoid watching my server fry. One way to head this off is to set up caching so that every page load isn’t executing a query on your server.

Again, there is no shortage of caching plug-ins for WordPress. After checking comments on my web host’s user forums, I went with W3 Total Cache. Not only does it provide caching for pages, feeds and other content, it will minify CSS, HTML and JavaScript.

It loads like any WordPress plugin, but it has many options. This tutorial gives great step-by-step directions.

3. Remove unneeded PHP calls

Most WordPress themes, including the one I built for my site, include calls designed to fetch info from the MySQL database. The idea is to make the theme easily customizable via the admin and make the theme portable to millions of users.

But unless you plan to change your site’s name, tagline or url often, there’s no reason to have your theme fetch those every time it loads your header.

For example, in my header.php file, I changed this:

<h1><a href="<?php bloginfo('url'); ?>"><?php bloginfo('name'); ?></a></h1>

To this:

<h1><a href="http://www.anthonydebarros.com">Anthony DeBarros</a></h1>

That’s two fewer calls each time that page loads. Depending on your theme, you might be able to find many more calls to eliminate.

More reading:

There’s a lot more you can tweak under the hood, and that means plenty of chances to learn more goodness about server configurations. Here are some of the places where I picked up these tricks and where you can find a lot more detail. Happy optimizing!

http://yoast.com/speed-up-wordpress/
http://www.ardamis.com/2010/08/03/how-to-speed-up-wordpress/
http://wpgarage.com/tips/38-ways-to-optimize-and-speed-up-your-wordpress-blog/
http://www.gadgetcage.com/5-best-wordpress-plugins-to-improve-the-loading-speed-of-a-blog/8038/
http://mashable.com/2010/07/19/speed-up-wordpress/

Nov 5, 2010 | Programming, WordPress
No Comments »