Since then, I’ve made a few tweaks, but one thing I hadn’t done all year was post the theme — which I call Goshen — for anyone to use. Today I fixed that and pushed the files up to their own repository on Github. You can download the files and hack away. (In your WordPress install, under /wp-content/themes/, create a folder called Goshen and unzip the files there; then you can activate the theme via the dashboard.)

I’ll continue to tweak when I have time. I can’t say enough about how much WordPress theme hacking has taught me about HTML, CSS, templates and web design. If you want to start from scratch, I recommend this excellent tutorial. You’ll discover that WordPress themes have only a few moving parts. Mastering them will let you make your site exactly what you want it to be.

So I did, and I’m sharing. You can grab the code for the CDC-flu-scraper on Github.

The code uses the Mechanize and BeautifulSoup modules for web browsing and html parsing, respectively. Much of what I demonstrate here I started learning via Ben Welsh’s fine tutorial on web scraping.

We’re still early in flu season, but if you watch this data each week you’ll see the activity pick up quickly.

Update 10/22/2011: Ben Welsh has lent some contributions to this scraper, adding JSON output and turning it into a function. Benefits of social coding 101 …

Many newsroom types spend the day in Windows without the option of Ubuntu or another Linux OS. I’ve been planning some training around Python soon, so I compiled this quick setup guide as a reference. I hope you find it helpful.

Set up Python

Get started:

1. Visit the official Python download page and grab the Windows installer. Choose the 32-bit or 64-bit version, depending on your version of Windows 7 (right-click the Computer icon on your desktop and select Properties to find out which one you have). Note: Python currently exists in two versions, the older 2.x series and newer 3.x series (for a discussion of the differences, see this). This tutorial focuses on the 2.x series.

2. Run the installer and accept all the default settings, including the “C:\Python27″ directory it creates.

3. Next, set the system’s PATH variable to include directories that include Python components and packages we’ll add later. To do this:

• Right-click Computer and select Properties.
• In the dialog box, select Advanced  System Settings.
• In the next dialog, select Environment Variables.
• In the User Variables section, edit the PATH statement to include this:

C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;

4. Now, you can open a command prompt (Start Menu|Accessories or Start Menu|Run|cmd) and type:

C:\> python

That will load the Python interpreter:

Python 2.7.2  (default, Jun 12 2011, 14:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or license for more information.
>>>

Because of the settings you included in your PATH variable, you can now run this interpreter — and, more important, a script — from any directory on your system.

Press Control-Z to exit the interpreter and get back to a C: prompt.

Set up useful Python packages

1. setuptools offers the helpful easy_install utility for installing Python packages. Grab the appropriate version for your system and install.

2. pip is another package installer that improves on setuptools. Having pip and setuptools will cover most of your installation needs, so go ahead and add pip. Now that you’ve installed setuptools, you can add pip by typing this at any command prompt:

easy_install pip

Notice that easy_install executes without needing to be told where on the system it’s located. That’s the benefit of adjusting your PATH variable earlier.

3. Mechanize and BeautifulSoup are must-have utilities for web scraping, and we’ll add those next:

pip install mechanize
pip install BeautifulSoup==3.2

4. csvkit, which I recently covered here, is a great tool for dealing with comma-delimited text files. Add it:

pip install csvkit

You’re now set to get started using and learning Python under Windows 7. If you’re looking for a handy guide, start with the Official Python tutorial.

csvkit is a Python-based Swiss Army knife of utilities for dealing with, as its documentation says, “the king of tabular file formats.” It lets you examine, fix, slice, transform and otherwise master text-based data files (and not only the comma-delimited variety, as its name implies, but tab-delimited and fixed-width as well). Christopher Groskopf, lead developer on the Knight News Challenge-winning Panda project and recently a member of the Chicago Tribune’s news apps team, is the primary coder and architect, but the code’s hosted on Github and has a growing list of contributors.

As of version 0.3.0, csvkit comprises 11 utilities. The documentation describes them well, so rather than rehash it, here are highlights of three of the utilities I found interesting during a recent test drive:

csvcut: Henceforth, this utility will likely meet every csv file I get. To start, it will describe the file contents for me: If I want a quick scan of the column names and their order in the file, I just type:
 

csvcut -n filename

The output is an indexed list of column names, assuming the first row of the file is a header row. If not, you get the first row of data, which can be handy as well.

Still, the “cut” part of this utility is its killer feature, extracting columns from the file in the order you choose. You might use this to subset the file before importing to a database or to quickly reorder columns before embarking on analysis.

To extract the seventh, first and second columns from the file, in that order, it’s as simple as:
 

csvcut -c 7,1,2 filename

Really nice.

csvsql: Send in a csv, and it returns a CREATE TABLE statement for your SQL database. The first time I ran this and saw the result, I did a double-take of pure joy. Then I got slightly depressed thinking about times I wrote code to import 256-column csv files into SQL Server. No more. You just type a statement like this:
 

csvsql -i postgresql filename

That produces a CREATE TABLE statement with syntax appropriate to PostgreSQL. Plenty of SQL flavors are available too — the utility uses SQLAlchemy’s dialect collection to offer syntax options for SQL Server, MySQL, Oracle and others.

Another killer feature: You can add an "inserts" argument to have csvsql generate a SQL INSERT statement for each row of the CSV. Having been flummoxed by SQL Server’s import wizard more than once, I can tell you that inserting data by row is a great alternative, especially if you’re trying to isolate a problem row.

csvstat: Returns basic descriptive statistics for each column in the file. Results include overall row count, the data type for each column, and descriptives including min, max, sum, median, most frequent values, etc. Very handy for a quick read on what you have in the file.

Those three jumped out at me, but there are more. Other utilities will convert files to csv, output a csv as JSON, or merge, clean and stack files. The fact you can pipe output from one utility to another creates a powerful scenario.

It’s great work and an example of the kinds of tools journalists can build to deal with common problems we face. I’ll be watching this develop with great anticipation.

Speaking of those, here’s what I’ve been using to make life easier:

Akismet: Gets rid of a ton of comment spam for various Russian “services” so I can spend my time doing other things. You’ll need to sign up for an API key, but otherwise it’s simple and effective.

Contact Form 7: After trying a few contact plugins, I settled on Contact Form 7 and have had great results. It powers my Contacts page, which I prefer to use instead of posting an email address. For spam filtering, I implemented the quiz feature, but the plugin also supports CAPTCHA. I rarely get spam.

Google XML Sitemaps: Generates a sitemap.xml file that Google and other search engines use to index the site. Lets me include or exclude content and control how often to update the file.

Jetpack: Another plugin I’ve yet to fully tap. The site stats alone (formerly provided by WP Stats) are a must-have, and I’m also using the function that shows pop-up profiles if you hover over a Gravatar. But this Swiss Army Knife of a plugin has shortcodes for embedding media, a spelling/style checker and more.

Twitter for WordPress: Easy way to call and display my tweets in the sidebar.

W3 Total Cache: This plugin provides options to cache my pages and “minify” code so the site can load faster. I’ve seen a considerable boost in speed since using it, although configuration is tricky. (See my post on the topic for details.)

 WP-Optimize: Cleans rows I don’t need from the MySQL database running the site, including post revisions. If you’re like me and edit yourself a lot, those can add up. Yet another way to keep the motor running tight.

WPtouch Pro: Automatically serves mobile-ready versions of this site optimized for small screen. There’s a free version that’s just fine; I opted for the $39 version that includes extra iPad support.

That’s the collection. Any recommendations? Share them below.

1. Optimize the database.

WordPress handily saves every revision of every page and post you create. That’s great for retrieving a lost paragraph, but it can add hundreds of rows to the wp_posts table — especially if, like me, you save revisions often. Eventually, the database grows big enough to affect query performance, which can slow page loads.

There’s a remedy: Delete those unnecessary rows and optimize your tables.

The WordPress plugin directory lists several database optimizers, but they make me nervous. They all claim to optimize but don’t say exactly what changes they make. Instead, I dug straight into my site’s MySQL database using my web host’s phpMyAdmin interface.

Sanity check: Back up your database first.

Once that’s done, run this in the SQL pane:

DELETE FROM wp_posts WHERE post_type = "revision";

Next, use phpMyAdmin to optimize your tables (Structure Pane > Select All > choose Optimize Table from the drop down).

This trick shrank my database from 5.5MB to less than 1MB.

2. Install a caching plugin.

If something I write here goes viral, I’d like to avoid watching my server fry. One way to head this off is to set up caching so that every page load isn’t executing a query on your server.

Again, there is no shortage of caching plug-ins for WordPress. After checking comments on my web host’s user forums, I went with W3 Total Cache. Not only does it provide caching for pages, feeds and other content, it will minify CSS, HTML and JavaScript.

It loads like any WordPress plugin, but it has many options. This tutorial gives great step-by-step directions.

3. Remove unneeded PHP calls

Most WordPress themes, including the one I built for my site, include calls designed to fetch info from the MySQL database. The idea is to make the theme easily customizable via the admin and make the theme portable to millions of users.

But unless you plan to change your site’s name, tagline or url often, there’s no reason to have your theme fetch those every time it loads your header.

For example, in my header.php file, I changed this:

<h1><a href="<?php bloginfo('url'); ?>"><?php bloginfo('name'); ?></a></h1>

To this:

<h1><a href="http://www.anthonydebarros.com">Anthony DeBarros</a></h1>

That’s two fewer calls each time that page loads. Depending on your theme, you might be able to find many more calls to eliminate.

More reading:

There’s a lot more you can tweak under the hood, and that means plenty of chances to learn more goodness about server configurations. Here are some of the places where I picked up these tricks and where you can find a lot more detail. Happy optimizing!

http://yoast.com/speed-up-wordpress/
http://www.ardamis.com/2010/08/03/how-to-speed-up-wordpress/
http://wpgarage.com/tips/38-ways-to-optimize-and-speed-up-your-wordpress-blog/
http://www.gadgetcage.com/5-best-wordpress-plugins-to-improve-the-loading-speed-of-a-blog/8038/
http://mashable.com/2010/07/19/speed-up-wordpress/

Announced today, the first two out of the gate offer access to our Best-Selling Books list and our archive of pro sports player salaries. Both data sets represent many years of effort by the database team at USA TODAY. To build these APIs, we were fortunate to team up with some great folks in our IT group. It’s been a great learning experience. We’re proud of the data and excited about opening it to the public.

Our new developer site has the details. You can register for API keys, read the documentation and post questions in forums. We want your feedback — it will help us make the APIs better and steer us as we make more of data available in the months ahead. Also be sure to read the terms of use.

About our first two APIs:

The Best-Selling Books API contains the weekly top 150 books from our list, including metadata about each title.

The Sports Salaries API contains annual salary information for players in Major League Baseball, the National Football League, National Basketball Association, and National Hockey League.

For more information, check out our official press release. There’s also a page up at programmableweb.com.

A quick Google search turned up just the tutorial I needed from the design blog Hongkiat.com:

How to Separate Comments and Trackbacks [Wordpress Tips].

I found a ton of solutions (some from people with no clue about the difference between median and average), but the one below — adapted from a post by Adam Machanic at sqlblog.com — was the best. It produces accurate results and is fairly speedy to boot.

Here’s an example. Consider this table with student grades from two courses:
.

We’d like to find the median grade in each class. Here’s the script:

 

 
DECLARE @tmp TABLE (
   ID INT IDENTITY(1,1),
   Class VARCHAR(50),
   FirstName VARCHAR(50),
   Grade DECIMAL(5,1)
)
 
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Math', 'Bob', 65)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Math', 'Joe', 72)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Math', 'Sally', 95)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Science', 'Bob', 65)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Science', 'Joe', 81)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Science', 'Sally', 81)
INSERT INTO @tmp (Class, FirstName, Grade)
   VALUES ('Science', 'Mike', 72)</code>
 
SELECT
x.Class,
CAST(AVG(x.Grade) AS DECIMAL(6,1)) AS 'Median'
FROM
   (
     SELECT
     Class,
     FirstName,
     Grade,
     ROW_NUMBER() OVER (
     PARTITION BY Class
     ORDER BY Grade ASC, ID ASC) AS RowAsc,
     ROW_NUMBER() OVER (
     PARTITION BY Class
     ORDER BY Grade DESC, ID DESC) AS RowDesc
     FROM @tmp
   ) AS x
WHERE
RowAsc IN (RowDesc, RowDesc - 1, RowDesc + 1)
GROUP BY x.Class
ORDER BY x.Class

That’s a lot. Let’s break down what’s happening.

First, to set up the data, we’re creating a temporary table and inserting seven rows.

Then comes the main SELECT, which includes a subquery that’s the meat of the goodness behind this. The subquery uses the ROW_NUMBER function to create ascending and descending row identifiers based on the ordering of the grades. It’s easier to visualize if you see what the subquery’s creating:
.

The RowAsc and RowDesc fields reflect the ordered row numbers of the grades per each class (i.e., partition). Given those, it’s easy to find the median. For the math class, with an odd number of students, the median occurs where RowAsc equals RowDesc. For the science class, with an even number of students, the median is the average of the two grades where RowAsc and RowDesc are within one of each other.

(Whenever you’re looking for the median in an ordered list that has an even number of values, averaging the two middle numbers gives the answer.)

That’s what our main query does when it pulls from the subquery result set. It looks for:
 

RowAsc IN (RowDesc, RowDesc - 1, RowDesc + 1)

and then averages what it finds. The answer is 72 for the math class and 76.5 for science.

This solution works with T-SQL and Microsoft SQL Server. With minor tweaks, it also runs fine in PostgreSQL 8.4, which also implements the ROW_NUMBER function. MySQL does not support that function, so you’ll have to search for another option.

Good stuff and pretty handy for extracting large numbers of median values without having to resort to a stats program.

Starting today, I’m using a new theme I’ve crafted, which I am calling Goshen. It’s designed to be minimal but readable. The theme I was using previously, Porfolio_AD, is going to rest comfortably on a beach somewhere and probably won’t be coming out of retirement any time soon. You can still download it — I know of one person who actually did implement it on a site, much to my shock — and play to your heart’s content.

Original post:

I make no claim to having any design sense, but if you like the WordPress theme I made for my site, you’re welcome to it. Grab it right here.

Portfolio AD is a one-column theme with a two-column sidebar. Both columns in the sidebar can display widgets. The header is a modification of the well-worn Hemingway theme, and the rest is my take-off of a site built via this tutorial from WPDesigner.com. It works fine with WordPress 3.0.

What you see here is the result of constant tinkering. I coded up a theme of my own to learn more about what WordPress can do, and in the process I have learned tons about CSS, HTML, web hosting and content management systems in general. If you have the time, it’s well worth the investment.

Use the theme as a starting point for your own tinkering and have fun!