Many newsroom types spend the day in Windows without the option of Ubuntu or another Linux OS. I’ve been planning some training around Python soon, so I compiled this quick setup guide as a reference. I hope you find it helpful.

Set up Python

Get started:

1. Visit the official Python download page and grab the Windows installer. Choose the 32-bit or 64-bit version, depending on your version of Windows 7 (right-click the Computer icon on your desktop and select Properties to find out which one you have). Note: Python currently exists in two versions, the older 2.x series and newer 3.x series (for a discussion of the differences, see this). This tutorial focuses on the 2.x series.

2. Run the installer and accept all the default settings, including the “C:\Python27″ directory it creates.

3. Next, set the system’s PATH variable to include directories that include Python components and packages we’ll add later. To do this:

• Right-click Computer and select Properties.
• In the dialog box, select Advanced  System Settings.
• In the next dialog, select Environment Variables.
• In the User Variables section, edit the PATH statement to include this:

C:\Python27;C:\Python27\Lib\site-packages\;C:\Python27\Scripts\;

4. Now, you can open a command prompt (Start Menu|Accessories or Start Menu|Run|cmd) and type:

C:\> python

That will load the Python interpreter:

Python 2.7.2  (default, Jun 12 2011, 14:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or license for more information.
>>>

Because of the settings you included in your PATH variable, you can now run this interpreter — and, more important, a script — from any directory on your system.

Press Control-Z to exit the interpreter and get back to a C: prompt.

Set up useful Python packages

1. setuptools offers the helpful easy_install utility for installing Python packages. Grab the appropriate version for your system and install.

2. pip is another package installer that improves on setuptools. Having pip and setuptools will cover most of your installation needs, so go ahead and add pip. Now that you’ve installed setuptools, you can add pip by typing this at any command prompt:

easy_install pip

Notice that easy_install executes without needing to be told where on the system it’s located. That’s the benefit of adjusting your PATH variable earlier.

3. Mechanize and BeautifulSoup are must-have utilities for web scraping, and we’ll add those next:

pip install mechanize
pip install BeautifulSoup==3.2

4. csvkit, which I recently covered here, is a great tool for dealing with comma-delimited text files. Add it:

pip install csvkit

You’re now set to get started using and learning Python under Windows 7. If you’re looking for a handy guide, start with the Official Python tutorial.

csvkit is a Python-based Swiss Army knife of utilities for dealing with, as its documentation says, “the king of tabular file formats.” It lets you examine, fix, slice, transform and otherwise master text-based data files (and not only the comma-delimited variety, as its name implies, but tab-delimited and fixed-width as well). Christopher Groskopf, lead developer on the Knight News Challenge-winning Panda project and recently a member of the Chicago Tribune’s news apps team, is the primary coder and architect, but the code’s hosted on Github and has a growing list of contributors.

As of version 0.3.0, csvkit comprises 11 utilities. The documentation describes them well, so rather than rehash it, here are highlights of three of the utilities I found interesting during a recent test drive:

csvcut: Henceforth, this utility will likely meet every csv file I get. To start, it will describe the file contents for me: If I want a quick scan of the column names and their order in the file, I just type:
 

csvcut -n filename

The output is an indexed list of column names, assuming the first row of the file is a header row. If not, you get the first row of data, which can be handy as well.

Still, the “cut” part of this utility is its killer feature, extracting columns from the file in the order you choose. You might use this to subset the file before importing to a database or to quickly reorder columns before embarking on analysis.

To extract the seventh, first and second columns from the file, in that order, it’s as simple as:
 

csvcut -c 7,1,2 filename

Really nice.

csvsql: Send in a csv, and it returns a CREATE TABLE statement for your SQL database. The first time I ran this and saw the result, I did a double-take of pure joy. Then I got slightly depressed thinking about times I wrote code to import 256-column csv files into SQL Server. No more. You just type a statement like this:
 

csvsql -i postgresql filename

That produces a CREATE TABLE statement with syntax appropriate to PostgreSQL. Plenty of SQL flavors are available too — the utility uses SQLAlchemy’s dialect collection to offer syntax options for SQL Server, MySQL, Oracle and others.

Another killer feature: You can add an "inserts" argument to have csvsql generate a SQL INSERT statement for each row of the CSV. Having been flummoxed by SQL Server’s import wizard more than once, I can tell you that inserting data by row is a great alternative, especially if you’re trying to isolate a problem row.

csvstat: Returns basic descriptive statistics for each column in the file. Results include overall row count, the data type for each column, and descriptives including min, max, sum, median, most frequent values, etc. Very handy for a quick read on what you have in the file.

Those three jumped out at me, but there are more. Other utilities will convert files to csv, output a csv as JSON, or merge, clean and stack files. The fact you can pipe output from one utility to another creates a powerful scenario.

It’s great work and an example of the kinds of tools journalists can build to deal with common problems we face. I’ll be watching this develop with great anticipation.

– “Free Software: From Spreadsheets to GIS” with Jacob Fenton of the Investigative Reporting Workshop. Here is part 1, and here’s part 2.

– “APIs: Making the Web a Data Medium” with Derek Willis of The New York Times.

* Those of us with a few miles on the tires remember that the conference used to go by the name NICAR — for National Institute for Computer-Assisted Reporting. People still call it that.

——–

Data journalists spend lots of time wrestling dirty data, so when I heard the News Applications team at the Chicago Tribune raving about the data-handling abilities of Freebase Gridworks, my interest was piqued. Anything that can lessen the pain of cleaning data is worth a closer look!

Freebase Gridworks is a Java-based app that runs locally in your web browser. The makers’ pitch describes it best:

… A power tool that allows you to load data, understand it, clean it up, reconcile it internally, augment it with data coming from Freebase, and optionally contribute your data to Freebase for others to use. All in the comfort and privacy of your own computer.

Installation is simple. I chose to load Gridworks on my Windows XP-based work laptop, although you can download Mac and Linux versions from the code page. I was up and running in about five minutes, which included loading a new version of Java. Once running, the opening screen looks like so (click for larger version):

You can open an existing project or create a new one by importing a data file — and Gridworks hints at its utility by providing options to parse delimited or non-delimited files, limit the import to specific rows, etc. For testing, I grabbed the Academic Libraries: 2008 Public Use Data file from the National Center for Education Statistics — a tab-delimited text file of about 4,100 rows.

Import was a cinch. Gridworks guessed correctly at the file format and split the columns perfectly:

First thing I tried was data cleanup. Some of the cities in the “CITY_M” field were in uppercase and some were capitalized normally. Each column header has a menu of manipulation options, so I chose Edit Cells > Common Transforms > To Titlecase:

Gridworks chugged along for a few seconds (a progress bar might be handy), but soon enough it returned all the cities in the correct case. Nice!

Next, the ZIP_M field (and ZIP also) had a mix of five-digit zips and some with the “plus 4″ extension. To separate the plus 4′s into their own field, I chose Edit Column > Split Into Several Columns. It produced this dialog:

I opted to split the column by field length and typed the values “5,4″ for the string lengths. To preserve the leading zeros in the zips and extensions, I unchecked the box “guess cell type” to keep the fields as text. Gridworks chugged along again, then produced the result, automatically renaming the fields in the process:

Another handy feature of Gridworks is its ability to edit field values en masse. If you hover your mouse over a cell, an “edit” button appears:

Clicking it brings up a dialog box where you can change the cell’s value — and apply that change to all other cells with the same content. Handy! Here’s how you could change all the state names of “AL” to “Alabama”:

Data cleanup is clearly a strength, but Gridworks also offers plenty of ways to explore data by creating facets, or summaries of data (think using COUNT and GROUP BY in SQL). It produces summary tables that let you quickly find all the unique values in a column — and edit them if you need to create consistency (i.e. company names spelled several ways).

Finally, Gridworks lets you export your revised data back to Excel or tab/comma-delimited text files, among other options. Very, very useful.

Judging by its revision history, Freebase Gridworks is very much an evolving tool but one worth keeping tabs on. This little test drive has probably just scratched the surface of the ways you can use it to standardize your data, but you can get more ideas via the demo videos on the product’s home page.

Now add me to the list. Michelle interviewed me right after one of this winter’s east coast blizzards, and my cabin fever shows in the sheer verbosity of my responses. But it was fun reliving my early days — when I discovered the power of merging data and reporting. Here’s one quote:

A reporter in the newsroom came to me and said, “Hey, it would be really good if we could figure out what the most valuable properties are in the city of Poughkeepsie. And I thought to myself, “You know, this might be a good opportunity for me to go and make friends with the IT guy over in City Hall.” I went over and visited him, he was down in the basement of City Hall, in the computer room. Back in those days, they all had big mainframe computers in an air-conditioned room.

Actually, what I first did was I went to the tax assessor’s office, and I said, “I want a list of all the properties in the city of Poughkeepsie and how much they’ve been assessed for.” And they pointed me over to the corner where there were these big books filled with computer printouts, and they said, “Well, all the numbers are there, and you can just start copying them down.” And I thought to myself, “If they were printed on this piece of paper that looks like computer paper, then certainly they are in a computer somewhere in this building. And I can get that data on a disk that I can bring over and put into my computer.” And that’s how I really started figuring out that we can do computer-assisted reporting by going to the government and getting data.

That’s what I did. I went to visit that guy in City Hall, and I said, “Look, I know you’ve got a file on your computer. I’d love to have you put it on this floppy disk for me.” And he had to check with the local attorneys, and get their permission, and I called up a sunshine advocate in New York state and got him to weigh in, and they agreed that, “Yeah, the law says we can do this.” The next thing I know, I had that data on the computer and was going through it in Paradox. We wound up writing a couple of stories about different properties.

A hat tip to Michelle for a smart way to gain insight into our slice of journalism.

Why? Because most music recorded in the ’80s — for all its supposed ambition and technical innovation — sounds more dated, more processed and more fake today than the music of the ’60s and ’70s, including disco. Line up Abbey Road or Dark Side of the Moon next to anything by Duran Duran or Human League and the point is made.

What hurt ’80s music most was the rush to digital sounds. Musicians grabbed every gizmo they could find — synthesizers, drum machines, vocal effects, digital guitar processors — and abandoned their lovely analog gear. When Phil Collins’ engineer figured out how to use a noise gate to make his drums sound as big as a 747, everyone copied. Songs now revolved not around good lyrics or melodies but the sounds of these machines. It all had a big wow factor, but it lacked one important quality:

None of it was timeless.

Oh, people thought it was. That’s what it feels like in the midst of every movement. “This will last forever.” Well …

In 1991, the music of the 1980s officially died. That’s when Nirvana released Nevermind and Pearl Jam exploded with Ten, both featuring a sound that was entirely a return to all that the ’80s had abandoned — authentic instruments without a lot of gimmicks. Video may have killed the radio star, but it couldn’t kill what was timeless.

As a journalist who loves technology, I wonder whether we’ll look back in 20 years and have a similar take on this first decade of the 2000′s. The Twittersphere is filled daily with reports of new apps, new sites, new data visualizations. People Tweet every other sentence from conferences where digital gurus explain where the news business is heading, maybe. Having gorged on print profits far too long, we news types are running towards all things digital hoping for a cure for our indigestion.

A lot of it is interesting, some certainly carries the wow-factor, and some of it is going to be truly useful. But how much of it will last? How much is timeless? How can we even tell?

In the ’80s, pop music became all about the technology and very little about the song. In journalism, we don’t have songs; we have stories.

In music, a  great song is timeless. In journalism, a great story is.

In music, the song transcends the instrument — it sounds great on guitar or piano or both. In journalism, the story transcends the medium — you can tell it with photo, graphic, app, text or all.

But an instrument without a song is nothing. So is a medium without a story.

I love apps. I love data. I love visualizations. But unless these toys of ours deliver a great story — one that moves me like the best, most authentic music — they’ll have all the lasting impact of Wang Chung.