parsing Wikileaks’ data
Mostly developed in 2o11 already, version 0.8a of the parsefile project is online now. In 2o11, I also programmed two additional algorithms for the project, parsing the massive Wikileaks leak about the Iraq war and showing unit movements on a map. Finally, I want to share how to do this.
What we need
For parsing, we need the CSV data, of course. It’s the Wikileaks CSV file (iraq-war-diary-redacted.csv), which can be found e.g. on The Pirate Bay (no link provided here).
For visualizing, we need the Iraq map as published by the U.S. Government in 2003, in the resolution 2000×2161 pixels (if it’s not in the right resolution, make sure to resize it before you use it!).
On the software side, we need the parsefile project, of course, and these two additional algorithms, programmed by me: war_units and war_units_iraq. The algorithm package (including source code) can be found here (tar.gz, 45,2 KB) - just copy all of the included files into the “parsefile” directory, using the same folder structure as created inside the package.
Parsing the CSV data
The algorithm “war_units” will be used to parse the original CSV data and create two output files, one for the movements (war_units_moves.txt) and one for the list of the units (war_units_table.txt).
Make sure you added the “war_units” algorithm and downloaded the CSV data to put it into the “input” folder of parsefile. Rename it into “iraq.csv”.
Start parsefile and type the following commands:
addfile input/iraq.csv
addalgo war_units
set war_units.filter=5
start
The CSV file will be parsed and the two output files will be created in the “ourput” folder. The filter has been set to only parse units with at least 5 actions. Warning: Parsing will take a very long time!
Visualizing the movement data
The algorithm “war_units_iraq” uses these two files to show a map with the visualized unit movements. Make sure you added the “war_units_iraq” algorithm and parsed the CSV file. Now start “parsefile” and run the following commands:
addalgo war_units_iraq
start
After a while sorting the data, the interactive map will be shown. On the right side you can select one of the parsed units and on the map the movement profile of the selected unit will be shown.
Screenshots
(starting the parsing)
(parsing the data)
(visualized data on map)