Changes from v0.1a to v0.2a

The global variables environment (vars class) has been added. It allows the user and the parsing algorithms to set, unset and read global variables.

New command line commands are: set, unset, get.
The command line command showconf has been extended to show all defined global variables.

Important change: The algorithm class has been changed. The parse function has a new parameter for the global variables enviroment. All parsing algorithms from v0.1a should be changed by adding the new parameter to the parse function.

See Creating new algorithms for more details.

Important change: The file format for the configuration files has been changed. It now includes the global variables. However, older configuration files (from v0.1a) are still supported.

Changes from v0.2a to v0.3a

Added the program arguments --start and -s to enable immediate starting of the loaded configuration file. No command line will be loaded. See Usage for more details.

Removed bug from the parsing of program arguments that occured when the name of the configuration file included spaces.

Changes from v0.3a to v0.4a

The global variables environment (vars class) has been made safer by adding an array that holds the information, whether a variable is a string or not, so other variables can be skipped while performing string-only operations. Before, these operations were not possible as soon as non-string variables have been added.

Important change: The algorithm class has been changed. The init function has a new parameter for the global variables enviroment. All parsing algorithms should be changed by adding the new parameter to the init function.

See Creating new algorithms for more details.

For compatibility reasons, the very same argument passed to the parse function remains.

Changes from v0.4a to v0.5a

Support for reading binary files has been added. The output file will be written in binary mode, supporting binary output files, too. The compatibility of algorithms will now be checked by introducing version constants (in version.h) and the new get_program_version function for algorithms (see below).

If you want to use the new binary mode (all input files will be handled as binary files), you have to set the global variable binary to any value, e.g. by using the command set binary, see Binary mode for details.

Important change: The algorithm class has been changed. The init and parse functions have new parameters for the size of the used buffers. Algorithms that write into the output/content buffers have to use these new arguments for providing the size of their data, because binary data is now supported and the strlen function cannot be used anymore on the buffers.

Important change: The algorithm class has been changed. The parse function doesn't include any argument for the global variables environment anymore. If needed in this function, the pointer has to be saved by the init function.

Important change: The algorithm class has been changed. For compatibility checks, every algorithm has to include the get_program_version function to return the used version of the main program. The version constants to be used can be found in the version.h header file of the main program.

All parsing algorithms from previous versions need to be changed by updating their member functions according to these changes. See Creating new algorithms for more information.

Important change: The vars class has been changed. There was a major bug in the set functions which has been removed. Another bug in the save_strings function, which could led to incomplete configuration files, has been removed. Most of the functions now return a boolean value for error handling and are able to handle memory shortages. All parsing algorithms need to use the new vars class.

Important change: The filelist class has been changed. It now handles relative path names in a better way, that will later be needed for HTTP support. Error handling has been added to the save and open functions. Also, the add_file function of the class is now able to tell whether a file has already existed and not been added. All parsing algorithms should use the new filelist class.

The parser class has been changed. Error handling has been added to the write_conf and read_conf functions. The read_conf function is now able to handle memory shortages.

The set command now doesn't need a value anymore. If used without value, it will set the variable to an empty string.

Changes from v0.5a to v0.6a

Important change: The algorithm class has been changed. Use of long instead of int. All algorithms need to be changed, for details see Creating new algorithms.

Important change: The filelist class has been changed. A major bug in the add_file function has been removed. The function also has a new parameter for the name of the algorithm that adds the file. In urls, it now ignores everything behind #. Like the "algorithm" class, the class now uses long instead of int. All algorithms using this class should be updated.

Important change: The vars class has been changed. Bugs in the open and add_alloc functions have been removed. Like the "algorithm" class, the class now uses long instead of int. All algorithms using this class need to be updated.

Important change: The file format of the configuration files has been changed. Global variables will be read before the parser configuration (used algorithms), so that the variables will affect the added algorithms correctly. Older configuration files are not supported anymore. See Reference: Configuration file formats for more information.

The new console command clearvars removes all added global variables. The clear console command has been updated to remove all added global variables, too.

Changes from v0.6a to v0.7a

Important change: The algorithm class has been changed. A new member function multiple_instances has been added, which returns, whether multiple instances of an algorithm are allowed. All algorithms need to be changed by adding the function, for details see Creating new algorithms.

bool init(char ** output, unsigned long * output_size, void * v)

Will be called when the algorithm is added to a file.

The output argument points to the output buffer, but *output will be NULL if no output has been created. Use the malloc/calloc functions to create the output buffer, it will be freed by the main program, but make sure to free any existing output buffer before by using the free function. In this case, you ideally would include the existing output in your new output.

The output_size argument points to a variable holding the size of the output buffer. If you change the output you have to change the value of this variable accordingly.

In the end of the parsing process, the output buffer (if existing) will be written to the output file. Both, text and binary output, are supported since version 0.5a.

Usually you will not need to use the output buffer in the init function already.

The v argument points to the vars object used for the global variable environment. You only need to use it if you want to set, unset or read global variables set by the user or other algorithms. If you want to use the vars object, you have to include the header file (../../src/vars.h). You also need to add the source code for the filelist object (../../src/vars.cpp) when compiling the algorithm. See Compiling algorithms for more information.

If you want to use global variables in your parse function, you have to save the pointer submitted to the init function in a variable inside the source code of your algorithm.

Please name your variables in the following format: <algoname>.<variable>. For example, see the variables used by the HTMLSpider algorithm:

html_spider.max_files_to_add
html_spider.tmp_added_files

Return true, if the loading of the algorithm was successfull or you don't need to load anything.
Returning false would mean, that the loading of the algorithm failed and it cannot be used for parsing the file.

bool parse(const char * fname, char ** content, unsigned long * content_size, char ** output, unsigned long * output_size, void * flist)

Will be called to parse the content of the file.

The content argument will point to the content buffer, containing the file content to parse. Make sure that you free the memory of the old content buffer by using the free function if you intend to write content that could be larger than the old one.

Use the malloc/calloc functions to create a larger content buffer, the program will free it automatically in the end.

The content_size argument points to a variable holding the size of the current content. If you change the content you have to change the value of this variable accordingly.

The output argument points to the output buffer, but *output will be NULL if no output has been created. Use the malloc/calloc functions to create the output buffer, it will be freed by the main program, but make sure you free any existing output buffer before by using the free function. In this case, you ideally would include the existing output in your new output.

The output_size argument points to a variable holding the size of the output buffer. If you change the output you have to change the value of this variable accordingly.

In the end of the parsing process, the output buffer (if existing) will be written to the output file. Both, text and binary output, are supported (since version 0.5a).

The fname argument contains the string with the file name. Usually, you don't need to use it, because the file has already been opened and read by the main program. Use the content buffer instead, to make sure, you parse the right content, not the original file. However, the fname argument can be useful for documentation purposes or if you need to find out the path of the file.

Note: The file name can be local (without directories or with sub-directories only) or global (complete path, url). Basically, the program supports urls starting with http://, https://, ftp:// and file:// (to be implemented by version 1.0b).

The flist argument points to the filelist object used for the list of files to parse. You only need to use it if you want to manipulate the list, e.g. adding additional files to parse in the end of the process. If you want to use the filelist object, you have to include the header file (../../src/filelist.h). You also need to add the source code for the filelist object (../../src/filelist.cpp) when you compile your algorithm. See Compiling algorithms for more information.

Return true if the parsing was successfull.

Returning false would indicate an error but not stop the rest of the parsing process. This can lead to unexpected results you should warn the user about.

void clear()

Is called when the algorithm is removed from the file or the parsing process is either cancelled or finished. You have to free used memory here, except for the content or output buffers, this will be done by the program.

const char * get_name() const
const char * get_version() const
const char * get_author() const

These three functions should return static strings containing name, version and author of the algorithm. They will be used by the main program to show more information about the added algorithm.

unsigned int get_program_version() const

This function has to return the version of the main program for which your algorithm has been written. Older versions of the main program will not be able to use your algorithm. If the version of the main program is higher than the version used by you, a warning message will be shown.

Make sure to update your algorithm with each new version of the main program. Check the Changelog section of the readme file to learn about possible changes to make to your algorithm.

The version constants that you should use are defined in the version.h header file of the main program, which should be included. You can also use the PARSEFILE_VER_CURRENT constant, which will be automatically updated in every new version of the main program, provided you don't use a copy of an outdated header file.

Example: If your algorithm source code is located inside the program's algo/src folder, use

#include "../../src/version.h"

to include the original (up-to-date) header file.

bool multiple_instances() const

This function has to return, whether multiple instances of the algorithm are allowed.

Note, that each instance has access to the same global variables environment. It is not possible to define different variables with the same name for different instances of your algorithm.

Full package

Minimal package

parsefile - executable program binary
README.txt - readme file
algos/ - folder for parsing algorithms
algos/command_line.so - the command_line parsing algorithm
algos/count_strings.so - the count_string parsing algorithm
algos/html_body.so - the html_body parsing algorithm
algos/html_remove_comments.so - the html_remove_comments parsing algorithm
algos/html_remove_tags.so - the html_remove_tags parsing algorithm
algos/html_spider.so - the HTMLSpider parsing algorithm
algos/print_binary.so - the print_binary parsing algorithm
algos/print_content.so - the print_content parsing algorithm
algos/show_content.so - the show_content parsing algorithm
algos/wait.so - the wait parsing algorithm
conf/ - folder for configuration files
output/ - folder for output files