CorpusMasker v. 0.1 - December 02, 2007

---------------------------------------------------------------------------------

Author: Johannes Dellert <jdellert@sfs.uni-tuebingen.de>
		    SFB 441: Linguistic Data Structures
		    University of Tübingen, Germany
		
---------------------------------------------------------------------------------

Copyright: 
University of Tübingen
Project C2: Sustainability of Linguistic Data
SFB 441: Linguistic Data Structures
Nauklerstraße 35
72074 Tübingen, Germany

This software may be used freely for educational and scientific purposes.
Future releases will be under GPL or similar.

---------------------------------------------------------------------------------
		
CorpusMasker is a software tool for parametrized masking of linguistic resources.

---------------------------------------------------------------------------------

This distribution contains four java programs:

1. CorpusMaskerGUI
	- a graphical frontend to mask content in XML files
	- type "java CorpusMaskerGUI" to start
	
2. CorpusMasker
	- a console tool to mask content in XML files
	- for usage information, type "java CorpusMasker"
	
3. CorpusDemasker
	- a console tool to revert the masking as performed by CorpusMasker
	- for usage information, type "java CorpusDemasker"

4. TextExtractor
	- a console tool to extract content from XML files
	- used mainly for debugging purposes
	- useful for getting a better impression of the masking quality

------------------------------------------------------------------------------------

Usage example:

Assume we have a corpus XML file "corpus.xml" with sentences in a format like the following:

<s>
  <node tag="np">
    <token>
      <pos>N</pos>
      <orth>Peter</orth>
    </token>
  </node>
  <node tag="vp">
    <token>
      <pos>V</pos>
      <orth>walks</orth>
    </token>
  </node>
</s>

We want to mask this corpus by stripping off all the leaves (i.e. the content of the orth elements):

# java CorpusMasker --strip corpus.xml corpus-masked.xml

The output in corpus-masked.xml will then look like the following:

<s>
  <node tag="np">
    <token>
      <pos>N</pos>
      <orth></orth>
    </token>
  </node>
  <node tag="vp">
    <token>
      <pos>V</pos>
      <orth></orth>
    </token>
  </node>
</s>

Another option would be to replace the orth content by sequences of X chars (word length information is retained):

# java CorpusMasker --x-replace corpus.xml corpus-masked.xml

Now the output would be the following:

<s>
  <node tag="np">
    <token>
      <pos>N</pos>
      <orth>Xxxxx</orth>
    </token>
  </node>
  <node tag="vp">
    <token>
      <pos>V</pos>
      <orth>xxxxx</orth>
    </token>
  </node>
</s>

And now for the more advanced options:

CorpusMasker offers the possibility to generate random replacement patterns that observe the general phonological pattern of the original tokens. These patterns can be stored in a dictionary mapping original tokens to their masked counterparts. This makes it possible to revert the masking using the dictionary file produced during the process.

To mask a corpus using this method, do the following:

# java CorpusMasker --buildDic --ccf corpus.xml corpus-masked.xml

The --ccf option will cause the program to prompt the user for a Character Class File. Since in this case, we are dealing with an English language corpus, the file lang/english.ccf should be used instead of the default lang/german.ccf.
For more information on the CCF file format, see below.

When prompted for the location of the dictionary file, let us store it under dict.txt.

Let us have a look at the output:

<s>
  <node tag="np">
    <token>
      <pos>N</pos>
      <orth>Dabol</orth>
    </token>
  </node>
  <node tag="vp">
    <token>
      <pos>V</pos>
      <orth>vimpx</orth>
    </token>
  </node>
</s>

The dictionary file we created should now contain the following entries:

N Peter Dabol
V walks vimpx

For more information on the dictionary format, see the format specification below.

In order to recover the original version using the masked version and the dictionary, type:

# java CorpusDemasker dict.txt corpus-masked.xml corpus-back.xml

corpus-back.xml will be identical to the original corpus.xml

The last and most complex option is to make use of the program's ability to infer morphology and leave it untouched.

To try this out, we would type:

# java CorpusMasker --buildDic-aff --ccf corpus.xml corpus-masked.xml

A typical result of this command would look like the following:

<s>
  <node tag="np">
    <token>
      <pos>N</pos>
      <orth>Tuker</orth>
    </token>
  </node>
  <node tag="vp">
    <token>
      <pos>V</pos>
      <orth>herks</orth>
    </token>
  </node>
</s>

Observe that the third person singular -s and the -er in Peter have been preserved, both because a sufficiently large percentage (by default 2 %) of the words in their respective POS class ended in the same affix.

There are multiple options to tune this morphology induction.

If you want more affixes to be preserved, decrease the corresponding parameter below its default value (2 % = 0.02):

# java CorpusMasker --buildDic-aff --ccf --affixDefOccPerWord=0.01 corpus.xml corpus-masked.xml

If you want small word classes to be excluded from masking, you have two options:

Either define POS classes whose members will not be masked at all:

# java CorpusMasker --buildDic --preservePOS=CONJ,PREP,NUM corpus.xml corpus-masked.xml

Or use the other affix extraction parameter to decrease the number of distinct words that an affix has to occur in in order to be preserved. This is the defMinOccurs parameter:

# java CorpusMasker --buildDic-aff --ccf --affixDefMinOccurs=1 corpus.xml corpus-masked.xml

The default value (10) prevents all parts of words in small word classes from being recognized as affixes, resulting in no masking effect at all for those POS classes. If this is a desired effect, you can decrease the number to leave words not relevant for the protected content intact and available for inspection.

Sometimes you might want to exclude a whole POS class from affix recognition. For instance, trying to find affixes in words of type "named entity" is mostly useless, if not dangerous because it could disclose sensitive data. To prevent this from happening, you can type

# java CorpusMasker --buildDic-aff --noMorph=NE corpus.xml corpus-masked.xml

All tokens with POS class NE will be completely masked.
 

Another important question has remained unanswered so far: 
How does the program know where to find the content information and the assigned POS tags? 

The answer is that our example corpus adhered to the default values derived from the format this software was originally designed for, where content is always found as CDATA in "orth" elements and the POS information is encoded as CDATA within "pos" elements. 

The mechanisms for also covering other formats are still in development, so you might encounter some unexpected behaviour. However, in principle it is possible to freely define the locations of content and POS information using a subset of XPath.

Valid expressions may contain child axes (/), attribute axes(/@) and the wildcard for arbitrarily many levels (..).

A few examples of XPath expressions that will be correctly evaluated:

../orth (default)
../pos (default)
/s/node/@tag
/s/node/../token/@pos

The severe restrictions on XPaths that can be used are due to the fact that for performance reasons, documents are processed sequentially using SAX parsers, making it impossible to provide for more expressive power in the XPath expressions.

-------------------------------------------------------------------------------------

The following two file formats are used by the tool:

I) Conversion Dictionary Format
	- a simple text format containing lines of the following schema:
		[POS class] [original] [masked]
	- lines are sorted in POS class blocks with empty lines in between
	- this format is used to store masking information
	- files in this format can be used to revert the masking process using CorpusDemasker

II) Character Class File (CCF) Format
	- this format is used to specify character classes for replacement during pattern generation
	- replacement will only take place within these character classes
	- in the first two lines, punctuation and numerals are defined:
		PUNCT [a list of punctuation characters separated by spaces]
		DIGITS [a list of numeral characters separated by spaces]
	- the other lines contain one character class each
		- the members of one class are separated by spaces
		- members in braces will not occur in replacement patterns
			==> do not put all members of a class into braces!
			==> for short words, making output classes smaller than input classes
				  might hamper the reversibility of the masking process
				
------------------------------------------------------------------------------------
	
	 


