Text_LanguageDetect
[ class tree: Text_LanguageDetect ] [ index: Text_LanguageDetect ] [ all elements ]

Class: Text_LanguageDetect

Source Location: /Text_LanguageDetect-0.3.0/Text/LanguageDetect.php

Class Overview


Language detection class


Author(s):

Version:

  • Release: @package_version@

Copyright:

  • 2005 Nicholas Pisarro

Methods


Child classes:

Text_LanguageDetect_Parser
This class represents a text sample to be parsed.

Inherited Variables

Inherited Methods


Class Details

[line 68]
Language detection class

Requires the langauge model database (lang.dat) that should have accompanied this class definition in order to be instantiated.

Example usage:

  1.  require_once 'Text/LanguageDetect.php';
  2.  
  3.  $l = new Text_LanguageDetect;
  4.  
  5.  $stdin fopen('php://stdin''r');
  6.  
  7.  echo "Supported languages:\n";
  8.  
  9.  try {
  10.      $langs $l->getLanguages();
  11.  catch (Text_LanguageDetect_Exception $e{
  12.      die($e->getMessage());
  13.  }
  14.  
  15.  sort($langs);
  16.  echo join(', '$langs);
  17.  
  18.  while ($line fgets($stdin)) {
  19.      print_r($l->detect($line4));
  20.  }



[ Top ]


Method Detail

__construct (Constructor)   [line 181]

Text_LanguageDetect __construct( )

Constructor

Will attempt to load the language database. If it fails, you will get an exception.


[ Top ]

clusteredSearch   [line 1350]

array clusteredSearch( string $str)

Perform an intelligent detection based on clusterLanguages()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

This compares the sample text to top the top level of clusters. If the sample is similar to the cluster it will drop down and compare it to the languages in the cluster, and so on until it hits a leaf node.

this should find the language in considerably fewer compares (the equivalent of a binary search), however clusterLanguages() is costly and the loss of accuracy from this technique is significant.

This method may need to be 'fuzzier' in order to become more accurate.

This function could be more useful if the universe of possible languages was very large, however in such cases some method of Bayesian inference might be more helpful.


Parameters:

string   $str     input string

[ Top ]

clusterLanguages   [line 1159]

array clusterLanguages( )

Cluster known languages according to languageSimilarity()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

Uses a nearest neighbor technique to generate the maximum possible number of dendograms from the similarity data.

  • Return: language cluster data
  • See: Text_LanguageDetect::languageSimilarity()
  • Deprecated: this function will eventually be removed and placed into the model generation class
  • Throws: Text_LanguageDetect_Exception
  • Access: public

[ Top ]

detect   [line 678]

mixed detect( string $sample, [int $limit = 0])

Detects the closeness of a sample of text to the known languages

Calculates the statistical difference between the text and the trigrams for each language, normalizes the score then returns results for all languages in sorted order

If perl compatible, the score is 300-0, 0 being most similar. Otherwise, it's 0-1 with 1 being most similar.

The $sample text should be at least a few sentences in length; should be ascii-7 or utf8 encoded, if another and the mbstring extension is present it will try to detect and convert. However, experience has shown that mb_detect_encoding() *does not work very well* with at least some types of encoding.

  • Return: sorted array of language scores, blank array if no useable text was found
  • See: _distance()
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

string   $sample     a sample of text to compare.
int   $limit     if specified, return an array of the most likely $limit languages and their scores.

[ Top ]

detectConfidence   [line 859]

array detectConfidence( string $sample)

Returns an array containing the most similar language and a confidence rating

Confidence is a simple measure calculated from the similarity score minus the similarity score from the next most similar language divided by the highest possible score. Languages that have closely related cousins (e.g. Norwegian and Danish) should generally have lower confidence scores.

The similarity score answers the question "How likely is the text the returned language regardless of the other languages considered?" The confidence score is one way of answering the question "how likely is the text the detected language relative to the rest of the language model set?"

To see how similar languages are a priori, see languageSimilarity()

  • Return: most similar language, score and confidence rating or null if no language is similar
  • See: Text_LanguageDetect::detect()
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

string   $sample     text for which language will be detected

[ Top ]

detectSimple   [line 819]

string detectSimple( string $sample)

Returns only the most similar language to the text sample

Calls $this->detect() and returns only the top result

  • Return: the name of the most likely language or null if no language is similar
  • See: Text_LanguageDetect::detect()
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

string   $sample     text to detect the language of

[ Top ]

detectUnicodeBlocks   [line 906]

array detectUnicodeBlocks( string $str, bool $skip_symbols)

Returns the distribution of unicode blocks in a given utf8 string

For the block name of a single char, use unicodeBlockName()

  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

string   $str     input string. Must be ascii or utf8
bool   $skip_symbols     if true, skip ascii digits, symbols and non-printing characters. Includes spaces, newlines and common punctutation characters.

[ Top ]

getLanguageCount   [line 360]

int getLanguageCount( )

Returns the number of languages that this object can detect
  • Return: the number of languages
  • Throws: Text_LanguageDetect_Exception
  • Access: public

[ Top ]

getLanguages   [line 402]

array getLanguages( )

Returns the list of detectable languages
  • Return: the names of the languages known to this object<<<<<
  • Throws: Text_LanguageDetect_Exception
  • Access: public

[ Top ]

languageExists   [line 372]

bool languageExists( mixed $lang)

Checks if the language with the given name exists in the database
  • Return: true if language model exists
  • Access: public

Parameters:

mixed   $lang     Language name or array of language names

[ Top ]

languageSimilarity   [line 1066]

array languageSimilarity( [string $lang1 = null], [string $lang2 = null])

Calculate the similarities between the language models

Use this function to see how similar languages are to each other.

If passed 2 language names, will return just those languages compared. If passed 1 language name, will return that language compared to all others. If passed none, will return an array of every language model compared to every other one.

  • Return: scores of every language compared or the score of just the provided languages or null if one of the supplied languages does not exist
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

string   $lang1     the name of the first language to be compared
string   $lang2     the name of the second language to be compared

[ Top ]

omitLanguages   [line 301]

int omitLanguages( mixed $omit_list, [bool $include_only = false])

Omits languages

Pass this function the name of or an array of names of languages that you don't want considered

If you're only expecting a limited set of languages, this can greatly speed up processing

  • Return: number of languages successfully deleted
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

mixed   $omit_list     language name or array of names to omit
bool   $include_only     if true will include (rather than exclude) only those in the list

[ Top ]

setNameMode   [line 440]

void setNameMode( integer $name_mode)

Sets the way how language names are accepted and returned.

Parameters:

integer   $name_mode     One of the following modes:
  1. - language name ("english")
2 - 2-letter ISO 639-1 code ("en") 3 - 3-letter ISO 639-2 code ("eng")

[ Top ]

setPerlCompatible   [line 416]

void setPerlCompatible( [bool $setting = true])

Make this object behave like Language::Guess
  • Access: public

Parameters:

bool   $setting     false to turn off perl compatibility

[ Top ]

unicodeBlockName   [line 935]

mixed unicodeBlockName( mixed $unicode)

Returns the block name for a given unicode value

If passed a string, will assume it is being passed a UTF8-formatted character and will automatically convert. Otherwise it will assume it is being passed a numeric unicode value.

Make sure input is of the correct type!

  • Return: the block name string or false if not found
  • Throws: Text_LanguageDetect_Exception
  • Access: public

Parameters:

mixed   $unicode     unicode value or utf8 char

[ Top ]

useUnicodeBlocks   [line 457]

void useUnicodeBlocks( [bool $setting = true])

Whether to use unicode block ranges in detection

Should speed up most detections if turned on (detault is on). In some circumstances it may be slower, such as for large text samples (> 10K) in languages that use latin scripts. In other cases it should speed up detection noticeably.

  • Access: public

Parameters:

bool   $setting     false to turn off

[ Top ]

utf8strlen   [line 1476]

int utf8strlen( string $str)

ut8-safe strlen()

Returns the numbers of characters (not bytes) in a utf8 string

  • Return: number of chars
  • Access: public

Parameters:

string   $str     string to get the length of

[ Top ]

_arr_rank   [line 493]

array _arr_rank( array $arr)

Converts a set of trigrams from frequencies to ranks

Thresholds (cuts off) the list at $this->_threshold

  • Return: ranks of trigrams
  • Access: protected

Parameters:

array   $arr     array of trigram

[ Top ]

_read_unicode_block_db   [line 1033]

array _read_unicode_block_db( )

Brings up the unicode block database
  • Return: the database of unicode block definitions
  • Throws: Text_LanguageDetect_Exception
  • Access: protected

[ Top ]

_unicode_block_name   [line 980]

mixed _unicode_block_name( int $unicode, array $blocks, [int $block_count = -1])

Searches the unicode block database

Returns the block name for a given unicode value. unicodeBlockName() is the public interface for this function, which does input checks which this function omits for speed.


Parameters:

int   $unicode     the unicode value
array   $blocks     the block database
int   $block_count     the number of defined blocks in the database

[ Top ]

_utf8char2unicode   [line 1495]

int _utf8char2unicode( string $char)

Returns the unicode value of a utf8 char

Parameters:

string   $char     a utf8 (possibly multi-byte) char

[ Top ]


Documentation generated on Mon, 16 Jan 2012 10:00:04 +0000 by phpDocumentor 1.4.3. PEAR Logo Copyright © PHP Group 2004.