Docs For Class Text_LanguageDetect

__construct (Constructor) [line 167]

Text_LanguageDetect __construct(
)

Constructor

Will attempt to load the language database. If it fails, you will get an exception.

Access: public

Overridden in child classes as:

Text_LanguageDetect_Parser::__construct(): Constructor

[ Top ]

clusteredSearch [line 1322]

array clusteredSearch(
string
$str)

Perform an intelligent detection based on clusterLanguages()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

This compares the sample text to top the top level of clusters. If the sample is similar to the cluster it will drop down and compare it to the languages in the cluster, and so on until it hits a leaf node.

this should find the language in considerably fewer compares (the equivalent of a binary search), however clusterLanguages() is costly and the loss of accuracy from this technique is significant.

This method may need to be 'fuzzier' in order to become more accurate.

This function could be more useful if the universe of possible languages was very large, however in such cases some method of Bayesian inference might be more helpful.

Return: language scores (only those compared)
See: Text_LanguageDetect::clusterLanguages()
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string $str — input string

[ Top ]

clusterLanguages [line 1131]

array clusterLanguages(
)

Cluster known languages according to languageSimilarity()

WARNING: this method is EXPERIMENTAL. It is not recommended for common use, and it may disappear or its functionality may change in future releases without notice.

Uses a nearest neighbor technique to generate the maximum possible number of dendograms from the similarity data.

Return: language cluster data
See: Text_LanguageDetect::languageSimilarity()
Deprecated: this function will eventually be removed and placed into the model generation class
Throws: Text_LanguageDetect_Exception
Access: public

[ Top ]

detect [line 653]

mixed detect(
string
$sample, [int
$limit = 0])

Detects the closeness of a sample of text to the known languages

Calculates the statistical difference between the text and the trigrams for each language, normalizes the score then returns results for all languages in sorted order

If perl compatible, the score is 300-0, 0 being most similar. Otherwise, it's 0-1 with 1 being most similar.

The $sample text should be at least a few sentences in length; should be ascii-7 or utf8 encoded, if another and the mbstring extension is present it will try to detect and convert. However, experience has shown that mb_detect_encoding() *does not work very well* with at least some types of encoding.

Return: sorted array of language scores, blank array if no useable text was found
See: Text_LanguageDetect::_distance()
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string	`$sample`	—	a sample of text to compare.
int	`$limit`	—	if specified, return an array of the most likely $limit languages and their scores.

[ Top ]

detectConfidence [line 834]

array detectConfidence(
string
$sample)

Returns an array containing the most similar language and a confidence rating

Confidence is a simple measure calculated from the similarity score minus the similarity score from the next most similar language divided by the highest possible score. Languages that have closely related cousins (e.g. Norwegian and Danish) should generally have lower confidence scores.

The similarity score answers the question "How likely is the text the returned language regardless of the other languages considered?" The confidence score is one way of answering the question "how likely is the text the detected language relative to the rest of the language model set?"

To see how similar languages are a priori, see languageSimilarity()

Return: most similar language, score and confidence rating or null if no language is similar
See: Text_LanguageDetect::detect()
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string $sample — text for which language will be detected

[ Top ]

detectSimple [line 794]

string detectSimple(
string
$sample)

Returns only the most similar language to the text sample

Calls $this->detect() and returns only the top result

Return: the name of the most likely language or null if no language is similar
See: Text_LanguageDetect::detect()
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string $sample — text to detect the language of

[ Top ]

detectUnicodeBlocks [line 881]

array detectUnicodeBlocks(
string
$str, bool
$skip_symbols)

Returns the distribution of unicode blocks in a given utf8 string

For the block name of a single char, use unicodeBlockName()

Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string	`$str`	—	input string. Must be ascii or utf8
bool	`$skip_symbols`	—	if true, skip ascii digits, symbols and non-printing characters. Includes spaces, newlines and common punctutation characters.

[ Top ]

getLanguageCount [line 342]

int getLanguageCount(
)

Returns the number of languages that this object can detect

Return: the number of languages
Throws: Text_LanguageDetect_Exception
Access: public

[ Top ]

getLanguages [line 383]

array getLanguages(
)

Returns the list of detectable languages

Return: the names of the languages known to this object<<<<<
Throws: Text_LanguageDetect_Exception
Access: public

[ Top ]

languageExists [line 354]

bool languageExists(
mixed
$lang)

Checks if the language with the given name exists in the database

Return: true if language model exists
Access: public

Parameters:

mixed $lang — Language name or array of language names

[ Top ]

languageSimilarity [line 1039]

array languageSimilarity(
[string
$lang1 = null], [string
$lang2 = null])

Calculate the similarities between the language models

Use this function to see how similar languages are to each other.

If passed 2 language names, will return just those languages compared. If passed 1 language name, will return that language compared to all others. If passed none, will return an array of every language model compared to every other one.

Return: scores of every language compared or the score of just the provided languages or null if one of the supplied languages does not exist
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

string	`$lang1`	—	the name of the first language to be compared
string	`$lang2`	—	the name of the second language to be compared

[ Top ]

omitLanguages [line 284]

int omitLanguages(
mixed
$omit_list, [bool
$include_only = false])

Omits languages

Pass this function the name of or an array of names of languages that you don't want considered

If you're only expecting a limited set of languages, this can greatly speed up processing

Return: number of languages successfully deleted
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

mixed	`$omit_list`	—	language name or array of names to omit
bool	`$include_only`	—	if true will include (rather than exclude) only those in the list

[ Top ]

setNameMode [line 421]

void setNameMode(
integer
$name_mode)

Sets the way how language names are accepted and returned.

Access: public

Parameters:

integer

$name_mode

—

One of the following modes:

- language name ("english")

2 - 2-letter ISO 639-1 code ("en") 3 - 3-letter ISO 639-2 code ("eng")

[ Top ]

setPerlCompatible [line 397]

void setPerlCompatible(
[bool
$setting = true])

Make this object behave like Language::Guess

Access: public

Parameters:

bool $setting — false to turn off perl compatibility

[ Top ]

unicodeBlockName [line 910]

mixed unicodeBlockName(
mixed
$unicode)

Returns the block name for a given unicode value

If passed a string, will assume it is being passed a UTF8-formatted character and will automatically convert. Otherwise it will assume it is being passed a numeric unicode value.

Make sure input is of the correct type!

Return: the block name string or false if not found
Throws: Text_LanguageDetect_Exception
Access: public

Parameters:

mixed $unicode — unicode value or utf8 char

[ Top ]

useUnicodeBlocks [line 438]

void useUnicodeBlocks(
[bool
$setting = true])

Whether to use unicode block ranges in detection

Should speed up most detections if turned on (detault is on). In some circumstances it may be slower, such as for large text samples (> 10K) in languages that use latin scripts. In other cases it should speed up detection noticeably.

Access: public

Parameters:

bool $setting — false to turn off

[ Top ]

utf8strlen [line 1448]

int utf8strlen(
string
$str)

UTF8-safe strlen()

Returns the numbers of characters (not bytes) in a utf8 string

Return: number of chars
Access: public

Parameters:

string $str — string to get the length of

[ Top ]

_arr_rank [line 472]

array _arr_rank(
array
$arr)

Converts a set of trigrams from frequencies to ranks

Thresholds (cuts off) the list at $this->_threshold

Return: ranks of trigrams
Access: protected

Parameters:

array $arr — array of trigram

[ Top ]

_bub_sort [line 504]

void _bub_sort(

&$arr, array
$arr)

Sorts an array by value breaking ties alphabetically

Access: protected

Parameters:

array	`$arr`	—	the array to sort
	`&$arr`	—

[ Top ]

_checkTrigram [line 247]

void _checkTrigram(
array
$trigram)

Checks if this object is ready to detect languages

Access: protected

Parameters:

array $trigram — Trigram data from database

[ Top ]

_convertFromNameMode [line 1607]

string|array _convertFromNameMode(
string|array
$lang, [boolean
$convertKey = false])

Converts an $language input parameter from the configured mode to the language name that is used internally.

Works for strings and arrays.

Return: Language name
Access: protected

Parameters:

string\|array	`$lang`	—	A language description ("english"/"en"/"eng")
boolean	`$convertKey`	—	If $lang is an array, setting $key converts the keys to the language name.

[ Top ]

_convertToNameMode [line 1647]

string|array _convertToNameMode(
string|array
$lang, [boolean
$convertKey = false])

Converts an $language output parameter from the language name that is used internally to the configured mode.

Works for strings and arrays.

Return: Language name
Access: protected

Parameters:

string\|array	`$lang`	—	A language description ("english"/"en"/"eng")
boolean	`$convertKey`	—	If $lang is an array, setting $key converts the keys to the language name.

[ Top ]

_distance [line 582]

int _distance(
array
$arr1, array
$arr2)

Calculates a linear rank-order distance statistic between two sets of ranked trigrams

Sums the differences in rank for each trigram. If the trigram does not appear in both, consider it a difference of $this->_threshold.

This distance measure was proposed by Cavnar & Trenkle (1994). Despite its simplicity it has been shown to be highly accurate for language identification tasks.

Return: the sum of the differences between the ranks of the two trigram sets
Access: protected

Parameters:

array	`$arr1`	—	the reference set of trigram ranks
array	`$arr2`	—	the target set of trigram ranks

[ Top ]

_get_data_loc [line 190]

string _get_data_loc(
string
$fname)

Returns the path to the location of the database

Return: expected path to the language model database
Access: protected

Parameters:

string $fname — File name to load

[ Top ]

_next_char [line 1515]

char _next_char(
string
$str, 
&$counter, [bool
$special_convert = false], int
$counter)

UTF8-safe fast character iterator

Will get the next character starting from $counter, which will then be incremented. If a multi-byte char the bytes will be concatenated and $counter will be incremeted by the number of bytes in the char.

Return: the next (possibly multi-byte) char from $counter
Access: protected

Parameters:

string	`$str`	—	the string being iterated over
int	`$counter`	—	the iterator, will increment by reference
bool	`$special_convert`	—	whether to do special conversions
	`&$counter`	—

[ Top ]

_normalize_score [line 614]

float _normalize_score(
int
$score, [int
$base_count = null])

Normalizes the score returned by _distance()

Different if perl compatible or not

Return: the normalized score
See: Text_LanguageDetect::_distance()
Access: protected

Parameters:

int	`$score`	—	the score from _distance()
int	`$base_count`	—	the number of trigrams being considered

[ Top ]

_readdb [line 218]

array _readdb(
string
$fname)

Loads the language trigram database from filename

Trigram datbase should be a serialize()'d array

Return: the language model data
Throws: Text_LanguageDetect_Exception
Access: protected

Parameters:

string $fname — the filename where the data is stored

[ Top ]

_read_unicode_block_db [line 1006]

array _read_unicode_block_db(
)

Brings up the unicode block database

Return: the database of unicode block definitions
Throws: Text_LanguageDetect_Exception
Access: protected

[ Top ]

_sort_func [line 543]

int _sort_func(
array
$a, array
$b)

Sort function used by bubble sort

Callback function for usort().

Return: 1 if $a is greater, -1 if not
See: Text_LanguageDetect::_bub_sort()
Access: protected

Parameters:

array	`$a`	—	first param passed by usort()
array	`$b`	—	second param passed by usort()

[ Top ]

_trigram [line 453]

array _trigram(
string
$text)

Converts a piece of text into trigrams

Return: array of trigram frequencies
Deprecated: Superceded by the Text_LanguageDetect_Parser class
Access: protected

Parameters:

string $text — text to convert

[ Top ]

_unicode_block_name [line 954]

mixed _unicode_block_name(
int
$unicode, array
$blocks, [int
$block_count = -1])

Searches the unicode block database

Returns the block name for a given unicode value. unicodeBlockName() is the public interface for this function, which does input checks which this function omits for speed.

Return: Block name, -1 if it failed
See: Text_LanguageDetect::unicodeBlockName()
Access: protected

Parameters:

int	`$unicode`	—	the unicode value
array	`$blocks`	—	the block database
int	`$block_count`	—	the number of defined blocks in the database

[ Top ]

_utf8char2unicode [line 1466]

int _utf8char2unicode(
string
$char)

Returns the unicode value of a utf8 char

Return: unicode value
Link: http://en.wikipedia.org/wiki/UTF-8
Access: protected

Parameters:

string $char — a utf8 (possibly multi-byte) char

[ Top ]

Class: Text_LanguageDetect

Class Overview

Author(s):

Version:

Copyright:

Variables

Methods

Child classes:

Inherited Variables

Inherited Methods

Class Details

Class Variables

$_clusters =

$_data_dir = '@data_dir@'

$_db_filename = 'lang.dat'

$_lang_db = array()

$_max_score = 0

$_name_mode = 0

$_perl_compatible = false

$_threshold = 300

$_unicode_db_filename = 'unicode_blocks.dat'

$_unicode_map =

$_use_unicode_narrowing = true

Method Detail

__construct (Constructor) [line 167]

clusteredSearch [line 1322]

Parameters:

clusterLanguages [line 1131]

detect [line 653]

Parameters:

detectConfidence [line 834]

Parameters:

detectSimple [line 794]

Parameters:

detectUnicodeBlocks [line 881]

Parameters:

getLanguageCount [line 342]

getLanguages [line 383]

languageExists [line 354]

Parameters:

languageSimilarity [line 1039]

Parameters:

omitLanguages [line 284]

Parameters:

setNameMode [line 421]

Parameters:

setPerlCompatible [line 397]

Parameters:

unicodeBlockName [line 910]

Parameters:

useUnicodeBlocks [line 438]

Parameters:

utf8strlen [line 1448]

Parameters:

_arr_rank [line 472]

Parameters:

_bub_sort [line 504]

Parameters:

_checkTrigram [line 247]

Parameters:

_convertFromNameMode [line 1607]

Parameters:

_convertToNameMode [line 1647]

Parameters:

_distance [line 582]

Parameters:

_get_data_loc [line 190]

Parameters:

_next_char [line 1515]

Parameters:

_normalize_score [line 614]

Parameters:

_readdb [line 218]

Parameters:

_read_unicode_block_db [line 1006]

_sort_func [line 543]

Parameters:

_trigram [line 453]

Parameters:

_unicode_block_name [line 954]