RAKE簡介

RAKE英文全稱為Rapid Automatic keyword extraction，中文稱為快速自動關鍵字提取，是一種非常高效的關鍵字提取算法，可對單個文檔進行操作，以實現對動態集合的應用，也可非常輕松地應用于新域，并且在處理多種類型的文檔時也非常有效。

2

算法思想

RAKE算法用來做關鍵詞(keyword)的提取，實際上提取的是關鍵的短語(phrase)，并且傾向于較長的短語，在英文中，關鍵詞通常包括多個單詞，但很少包含標點符號和停用詞，例如and，the，of等，以及其他不包含語義信息的單詞。

RAKE算法首先使用標點符號（如半角的句號、問號、感嘆號、逗號等）將一篇文檔分成若干分句，然后對于每一個分句，使用停用詞作為分隔符將分句分為若干短語，這些短語作為最終提取出的關鍵詞的候選詞。

最后，每個短語可以再通過空格分為若干個單詞，可以通過給每個單詞賦予一個得分，通過累加得到每個短語的得分。一個關鍵點在于將這個短語中每個單詞的共現關系考慮進去。最終定義的公式是:

3

算法步驟

(1)算法首先對句子進行分詞，分詞后去除停用詞，根據停 用詞劃分短語;

(2)之后計算每一個詞在短語的共現詞數,并構建 詞共現矩陣;

(3)共現矩陣的每一列的值即為該詞的度deg（是一個網絡中的概念，每與一個單詞共現在一個短語中，度就加1，考慮該單詞本身）,每個詞在文本中出現的次數即為頻率freq;

(4)得分score為度deg與頻率 freq的商,score越大則該詞更重 ;

(5)最后按照得分的大小值降序 輸出該詞所在的短語。

下面我們以一個中文例子具體解釋RAKE算法原理，例如“系統有聲音，但系統托盤的音量小喇叭圖標不見了”，經過分詞、去除停用詞處理 后得到的詞集W = {系統，聲音，托盤，音量，小喇叭，圖標，不見}，短語集D={系統，聲音，系統托盤，音量小喇叭圖標不見}，詞共現矩陣如表：

每一個詞的度為deg={"系統”：2，“聲音”：1,“托盤”:1; “音量” ：3; “小喇叭” ：3，“圖標” ：3，“不見” ：3}，頻率freq = { “系統” ：2, “聲音” ：1, “托盤” ：1 ；“音量” ：1；“小喇叭” ：1, “圖標”丄“不見” ：1}, score ={“系統”：1,“聲音”：1,“托 盤” ：1 ；“音量” ：1小喇叭” ：3, “圖標” ：3, “不見” ：3 },輸出結果為{音量小喇叭圖標不見 ,系統托盤，系統，聲音}

4

代碼實現

importstring

fromtypingimportDict,List,Set,Tuple

PUNCTUATION=string.punctuation.replace(''','')#Donotuseapostropheasadelimiter

ENGLISH_WORDS_STOPLIST:List[str]=[
'(',')','and','of','the','amongst','with','from','after','its','it','at','is',
'this',',','.','be','in','that','an','other','than','also','are','may','suggests',
'all','where','most','against','more','have','been','several','as','before',
'although','yet','likely','rather','over','a','for','can','these','considered',
'used','types','given','precedes',
]


defsplit_to_tokens(text:str)->List[str]:
'''
Splittextstringtotokens.
Behaviorissimilartostr.split(),
butemptylinesareomittedandpunctuationmarksareseparatedfromword.
Example:
split_to_tokens('Johnsaid'Hey!'(andsomeotherwords.)')->
->['John','said',''','Hey','!',''','(','and','some','other','words','.',')']
'''
result=[]
foritemintext.split():
whileitem[0]inPUNCTUATION:
result.append(item[0])
item=item[1:]
foriinrange(len(item)):
ifitem[-i-1]notinPUNCTUATION:
break
ifi==0:
result.append(item)
else:
result.append(item[:-i])
result.extend(item[-i:])
return[itemforiteminresultifitem]


defsplit_tokens_to_phrases(tokens:List[str],stoplist:List[str]=None)->List[str]:
"""
Mergetokensintophrases,delimitedbyitemsfromstoplist.
Phraseisasequenceoftokenthathasthefollowingproperties:
-phrasecontains1ormoretokens
-tokensfromphrasegoinarow
-phrasedoesnotcontaindelimitersfromstoplist
-eithertheprevious(notinaphrase)tokenbelongstostoplistoritisthebeginningoftokenslist
-eitherthenext(notinaphrase)tokenbelongstostoplistoritistheendoftokenslist
Example:
split_tokens_to_phrases(
tokens=['Mary','and','John',',','some','words','(','and','other','words',')'],
stoplist=['and',',','.','(',')'])->
->['Mary','John','somewords','otherwords']
"""
ifstoplistisNone:
stoplist=ENGLISH_WORDS_STOPLIST
stoplist+=list(PUNCTUATION)

current_phrase:List[str]=[]
all_phrases:List[str]=[]
stoplist_set:Set[str]={stopword.lower()forstopwordinstoplist}
fortokenintokens:
iftoken.lower()instoplist_set:
ifcurrent_phrase:
all_phrases.append(''.join(current_phrase))
current_phrase=[]
else:
current_phrase.append(token)
ifcurrent_phrase:
all_phrases.append(''.join(current_phrase))
returnall_phrases


defget_cooccurrence_graph(phrases:List[str])->Dict[str,Dict[str,int]]:
"""
Getgraphthatstorescooccurenceoftokensinphrases.
Matrixisstoredasdict,
wherekeyistoken,valueisdict(keyissecondtoken,valueisnumberofcooccurrence).
Example:
get_occurrence_graph(['Mary','John','somewords','otherwords'])->{
'mary':{'mary':1},
'john':{'john':1},
'some':{'some':1,'words':1},
'words':{'some':1,'words':2,'other':1},
'other':{'other':1,'words':1}
}
"""
graph:Dict[str,Dict[str,int]]={}
forphraseinphrases:
forfirst_tokeninphrase.lower().split():
forsecond_tokeninphrase.lower().split():
iffirst_tokennotingraph:
graph[first_token]={}
graph[first_token][second_token]=graph[first_token].get(second_token,0)+1
returngraph


defget_degrees(cooccurrence_graph:Dict[str,Dict[str,int]])->Dict[str,int]:
"""
Getdegreesforalltokensbycooccurrencegraph.
Resultisstoredasdict,
wherekeyistoken,valueisdegree(sumoflengthsofphrasesthatcontainthetoken).
Example:
get_degrees(
{
'mary':{'mary':1},
'john':{'john':1},
'some':{'some':1,'words':1},
'words':{'some':1,'words':2,'other':1},
'other':{'other':1,'words':1}
}
)->{'mary':1,'john':1,'some':2,'words':4,'other':2}
"""
return{token:sum(cooccurrence_graph[token].values())fortokenincooccurrence_graph}


defget_frequencies(cooccurrence_graph:Dict[str,Dict[str,int]])->Dict[str,int]:
"""
Getfrequenciesforalltokensbycooccurrencegraph.
Resultisstoredasdict,
wherekeyistoken,valueisfrequency(numberoftimesthetokenoccurs).
Example:
get_frequencies(
{
'mary':{'mary':1},
'john':{'john':1},
'some':{'some':1,'words':1},
'words':{'some':1,'words':2,'other':1},
'other':{'other':1,'words':1}
}
)->{'mary':1,'john':1,'some':1,'words':2,'other':1}
"""
return{token:cooccurrence_graph[token][token]fortokenincooccurrence_graph}


defget_ranked_phrases(phrases:List[str],*,
degrees:Dict[str,int],
frequencies:Dict[str,int])->List[Tuple[str,float]]:
"""
GetRAKEmeasureforeveryphrase.
Resultisstoredaslistoftuples,everytuplecontainsofphraseanditsRAKEmeasure.
Itemsaresortednon-ascendingbyRAKEmeasure,thanalphabeticallybyphrase.
"""
processed_phrases:Set[str]=set()
ranked_phrases:List[Tuple[str,float]]=[]
forphraseinphrases:
lowered_phrase=phrase.lower()
iflowered_phraseinprocessed_phrases:
continue
score:float=sum(degrees[token]/frequencies[token]fortokeninlowered_phrase.split())
ranked_phrases.append((lowered_phrase,round(score,2)))
processed_phrases.add(lowered_phrase)
#Sortbyscorethanbyphrasealphabetically.
ranked_phrases.sort(key=lambdaitem:(-item[1],item[0]))
returnranked_phrases


defrake_text(text:str)->List[Tuple[str,float]]:
"""
GetRAKEmeasureforeveryphraseintextstring.
Resultisstoredaslistoftuples,everytuplecontainsofphraseanditsRAKEmeasure.
Itemsaresortednon-ascendingbyRAKEmeasure,thanalphabeticallybyphrase.
"""
tokens:List[str]=split_to_tokens(text)
phrases:List[str]=split_tokens_to_phrases(tokens)
cooccurrence:Dict[str,Dict[str,int]]=get_cooccurrence_graph(phrases)
degrees:Dict[str,int]=get_degrees(cooccurrence)
frequencies:Dict[str,int]=get_frequencies(cooccurrence)
ranked_result:List[Tuple[str,float]]=get_ranked_phrases(phrases,degrees=degrees,frequencies=frequencies)
returnranked_result

執行效果：

if__name__=='__main__':
text='Mercy-classincludesUSNSMercyandUSNSComforthospitalships.Credit:USNavyphotoMassCommunicationSpecialist1stClassJasonPastrick.TheUSNavalAirWarfareCenterAircraftDivision(NAWCAD)LakehurstinNewJerseyisusinganadditivemanufacturingprocesstomakefaceshields.........'
ranked_result=rake_text(text)
print(ranked_result)

關鍵短語抽取效果如下：

[
('additivemanufacturingprocesstomakefaceshields.the3dprintingfaceshields',100.4),
('usnavyphotomasscommunicationspecialist1stclassjasonpastrick',98.33),
('usnavy’smercy-classhospitalshipusnscomfort.currentlystationed',53.33),
...
]