[ML] YAKE! 예제 및 실습 try-outs

 

  [ML] YAKE! (1) 논문 리뷰 포스트에서 YAKE!의 알고리즘과 피처에 대해서 설명했었다. 이번 포스트에서는 임의의 도큐먼트들을 가지고 실습을 해보았다.

GitHub - LIAAD/yake: Single-document unsupervised keyword extraction

 

 

 

  필자가 YAKE!를 써보면서 종합적으로 느꼈던 장점을 정리해보자면

 

• n-gram range를 heuristic하게 정하지 않아도 됨(물론 max값을 정해주는데 있어 experiments는 불가피하다)
• 딥러닝 모델들에 비해 빠른 속도 - 모델을 load하지 않고 local feature만 이용해 계산하기 때문
• 하이퍼파라미터 최적화 덜 필요 (안 필요하다는 뜻은 아니다....)

가장 중요한 성능에 대해서는 글쎄.... 미지수로 남겨놓아야 할 듯 하다.

 

 

 

1. Settings

  Statistical approach라고 해서 하이퍼파라미터 튜닝이 필요하지 않다는 뜻은 아니다. 논문에서 설명한 YAKE!의 overall flow 알고리즘을 revisit해보자.

 

 

• text : document
• \( w\): the number of \(n\)-grams
• \(n \): window size
• \( \Theta \): deduplication threshold
• language: text language

  마찬가지로 YAKE! 패키지를 사용해 실습할 때도 위 파라미터들을 specify 해 줄 수 있다.

 

language = "en"
max_ngram_size = 3
deduplication_thresold = 0.9
deduplication_algo = 'seqm'   # candidate keyword 간 relevance score을 도출할 때 쓸 알고리즘
windowSize = 1
numOfKeywords = 20

custom_kw_extractor = yake.KeywordExtractor(lan=language,
					    n=max_ngram_size, 
                                            dedupLim=deduplication_thresold, 
                                            dedupFunc=deduplication_algo,
                                            windowsSize=windowSize, 
                                            top=numOfKeywords, 
                                            features=None)
keywords = custom_kw_extractor.extract_keywords(text)

for kw in keywords:
    print(kw)

 

 

2. 실습

  사용할 텍스트는 2월 25일자의 BBC 뉴스 Ukraine conflict: Kyiv braces for Russian assault 이다.

 

import yake

text = """
Ukraine conflict: Kyiv braces for Russian assault

The sound of gunfire has echoed through Ukraine's capital Kyiv as Russian tanks were filmed entering the city for the first time. Ukraine's Ministry of Defence called on those living in the northern outskirts, where the tanks were seen, to make fire bombs "to neutralise the enemy". The European Union has announced new sanctions to freeze Vladimir Putin's personal assets in Europe. The Kremlin, meanwhile, says that Russia is ready for talks with Ukraine. Spokesman Dmitry Peskov said talks would have to be about Ukraine declaring a "neutral status" - which would include "demilitarisation". Russia wants Ukraine to rule out ever joining Nato. Ukraine's President Volodymyr Zelensky has repeatedly called for talks with Mr Putin, but has given no sign that he would agree to talks on the basis of "demilitarisation". Overnight, Kyiv was hit by blasts, with at least one block of flats damaged and several civilians injured. Russia has denied carrying out strikes. An unnamed source at Russia's defence ministry also said a plane shot down over Kyiv was Ukrainian, Reuters news agency reports. Ukraine has said the jet was Russian. As the second day of fighting began, Moscow - which is attacking from the east, north and south - appeared to have Kyiv firmly in its sights. Mayor Vitali Klitschko said "the enemy" was trying to put Kyiv "on its knees". On Friday afternoon, Russia's defence ministry announced it had captured the Antonov airport, using 200 helicopters and a landing force in order to take the base to the north of Kyiv from Ukraine. Western intelligence officials warned earlier that Russia was building an "overwhelming force" to take control of the city. Citizens in the northern Obolon district have been told to stay at home to avoid "active military operations" by city officials, Reuters adds. Obolon is the same area where it appeared tanks were filmed earlier in the day. The Ministry of Defence had already appealed to the district's residents on its Facebook page to "inform us of troop movements, to make Molotov cocktails [firebombs] and neutralise the enemy". Overnight, families took shelter in Kyiv's metro stations as aerial attacks struck the city, including the densely populated Pozniake area, injuring at least eight. "Putin, we want to see you slaughtered like an animal," one Kyiv resident told the BBC's Nick Beake. "How we can live through it in our time?" Oxana Gulenko asked Reuters as she cleaned up broken glass from one blast. "What should we think. Putin should be burnt in hell along with his whole family." Earlier, Russian Foreign Minister Sergei Lavrov said no talks could be held until the Ukrainian military lays down its arms. Mr Lavrov is also included in the EU's move to freeze personal assets - but it not clear whether he or President Putin have any significant assets in Europe. Ukraine's President Volodymyr Zelensky had earlier said Russia was seeking to "destroy Ukraine politically by destroying the head of state". "According to the information we have, the enemy has designated me as target number one. My family is target number two," he said. President Zelensky has vowed to stay in Kyiv and to continue fighting the attacks across Ukraine - which is coming from the north, south and east - ordering the call-up of conscripts and reservists in all of Ukraine's regions. The country's defence minister urged anyone able to hold a weapon to join the effort to repel Russia. There have already been stories of immense bravery in the face of stark odds - including that of 13 border guards on a tiny island in the Black Sea who refused to surrender to a Russian warship and were massacred in a bombardment. President Zelensky said they would be given posthumous war hero honours. Thursday also saw fighting around the site of the former nuclear power plant in Chernobyl. Ukrainian presidential adviser Mykhaylo Podolyak said the nuclear site itself had been lost following a "fierce battle". On Friday, Ukraine's nuclear agency said it was recording raised radiation levels in the area. A statement released by Russia's Ministry of Defence said levels were normal, adding an agreement had been reached "to ensure security of the power plant and sarcophagus of the Chernobyl Nuclear Plant". Ukraine says at least 137 people - civilians and soldiers - have been killed. Russia has not admitted any deaths, whilst both sides claim to have inflicted heavy losses on the other. President Vladimir Putin - who declared war in a dramatic televised address - has threatened any country attempting to interfere with "consequences you have never seen". The UK, EU and other allies vowed to impose tough new sanctions to punish Moscow, but said they would not send in troops. French President Emmanuel Macron held a telephone call with his Russian counterpart, in what was Mr Putin's first conversation with a Western leader in days, demanding an "immediate halt" to the offensive. The Kremlin, however, simply said the pair had a "serious and frank exchange of views". Mr Zelensky, meanwhile, hit out at the European leaders' response. "Like World War Two, you said never again, but here it is again and you're not doing enough to respond," he said.
"""

 

language = 'en'
max_ngram_size = 1
deduplication_threshold = 0.8
window_size = 4
top_k = 15


kw_extractor = yake.KeywordExtractor(lan=language,
                                    n=max_ngram_size,
                                    dedupLim=deduplication_threshold,
                                    windowsSize=window_size,
                                    top=top_k)
keywords = kw_extractor.extract_keywords(text)
keywords

 

 

키워드-\(S(kw)\)의 튜플이 top_k개 만큼의 element를 가진 리스트 형태로 최종 output이 도출된다.