首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >为什么utorrents到Torrent文件获取比我的python脚本更快?

为什么utorrents到Torrent文件获取比我的python脚本更快?
EN

Stack Overflow用户
提问于 2015-09-24 05:30:08
回答 1查看 1.9K关注 0票数 2

我正在尝试使用python脚本在.torrent文件中转换洪流磁铁urls。python连接到dht并等待元数据,然后从它创建洪流文件。

例如:

代码语言:javascript
复制
#!/usr/bin/env python
'''
Created on Apr 19, 2012
@author: dan, Faless

    GNU GENERAL PUBLIC LICENSE - Version 3

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

    http://www.gnu.org/licenses/gpl-3.0.txt

'''

import shutil
import tempfile
import os.path as pt
import sys
import libtorrent as lt
from time import sleep


def magnet2torrent(magnet, output_name=None):
    if output_name and \
            not pt.isdir(output_name) and \
            not pt.isdir(pt.dirname(pt.abspath(output_name))):
        print("Invalid output folder: " + pt.dirname(pt.abspath(output_name)))
        print("")
        sys.exit(0)

    tempdir = tempfile.mkdtemp()
    ses = lt.session()
    params = {
        'save_path': tempdir,
        'duplicate_is_error': True,
        'storage_mode': lt.storage_mode_t(2),
        'paused': False,
        'auto_managed': True,
        'duplicate_is_error': True
    }
    handle = lt.add_magnet_uri(ses, magnet, params)

    print("Downloading Metadata (this may take a while)")
    while (not handle.has_metadata()):
        try:
            sleep(1)
        except KeyboardInterrupt:
            print("Aborting...")
            ses.pause()
            print("Cleanup dir " + tempdir)
            shutil.rmtree(tempdir)
            sys.exit(0)
    ses.pause()
    print("Done")

    torinfo = handle.get_torrent_info()
    torfile = lt.create_torrent(torinfo)

    output = pt.abspath(torinfo.name() + ".torrent")

    if output_name:
        if pt.isdir(output_name):
            output = pt.abspath(pt.join(
                output_name, torinfo.name() + ".torrent"))
        elif pt.isdir(pt.dirname(pt.abspath(output_name))):
            output = pt.abspath(output_name)

    print("Saving torrent file here : " + output + " ...")
    torcontent = lt.bencode(torfile.generate())
    f = open(output, "wb")
    f.write(lt.bencode(torfile.generate()))
    f.close()
    print("Saved! Cleaning up dir: " + tempdir)
    ses.remove_torrent(handle)
    shutil.rmtree(tempdir)

    return output


def showHelp():
    print("")
    print("USAGE: " + pt.basename(sys.argv[0]) + " MAGNET [OUTPUT]")
    print("  MAGNET\t- the magnet url")
    print("  OUTPUT\t- the output torrent file name")
    print("")


def main():
    if len(sys.argv) < 2:
        showHelp()
        sys.exit(0)

    magnet = sys.argv[1]
    output_name = None

    if len(sys.argv) >= 3:
        output_name = sys.argv[2]

    magnet2torrent(magnet, output_name)


if __name__ == "__main__":
    main()

上面的脚本大约需要1+分钟来获取元数据并创建.torrent文件,而utorrent客户机只需几秒钟,为什么呢?

我如何使我的脚本更快?

我想获取1k+洪流附近的元数据。

例如,磁链

代码语言:javascript
复制
magnet:?xt=urn:btih:BFEFB51F4670D682E98382ADF81014638A25105A&dn=openSUSE+13.2+DVD+x86_64.iso&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.ccc.de%3A80

最新情况:

我已经在我的脚本中指定了这样的已知dht路由器urls。

代码语言:javascript
复制
session = lt.session()
session.listen_on(6881, 6891)

session.add_dht_router("router.utorrent.com", 6881)
session.add_dht_router("router.bittorrent.com", 6881)
session.add_dht_router("dht.transmissionbt.com", 6881)
session.add_dht_router("router.bitcomet.com", 6881)
session.add_dht_router("dht.aelitis.com", 6881)
session.start_dht()

但是它还是很慢,有时我会犯这样的错误

代码语言:javascript
复制
DHT error [hostname lookup] (1) Host not found (authoritative)
could not map port using UPnP: no router found

更新:

我编写了这个scmall脚本,它从DB中获取十六进制信息哈希,并尝试从dht中获取元数据,然后在DB中插入流文件。

由于我不知道如何保存状态,所以我使它无限期地运行,因此保持它的运行将获得更多的对等点,并且获取元数据将更快。

代码语言:javascript
复制
#!/usr/bin/env python
# this file will run as client or daemon and fetch torrent meta data i.e. torrent files from magnet uri

import libtorrent as lt # libtorrent library
import tempfile # for settings parameters while fetching metadata as temp dir
import sys #getting arguiments from shell or exit script
from time import sleep #sleep
import shutil # removing directory tree from temp directory 
import os.path # for getting pwd and other things
from pprint import pprint # for debugging, showing object data
import MySQLdb # DB connectivity 
import os
from datetime import date, timedelta

#create lock file to make sure only single instance is running
lock_file_name = "/daemon.lock"

if(os.path.isfile(lock_file_name)):
    sys.exit('another instance running')
#else:
    #f = open(lock_file_name, "w")
    #f.close()

session = lt.session()
session.listen_on(6881, 6891)

session.add_dht_router("router.utorrent.com", 6881)
session.add_dht_router("router.bittorrent.com", 6881)
session.add_dht_router("dht.transmissionbt.com", 6881)
session.add_dht_router("router.bitcomet.com", 6881)
session.add_dht_router("dht.aelitis.com", 6881)
session.start_dht()

alive = True
while alive:

    db_conn = MySQLdb.connect(  host = 'localhost',     user = '',  passwd = '',    db = 'basesite',    unix_socket='') # Open database connection
    #print('reconnecting')
    #get all records where enabled = 0 and uploaded within yesterday 
    subset_count = 5 ;

    yesterday = date.today() - timedelta(1)
    yesterday = yesterday.strftime('%Y-%m-%d %H:%M:%S')
    #print(yesterday)

    total_count_query = ("SELECT COUNT(*) as total_count FROM content WHERE upload_date > '"+ yesterday +"' AND enabled = '0' ")
    #print(total_count_query)
    try:
        total_count_cursor = db_conn.cursor()# prepare a cursor object using cursor() method
        total_count_cursor.execute(total_count_query) # Execute the SQL command
        total_count_results = total_count_cursor.fetchone() # Fetch all the rows in a list of lists.
        total_count = total_count_results[0]
        print(total_count)
    except:
            print "Error: unable to select data"

    total_pages = total_count/subset_count
    #print(total_pages)

    current_page = 1
    while(current_page <= total_pages):
        from_count = (current_page * subset_count) - subset_count

        #print(current_page)
        #print(from_count)

        hashes = []

        get_mysql_data_query = ("SELECT hash FROM content WHERE upload_date > '" + yesterday +"' AND enabled = '0' ORDER BY record_num ASC LIMIT "+ str(from_count) +" , " + str(subset_count) +" ")
        #print(get_mysql_data_query)
        try:
            get_mysql_data_cursor = db_conn.cursor()# prepare a cursor object using cursor() method
            get_mysql_data_cursor.execute(get_mysql_data_query) # Execute the SQL command
            get_mysql_data_results = get_mysql_data_cursor.fetchall() # Fetch all the rows in a list of lists.
            for row in get_mysql_data_results:
                hashes.append(row[0].upper())
        except:
            print "Error: unable to select data"

        print(hashes)

        handles = []

        for hash in hashes:
            tempdir = tempfile.mkdtemp()
            add_magnet_uri_params = {
                'save_path': tempdir,
                'duplicate_is_error': True,
                'storage_mode': lt.storage_mode_t(2),
                'paused': False,
                'auto_managed': True,
                'duplicate_is_error': True
            }
            magnet_uri = "magnet:?xt=urn:btih:" + hash.upper() + "&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.ccc.de%3A80"
            #print(magnet_uri)
            handle = lt.add_magnet_uri(session, magnet_uri, add_magnet_uri_params)
            handles.append(handle) #push handle in handles list

        #print("handles length is :")
        #print(len(handles))

        while(len(handles) != 0):
            for h in handles:
                #print("inside handles for each loop")
                if h.has_metadata():
                    torinfo = h.get_torrent_info()
                    final_info_hash = str(torinfo.info_hash())
                    final_info_hash = final_info_hash.upper()
                    torfile = lt.create_torrent(torinfo)
                    torcontent = lt.bencode(torfile.generate())
                    tfile_size = len(torcontent)
                    try:
                        insert_cursor = db_conn.cursor()# prepare a cursor object using cursor() method
                        insert_cursor.execute("""INSERT INTO dht_tfiles (hash, tdata) VALUES (%s, %s)""",  [final_info_hash , torcontent] )
                        db_conn.commit()
                        #print "data inserted in DB"
                    except MySQLdb.Error, e:
                        try:
                            print "MySQL Error [%d]: %s" % (e.args[0], e.args[1])
                        except IndexError:
                            print "MySQL Error: %s" % str(e)    

                    shutil.rmtree(h.save_path())    #   remove temp data directory
                    session.remove_torrent(h) # remove torrnt handle from session   
                    handles.remove(h) #remove handle from list

                else:
                    if(h.status().active_time > 600):   # check if handle is more than 10 minutes old i.e. 600 seconds
                        #print('remove_torrent')
                        shutil.rmtree(h.save_path())    #   remove temp data directory
                        session.remove_torrent(h) # remove torrnt handle from session   
                        handles.remove(h) #remove handle from list
                sleep(1)        
                #print('sleep1')

        print('sleep10')
        sleep(10)
        current_page = current_page + 1
    #print('sleep20')
    sleep(20)

os.remove(lock_file_name);

现在,我需要实现Arvid建议的新事物。

更新

我已经成功地实现了阿维德的建议。以及我在洪水支持论坛http://forum.deluge-torrent.org/viewtopic.php?f=7&t=42299&start=10中找到的更多扩展

代码语言:javascript
复制
#!/usr/bin/env python

import libtorrent as lt # libtorrent library
import tempfile # for settings parameters while fetching metadata as temp dir
import sys #getting arguiments from shell or exit script
from time import sleep #sleep
import shutil # removing directory tree from temp directory 
import os.path # for getting pwd and other things
from pprint import pprint # for debugging, showing object data
import MySQLdb # DB connectivity 
import os
from datetime import date, timedelta

def var_dump(obj):
  for attr in dir(obj):
    print "obj.%s = %s" % (attr, getattr(obj, attr))

session = lt.session()
session.add_extension('ut_pex')
session.add_extension('ut_metadata')
session.add_extension('smart_ban')
session.add_extension('metadata_transfer')  

#session = lt.session(lt.fingerprint("DE", 0, 1, 0, 0), flags=1)

session_save_filename = "/tmp/new.client.save_state"

if(os.path.isfile(session_save_filename)):

    fileread = open(session_save_filename, 'rb')
    session.load_state(lt.bdecode(fileread.read()))
    fileread.close()
    print('session loaded from file')
else:
    print('new session started')

session.add_dht_router("router.utorrent.com", 6881)
session.add_dht_router("router.bittorrent.com", 6881)
session.add_dht_router("dht.transmissionbt.com", 6881)
session.add_dht_router("router.bitcomet.com", 6881)
session.add_dht_router("dht.aelitis.com", 6881)
session.start_dht()

alerts = [] 

alive = True
while alive:
    a = session.pop_alert()
    alerts.append(a)
    print('----------')
    for a in alerts:
        var_dump(a)
        alerts.remove(a)


    print('sleep10')
    sleep(10)
    filewrite = open(session_save_filename, "wb")
    filewrite.write(lt.bencode(session.save_state()))
    filewrite.close()

让它运行了一分钟,收到了警报

代码语言:javascript
复制
obj.msg = no router found 

更新:

经过一些测试后,看起来

代码语言:javascript
复制
session.add_dht_router("router.bitcomet.com", 6881)

导致

代码语言:javascript
复制
('%s: %s', 'alert', 'DHT error [hostname lookup] (1) Host not found (authoritative)')

更新:我添加了

代码语言:javascript
复制
session.start_dht()
session.start_lsd()
session.start_upnp()
session.start_natpmp()

然后警觉起来

代码语言:javascript
复制
('%s: %s', 'portmap_error_alert', 'could not map port using UPnP: no router found')
EN

回答 1

Stack Overflow用户

回答已采纳

发布于 2015-09-27 23:14:34

正如MatteoItalia所指出的,引导DHT并不是瞬时的,有时可能需要一段时间。当引导过程完成时,没有明确的时间点,这是一个越来越多地连接到网络的连续体。

连接得越多,知道的节点越好、越稳定,查找就越快。一种将大部分引导过程(获得更多苹果与苹果之间的比较)进行分析的方法是,在获得警报之后开始计时(并在此之前推迟添加磁铁链接)。

添加dht引导节点将主要使引导成为可能,它仍然不一定特别快。您通常希望大约270个节点(包括替换节点)被视为引导。

要加快引导过程,可以做的一件事是确保会话状态,其中包括dht路由表。这将将上一次会话中的所有节点重新加载到路由表中,并且(假设您没有更改IP,并且一切正常工作),引导应该会更快。

确保您没有在会话构造函数中启动DHT (作为标志参数,只需传入add_default_plugins)、加载状态、添加路由器节点,然后是启动dht

不幸的是,要让它在内部工作,需要大量的移动部件,排序是很重要的,并且可能存在一些微妙的问题。

另外,请注意,保持DHT持续运行将更快,因为重新加载状态仍将通过引导,它将只是前面有更多的节点到ping,并尝试“连接”到。

禁用start_default_features标志也意味着UPnP和NAT将不会启动,如果使用这些标志,您也必须手动开始

票数 3
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/32753998

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档