Common revolvers in URLresolver that are not working.

zinge

New member
May 6, 2014
13
0
0
180upload is freezing xbmc, but I haven't figured out why yet. Need to turn on debugging and look into it more.
Looks like 180upload is freezing trying to display the google recaptcha:

Code:
19:58:57 T:2904351808   DEBUG: urlresolver: resolving using 180upload plugin
19:58:57 T:2904351808  NOTICE: urlresolver: 180upload: in get_media_url 180upload.com x42q6mrhi2hh
19:58:57 T:2904351808   DEBUG: DialogProgress::StartModal called (already running)!
19:58:57 T:2904351808   DEBUG: ------ Window Init (DialogProgress.xml) ------
19:58:57 T:2904351808  NOTICE: urlresolver: 180Upload - Requesting GET URL: http://www.180upload.com/x42q6mrhi2hh
19:58:58 T:2904351808   DEBUG: urlresolver: Google ReCaptcha
19:58:58 T:2847929408  NOTICE: Thread JobWorker start, auto delete: true
19:58:58 T:2847929408    INFO: easy_aquire - Created session to http://www.google.com
19:58:58 T:3036815360   DEBUG: ------ Window Init () ------
19:58:59 T:3036815360   DEBUG: ------ Window Deinit (DialogProgress.xml) ------
19:58:59 T:2847929408   DEBUG: CCurlFile::GetMimeType - http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA -> image/jpeg
19:58:59 T:2847929408   DEBUG: CFileCache::Open - opening <recaptcha/api/image> using cache
19:58:59 T:2847929408   DEBUG: CurlFile::Open(0x459bb50) http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA
19:58:59 T:2895963200  NOTICE: Thread FileCache start, auto delete: false
19:58:59 T:2895963200    INFO: CFileCache::Process - Hit eof.
19:58:59 T:2895963200   DEBUG: Thread FileCache 2895963200 terminating
19:58:59 T:2847929408   DEBUG: CCurlFile::GetMimeType - http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA -> image/jpeg
19:58:59 T:2847929408   DEBUG: Caching image 'http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA' to 'a/aab164da.jpg':
19:58:59 T:2847929408   DEBUG: cached image 'special://masterprofile/Thumbnails/a/aab164da.jpg' size 300x57
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::Initialize OMX.broadcom.image_encode input port 340 output port 341 m_handle 0x4592a30
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::AllocInputBuffers component(OMX.broadcom.image_encode) - port(340), nBufferCountMin(1), nBufferCountActual(1), nBufferSize(77824), nBufferAlignmen(16)
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::AllocOutputBuffers component(OMX.broadcom.image_encode) - port(341), nBufferCountMin(1), nBufferCountActual(1), nBufferSize(81920) nBufferAlignmen(16)
19:58:59 T:2847929408   DEBUG: COMXImageEnc::CreateThumbnailFromSurface : special://masterprofile/Thumbnails/a/aab164da.jpg width 300 height 57
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.image_encode handle 0x4592a30
19:59:30 T:2847929408   DEBUG: Thread JobWorker 2847929408 terminating (autodelete)

Also, it looks in in several places where urllib2.URLError is being caught, it might need to be changed to urllib2.HTTPError so that you can use e.code.
 
Last edited:

zinge

New member
May 6, 2014
13
0
0
This may be a Raspbmc issue instead of a URLResolver issue. Tailing the log after the screen freezes is showing XBMC still receiving CEC commands from the TV, and htop doesn't show much CPU usage at all. I guess I'll try posting about this on the Raspbmc forums.


Looks like 180upload is freezing trying to display the google recaptcha:

Code:
19:58:57 T:2904351808   DEBUG: urlresolver: resolving using 180upload plugin
19:58:57 T:2904351808  NOTICE: urlresolver: 180upload: in get_media_url 180upload.com x42q6mrhi2hh
19:58:57 T:2904351808   DEBUG: DialogProgress::StartModal called (already running)!
19:58:57 T:2904351808   DEBUG: ------ Window Init (DialogProgress.xml) ------
19:58:57 T:2904351808  NOTICE: urlresolver: 180Upload - Requesting GET URL: http://www.180upload.com/x42q6mrhi2hh
19:58:58 T:2904351808   DEBUG: urlresolver: Google ReCaptcha
19:58:58 T:2847929408  NOTICE: Thread JobWorker start, auto delete: true
19:58:58 T:2847929408    INFO: easy_aquire - Created session to http://www.google.com
19:58:58 T:3036815360   DEBUG: ------ Window Init () ------
19:58:59 T:3036815360   DEBUG: ------ Window Deinit (DialogProgress.xml) ------
19:58:59 T:2847929408   DEBUG: CCurlFile::GetMimeType - http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA -> image/jpeg
19:58:59 T:2847929408   DEBUG: CFileCache::Open - opening <recaptcha/api/image> using cache
19:58:59 T:2847929408   DEBUG: CurlFile::Open(0x459bb50) http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA
19:58:59 T:2895963200  NOTICE: Thread FileCache start, auto delete: false
19:58:59 T:2895963200    INFO: CFileCache::Process - Hit eof.
19:58:59 T:2895963200   DEBUG: Thread FileCache 2895963200 terminating
19:58:59 T:2847929408   DEBUG: CCurlFile::GetMimeType - http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA -> image/jpeg
19:58:59 T:2847929408   DEBUG: Caching image 'http://www.google.com/recaptcha/api/image?c=03AHJ_VutvOzV0t5j0YdLZi4f8-iMLTOsdK6tnlMU5QvO6stae56SZUGKIqwX0ljl4M1zj7S4Qw3uTajyqPh7JCWfZceQJGU1pPqSTMDwu6LBP6kIXNQNteCBXREWnKbHWBGv1W6MOf9sZtS1LBmwpTEqqrLEUjCK30xtQKOf91Ww8xKWR4PfVdXvevIIkvVTcmvMNrT6u34YVSn6YO6OZ5XfUvTutcAFHGA' to 'a/aab164da.jpg':
19:58:59 T:2847929408   DEBUG: cached image 'special://masterprofile/Thumbnails/a/aab164da.jpg' size 300x57
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::Initialize OMX.broadcom.image_encode input port 340 output port 341 m_handle 0x4592a30
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::AllocInputBuffers component(OMX.broadcom.image_encode) - port(340), nBufferCountMin(1), nBufferCountActual(1), nBufferSize(77824), nBufferAlignmen(16)
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::AllocOutputBuffers component(OMX.broadcom.image_encode) - port(341), nBufferCountMin(1), nBufferCountActual(1), nBufferSize(81920) nBufferAlignmen(16)
19:58:59 T:2847929408   DEBUG: COMXImageEnc::CreateThumbnailFromSurface : special://masterprofile/Thumbnails/a/aab164da.jpg width 300 height 57
19:58:59 T:2847929408   DEBUG: COMXCoreComponent::Deinitialize : OMX.broadcom.image_encode handle 0x4592a30
19:59:30 T:2847929408   DEBUG: Thread JobWorker 2847929408 terminating (autodelete)
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Thanks for all the updates!

Issues:
vidx.py doesn't work correctly for vidxden, as vidx.to doesn't seem to work anymore. I removed it (forcing vidxden.py to be used instead) which works better.

vidxden.py works correctly if the useragent string:
Code:
return "%s|User-Agent=%s"%(stream_url,'Mozilla%2F5.0%20(Windows%20NT%206.1%3B%20rv%3A11.0)%20Gecko%2F20100101%20Firefox%2F11.0')163c163
is removed from the returned url
Code:
return "%s" % (stream_url)
movshare.py is getting a null return from the flv parsing regex. I commented it out and changed it to:
Code:
if not r:
                #html = unwise.unwise_process(html)
                #html = re.compile(r'eval\(function\(p,a,c,k,e,(?:d|r)\).+?\.split\(\'\|\'\).*?\)\)').search(html).group()
                #html = jsunpack.unpack(html)
                #filekey = unwise.resolve_var(html, "flashvars.filekey")
                filekey = re.search('flashvars.filekey="(.+?)";', html).group(0)
which now works correctly.

180upload is freezing xbmc, but I haven't figured out why yet. Need to turn on debugging and look into it more.




Sorry for posting code in forum replies... I just started looking at this today. Is it better for me to clone the repo and submit pull requests?
Hey yes, this would be very helpful!

I've noticed the 180 crashing too, it's when it pulls in the captcha image.. but I'm not sure why, and if I remember right they now use either google recaptcha or solve media, and the issue is on only one of them (I think google recatpcha)

Edit - just saw your further posts, this has happened on my windows and atv2 boxes, so is not environment specific
 

zinge

New member
May 6, 2014
13
0
0
Hey yes, this would be very helpful!

I've noticed the 180 crashing too, it's when it pulls in the captcha image.. but I'm not sure why, and if I remember right they now use either google recaptcha or solve media, and the issue is on only one of them (I think google recatpcha)

Edit - just saw your further posts, this has happened on my windows and atv2 boxes, so is not environment specific
Pull request submitted: https://github.com/Eldorados/script.module.urlresolver/pull/172

Yeah, it looks like 180 is specifically failing on recaptcha. Do you happen to know another resolver that is working with recaptcha right now? I can try copying over the working code to 180.

Speaking of which, it looks like a decent number of these resolvers are using recaptcha or solve media. I was thinking of adding a common utils thing for human verification (I guess under plugins/lib?) that they could all pull from so you only need one instance of the solvemedia and recaptcha code. Thoughts?
 

zinge

New member
May 6, 2014
13
0
0
Since I'm using a Raspberry Pi as my XBMC host, my biggest bottleneck for this plugin is CPU time. I've been looking through the code trying to figure out how to speed up the listing of available streams once an episode is selected. It looks like the most expensive part CPU-wise is finding which resolver to use for each source (_find_resolvers() in types.py). I was looking at a couple different ways of speeding this up. I figured I'd mention them here and see if this was something you were interested in. If it is, I can work on modifying the way URLResolvers work to speed up the whole process. If not, I'll just hack at it on my Pi so it's faster for me :).

Option 1: Return the first working resolver
Right now, _find_resolvers check .valid_url on every installed resolver, and then returns a list of valid ones. However, the rest of the code just takes the first resolver in this list and uses it. The easiest/hackiest way to speed things up would be to immediately return once a valid resolver is found, instead of looking for all valid ones. This isn't the greatest idea because you may want to change the code in the future to cycle through all valid resolvers if one fails, instead of just using the first one in the list.

Option 2: Flip the loop
Instead of looping over sources and then checking them against each resolver, loop over resolvers and check them against all the sources. The benefit here is that you could compile the regex matches in each valid_url call, which might help a bit. I know python will cache the last couple regex matchers, but since there are a whole bunch of resolvers, I think it's being forced to recompile the regex matcher for each source. Not sure how much of a difference compiled regex speed makes, but it might help.

Option 3: Change how Resolvers are loaded
This would be a much larger code change. Basically, the section where you register the available resolver plugins would pull an attribute from each resolver that contains a regex pattern to match for that resolver. For _find_resolvers, instead of having to init and call a regex function from each resolver, you would have a dictionary or list of tuples of precompiled regex patterns and the name of the resolver they are for. I think this way is the most work, but might substantially speed of the _find_resolvers call... or I'm entirely wrong and it won't do much at all. :)

If any of these sound like they'd be worth doing, let me know and I'll take a stab at adding them to my fork. If not, I'll just try a couple things to speed up my Pi.

Thanks!
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Pull request submitted: https://github.com/Eldorados/script.module.urlresolver/pull/172

Yeah, it looks like 180 is specifically failing on recaptcha. Do you happen to know another resolver that is working with recaptcha right now? I can try copying over the working code to 180.

Speaking of which, it looks like a decent number of these resolvers are using recaptcha or solve media. I was thinking of adding a common utils thing for human verification (I guess under plugins/lib?) that they could all pull from so you only need one instance of the solvemedia and recaptcha code. Thoughts?
Yep that was something I was thinking about as well, we have placed a number of commonly used items into the lib folder, this would be a good idea too

I wrote up the structure for it in recent Ic*f*lms releases, check out my resolvers.py file how I separated it and added checks for invalid captchas entered, just to give you some ideas

Hugefiles is one I know of offhand that uses recaptcha, but works fine.. it actually uses 3 possible formats.. I had done the update to 180 to add recaptcha using proven code, so I'm not sure what's wrong.. it's the only resolver that has trouble, even in Ic*f*lms
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Since I'm using a Raspberry Pi as my XBMC host, my biggest bottleneck for this plugin is CPU time. I've been looking through the code trying to figure out how to speed up the listing of available streams once an episode is selected. It looks like the most expensive part CPU-wise is finding which resolver to use for each source (_find_resolvers() in types.py). I was looking at a couple different ways of speeding this up. I figured I'd mention them here and see if this was something you were interested in. If it is, I can work on modifying the way URLResolvers work to speed up the whole process. If not, I'll just hack at it on my Pi so it's faster for me :).

Option 1: Return the first working resolver
Right now, _find_resolvers check .valid_url on every installed resolver, and then returns a list of valid ones. However, the rest of the code just takes the first resolver in this list and uses it. The easiest/hackiest way to speed things up would be to immediately return once a valid resolver is found, instead of looking for all valid ones. This isn't the greatest idea because you may want to change the code in the future to cycle through all valid resolvers if one fails, instead of just using the first one in the list.

Option 2: Flip the loop
Instead of looping over sources and then checking them against each resolver, loop over resolvers and check them against all the sources. The benefit here is that you could compile the regex matches in each valid_url call, which might help a bit. I know python will cache the last couple regex matchers, but since there are a whole bunch of resolvers, I think it's being forced to recompile the regex matcher for each source. Not sure how much of a difference compiled regex speed makes, but it might help.

Option 3: Change how Resolvers are loaded
This would be a much larger code change. Basically, the section where you register the available resolver plugins would pull an attribute from each resolver that contains a regex pattern to match for that resolver. For _find_resolvers, instead of having to init and call a regex function from each resolver, you would have a dictionary or list of tuples of precompiled regex patterns and the name of the resolver they are for. I think this way is the most work, but might substantially speed of the _find_resolvers call... or I'm entirely wrong and it won't do much at all. :)

If any of these sound like they'd be worth doing, let me know and I'll take a stab at adding them to my fork. If not, I'll just try a couple things to speed up my Pi.

Thanks!
I think this is worth a good discussion, especially with XBMC being ported to so many small devices and how hugely popular RPi is.. I'm always in to optimizing our code!

Can you fire up a new dedicated thread where we can chat about it?
 

zinge

New member
May 6, 2014
13
0
0
Yep that was something I was thinking about as well, we have placed a number of commonly used items into the lib folder, this would be a good idea too

I wrote up the structure for it in recent Ic*f*lms releases, check out my resolvers.py file how I separated it and added checks for invalid captchas entered, just to give you some ideas

Hugefiles is one I know of offhand that uses recaptcha, but works fine.. it actually uses 3 possible formats.. I had done the update to 180 to add recaptcha using proven code, so I'm not sure what's wrong.. it's the only resolver that has trouble, even in Ic*f*lms
There must be something else on the 180upload page that's causing problems when it's parsed somehow. I'll take a look later today if I get a chance.
 

zinge

New member
May 6, 2014
13
0
0
Just to keep track, it looks like 119 in 180upload.py is where the freeze is happening:
Code:
117                 captchaimg = 'http://www.google.com/recaptcha/api/image?c='+part.group(1)
118                 img = xbmcgui.ControlImage(450,15,400,130,captchaimg)
119                 wdlg = xbmcgui.WindowDialog()
I don't really know anything about how the xbmc calls work yet though, so I have no idea why. I'll look at those a little later, unless someone has an idea.
 

zinge

New member
May 6, 2014
13
0
0
Just to keep track, it looks like 119 in 180upload.py is where the freeze is happening:
Code:
117                 captchaimg = 'http://www.google.com/recaptcha/api/image?c='+part.group(1)
118                 img = xbmcgui.ControlImage(450,15,400,130,captchaimg)
119                 wdlg = xbmcgui.WindowDialog()
I don't really know anything about how the xbmc calls work yet though, so I have no idea why. I'll look at those a little later, unless someone has an idea.
I'm going to give up on this for a bit. I just hit one with a solvemedia captcha that also froze xbmc, so it may not be that line number and may not be related to recaptcha at all.
 

zinge

New member
May 6, 2014
13
0
0
Every time. I select 180upload as the stream, it pops up a dialog that says Resolving..., then the dialog goes away and xbmc is frozen. I can still ssh in to the Pi. Not out of memory or CPU, and I can 'sudo initctl stop xbmc' and restart it the same way. I don't have to reboot the Pi. Turning on debug mode and tailing the log after it freezes is still showing that it's recognizing CEC input from my TV remote, but nothing happens on the screen until I stop xbmc.
 

mindgame

New member
Apr 13, 2013
119
0
0
It is working fine for me on raspbmc using gotv, gomovies and Ic*f*lms addon. What addon are you using?
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
I'm wondering if we should save the image locally first using urllib.. the grab of the image is what I assumed was causing it

Though I believe with solve media we already save the image locally, basically because we have to with the way they work... surprised that one crashes on you
 

firthy

New member
Jul 22, 2014
1
0
0
I'm having trouble with Mightyupload since the 1Ch**nel update, all I get is "Unable to resolve Mightyupload link. Player config not found"

I'm trying to figure out if this is just my set-up or if Mightyupload have changed their site and the URLresolver is out of date for other users as well.

Any info would be gratefully received. Thanks.
 

s7eele

Member
Sep 29, 2012
427
0
16
US of A (Deep Deep South)
Came looking on the forums because I am having far too many failures on links I know should be working but even before these recent issues I was experiencing the same problem described above with 180upload but running on OUYA. I'm still having problems with resolver but need to look around some more before posting on it. Gonna try the DNS change suggested before but I don't think that's it, think I am probably the cause. Just wanted to share that I have the same 180upload behavior running on an OUYA.
 

tknorris

New member
Feb 18, 2014
201
0
0
I'm wondering if we should save the image locally first using urllib.. the grab of the image is what I assumed was causing it

Though I believe with solve media we already save the image locally, basically because we have to with the way they work... surprised that one crashes on you
FYI, I'm getting lock ups on Linux using 180upload, vidplay or hugefiles urlresolvers. I haven't looked into detail yet where the resolver is locking but it's a hard lock that requires XBMC be restarted to get the system back. The one thing I did notice is all three sites have a common captcha function. I have no idea if that's causing it or not yet though.
 

tknorris

New member
Feb 18, 2014
201
0
0
OK, so there are problem(s) with 180upload, vidplay and hugefiles are many fold.
1) It appears that the double dialog in the resolver plugins causes XBMC crashes. This can be resolved by just removing them.
2) The solvemedia captcha api has changed and requires additional input data to post correctly.

I'll fix both of these today sometime and submit a fix once they all 3 work.