Optimizations for slower-CPU devices

zinge

New member
May 6, 2014
13
0
0
New thread for discussing how to optimize parts of URLResolver for speed improvements on slower-CPU devices like the RPi.

Since I'm using a Raspberry Pi as my XBMC host, my biggest bottleneck for this plugin is CPU time. I've been looking through the code trying to figure out how to speed up the listing of available streams once an episode is selected. It looks like the most expensive part CPU-wise is finding which resolver to use for each source (_find_resolvers() in types.py). I was looking at a couple different ways of speeding this up. I figured I'd mention them here and see if this was something you were interested in. If it is, I can work on modifying the way URLResolvers work to speed up the whole process. If not, I'll just hack at it on my Pi so it's faster for me :).

Option 1: Return the first working resolver
Right now, _find_resolvers check .valid_url on every installed resolver, and then returns a list of valid ones. However, the rest of the code just takes the first resolver in this list and uses it. The easiest/hackiest way to speed things up would be to immediately return once a valid resolver is found, instead of looking for all valid ones. This isn't the greatest idea because you may want to change the code in the future to cycle through all valid resolvers if one fails, instead of just using the first one in the list.

Option 2: Flip the loop
Instead of looping over sources and then checking them against each resolver, loop over resolvers and check them against all the sources. The benefit here is that you could compile the regex matches in each valid_url call, which might help a bit. I know python will cache the last couple regex matchers, but since there are a whole bunch of resolvers, I think it's being forced to recompile the regex matcher for each source. Not sure how much of a difference compiled regex speed makes, but it might help.

Option 3: Change how Resolvers are loaded
This would be a much larger code change. Basically, the section where you register the available resolver plugins would pull an attribute from each resolver that contains a regex pattern to match for that resolver. For _find_resolvers, instead of having to init and call a regex function from each resolver, you would have a dictionary or list of tuples of precompiled regex patterns and the name of the resolver they are for. I think this way is the most work, but might substantially speed of the _find_resolvers call... or I'm entirely wrong and it won't do much at all. :)

If any of these sound like they'd be worth doing, let me know and I'll take a stab at adding them to my fork. If not, I'll just try a couple things to speed up my Pi.
Thanks!
I think this is worth a good discussion, especially with XBMC being ported to so many small devices and how hugely popular RPi is.. I'm always in to optimizing our code!

Can you fire up a new dedicated thread where we can chat about it?
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Lets get some peeps involved here

For starters and lower impact I like option #1, sounds like a logical thing that should have been considered initially

And to take on something to the side of this topic, some recent resolver updates have caused slowdowns as well, some users have pointed out that in the latest 2.2.0 release there was a noticable drop, would be good to identify the culprit and make sure any resolvers that have logins are not trying to communicate to a site when doing the initial valid host check
 

zinge

New member
May 6, 2014
13
0
0
Lets get some peeps involved here

For starters and lower impact I like option #1, sounds like a logical thing that should have been considered initially

And to take on something to the side of this topic, some recent resolver updates have caused slowdowns as well, some users have pointed out that in the latest 2.2.0 release there was a noticable drop, would be good to identify the culprit and make sure any resolvers that have logins are not trying to communicate to a site when doing the initial valid host check
I added a couple log_debug statements to find_resolvers to print out the name of each as it is checking valid_url and didn't see any standing out as taking particularly long. Just a lot of them to go through, over and over. Were there a lot more resolvers added to the list in 2.2.0?
 

zinge

New member
May 6, 2014
13
0
0
I added a couple log_debug statements to find_resolvers to print out the name of each as it is checking valid_url and didn't see any standing out as taking particularly long. Just a lot of them to go through, over and over. Were there a lot more resolvers added to the list in 2.2.0?
It looks like alldebrid, realdebrid, and rpnet are the only ones doing anything other than regex or string matching in valid_url. They are getting a pattern to match from an http call. However, temporarily removing those 3 doesn't seem to result in much of a speedup, so I don't think that's the culprit.
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Yep a dozen or more were addon in recent release, I noted as many as I could remember in the changelog

Those pay ones also shouldn't stop immediately if they don't have login credentials entered
 

al7oumame

New member
Mar 13, 2013
6
0
0
hi is it possible to mke some resolvers work like in this addon I used it for a while and noticed no buffering nd more stability or maybe we can use it when needed especially for uptobox links and others like the youtube addon for youtube links.what do u think?
cheers
 

nothere

Banned
Nov 25, 2012
1,036
0
0
hi is it possible to mke some resolvers work like in this addon I used it for a while and noticed no buffering nd more stability or maybe we can use it when needed especially for uptobox links and others like the youtube addon for youtube links.what do u think?
cheers
Interesting addon. Hmmmm. I wonder what martijns take is on that?
 

Eldorado

Moderator
Staff member
May 7, 2012
990
0
16
Lets not derail this thread

Side note to that addon - it uses accounts to each site, most sites give a small speed boost to users with accounts vs anonymous

Stability is because you are accessing your own files, not files uploaded by someone else and being downloaded by thousands of anonymous

In the end this addon doesn't do anything different that we are, which is finding the final download link and passing it to xbmc.. there's no magic behind the scenes

Move further discussion to a new thread, this one is concerning optimizing the code of urlresolver for small devices
 
Last edited:

al7oumame

New member
Mar 13, 2013
6
0
0
I am sorry eldorado I didn't khnow that , i m not a dev :( .
I just thought it is like youtube addon is working good on small devices maybe this will do the same .
I think it is possible to play public files (not my own ones ) using this format by invoking the plugin with: plugin://plugin.video.cloudstream/?mode=streamURL&url=http://domain.com/#####)"
anyway i just wanted to give some ideas and help, I like your work and like it to be the best :) I already created a special thread (was hoping to add more addons to the resolver like youtube so it can play more sources (like mega ,uptobox and bestreams.net ) )

thanks :)
 
Thread starter Similar threads Forum Replies Date
L Indigo Tool 2
B Apple TV 2 3