This patch saves 1600 context lookups on a 1600 course category. rcache
does help a bit with small categories, but on large setups, this is
not sustainable.
And it's not needed either. We have the data right at our fingertips.
Just get it when it's cheap...
A walkthrough of course-login-as functionality shows that is
Just Works, except that get_my_courses() was showing all the
courses. So we fix it.
And cleanup load_all_capabilities() - things just work
transparently.
- Field handling moves back to get_my_courses() and now we have
almost all the fields that the old get_my_courses() did
(except for summary, which is *huge*) so get_my_courses() asks
for a lot of fields, but the get_user_courses_bycap() defaults
are _much_ leaner now.
I think this makes sense ;-)
- get_my_courses() now caches the course ids for the currently logged in
user in $USER->mycourses -- as a _string_. This is magnitudes more efficient
than having it as an array.
The cache makes a difference, but it's not very visible on
normal pageloads (with my courses block, for example).
However, over 100 iterations, for a user with 50 enrolments in a site
with 6K courses, we go from 4.3s to 0.6s. And the DB queries are *cheap*.
$tt = microtime(true);
for($n=0;$n<100;$n++) {
get_my_courses($USER->id, 'sortorder ASC');
}
error_log("took " . (microtime(true) - $tt));
get_user_courses_bycap() replaces get_courses_bycap_fromsess().
Using a combination of in-DB enrolments and in-session capability
checks, we narrow down the courses we fetch from the DB for checking.
This patch adds a small DB query, and has has a moderate impact on
the timings observable on my laptop (~300ms?), but reduces
*significantly* the bandwidth used, which in cluster environments
with frontends separate from backends has a serious impact.
get_my_courses() goes from a bazillion queries (500 in some sample
cases) to 1 for the logged-in user, and 4 for a non-logged-in user.
One of those queries brings a *lot* of data across (all rows from
mdl_course) so there is room for serious optimisation.
However, this clocks at ~300 ms on my laptop, costly, but not
the end of the world. If your PHP-DB link has bandwidth probs
it might be a problem.
A few important changes to get_my_courses()
- (Compat ALERT!) the default fields are less than before --
(will be followed by patches that fix the callers!) our defaults
had grown to quite a bit because of the crazy caching scheme it had
- the $fields parameter is to name _additional_ fields you need, and
ideally wants them passed as an array. Will cope with old-style
strings too.
- the returned courses have an extra property "context" that is
a fully valid context object, so if the caller needs to perform
further accesslib checks, we save a query per course down the road
The work is done in the newfangled get_courses_bycap_fromsess()
which is brute-force but fast. I'm sure we can optimise the common
cases a lot in it if we try. It'd be worthwhile to
- reduce the rows we grab - that's really boneheaded
- if we copy and tweak the logic of has_cap_fromsess() in it
it can be made even faster
* All updates to user.lastaccess and user_lastaccess.timeaccess are paced to
60s of the last update on the same record -- this should reduce the heat
on those tables.
* Updates/inserts to user_lastaccess are down with raw SQL to avoid costly
schema lookups on every request.
Move the fixups for orphan courses to the newly minted
fix_coursecategory_orphans() -- and optimise it to take only 1 dbquery for
the common case.
If we do find lots of orphans, we issue 2 updates per orphan.
This cuts down dbqueries drastically - we used to have 2x the number of
courses in the site.
Two improvements for fix_course_sortorder()
- If the category has more courses than the shift range
use a larger shift range to avoid overlapping with itself
- If things do go wrong during the per-course sortorder updates,
rollback and try and call ourselves with a 'safe' flag.
Still - far from perfect. Probably the global sortorder approach
is broken. The sanest way is to rework things to always join against
course_categories and order by the combined sortorders.
I've tested both the "IN" subquery and the "FROM" subquery and, under sites
with more courses, the "FROM" one seems to perform slightly better. MDL-9381
Merged from MOODLE_18_STABLE
This makes get_my_courses() faster/lighter in 2 ways
- We fetch a ton less data from the DB
- We store a tone less data in the Session
In a test environment, with a student enrolled in 3 _empty_ courses
(just created, empty modinfo, etc), this shaves 50% off the session
size on-disk.
The problem is callers that may be expecting a given field to appear by
default. Next step: walk all callers...
