Closed Bug 999445 Opened 11 years ago Closed 11 years ago

Decrease in number of users getting OpenGL layers on OS X (Intel Graphics, OSX 10.6 and 10.7, Firefox 27.0+)

Categories

(Core :: Graphics, defect)

Product:
Core
Component:
Graphics
Platform:
x86
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla32
Tracking Flags:
Tracking Status
firefox30 --- verified
firefox31 --- verified
firefox32 --- verified
b2g-v1.4 --- fixed

People

(Reporter: jrmuizel, Assigned: bjacob)

References

(Depends on 1 open bug)

Details

Attachments

(2 files)

Table showing corresponding decrease of WebGL success rates
1.38 KB, text/html
Details
dont-require-gl-version
1.25 KB, patch
jrmuizel
: review+
Sylvestre
: approval-mozilla-aurora+
Sylvestre
: approval-mozilla-beta+
Details | Diff | Splinter Review
http://people.mozilla.org/~bjacob/gfx_features_stats/#mac shows a large decrease in the number of users getting layers on OS X. This seems bad.
Wasn't the 100% rate that we used to measure unplausible in the first place? I thought of that as a bug in how we record layers initialization success on Mac (basically, if we blacklist based on OSX version before we even try, so we only record successes) and was sort of pleased to see it drop to now realistic, and still quite high, levels.
We only support 10.6 and later. We blacklist layers only on some 10.6 that are behind on their system updates which should be very minimal. I think the current drop is really unrealistic.
All Intel Graphics users, concentrated on only two device IDs: bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | wc -l 584 bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | wc -l 582 bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(AdapterDeviceID: 0x[a-fA-F0-9]*\).*/\1/g' | sort | uniq -c 455 AdapterDeviceID: 0x27a2 127 AdapterDeviceID: 0x2a02
Looked up these two device IDs in GfxDriverInfo.cpp; 0x27a2 is the Intel 945 (from the GMA 950 family) from a very, very long time ago (no hardware vertex shaders...) 0x2a02 is the Intel GL960 (from the GMA X3000 family) from circa 2006 or 2007, an actual hardware implementation of all of OpenGL 2 AFAIK, so that's more concerning IMHO. The next thing is OSX versions. It's 10.6 and 10.7 exclusively: bjacob:~/crash-stats$ zcat 20140416-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Mac OS X.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 446 Mac OS X 10.6.8 130 Mac OS X 10.7.5 1 Mac OS X 10.7.4 1 Mac OS X 10.7.0 1 Mac OS X 10.6.7 1 Mac OS X 10.6.6 1 Mac OS X 10.6.4 1 Mac OS X 10.6.3
In conclusion, this is entirely made of Intel Graphics on OSX 10.6 and 10.7. This smells like something about being low-spec in some kind of way...
Regarding Firefox versions, the thing of note is that (within noise) this only started appearing significantly during February with Firefox 27, and become much more prevalent during the first half of April on Firefox 28. bjacob:~/crash-stats$ zcat 20140201-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 5 Firefox 27.0 3 Firefox 28.0 bjacob:~/crash-stats$ zcat 20140215-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 24 Firefox 28.0 22 Firefox 27.0.1 14 Firefox 27.0 2 Firefox 29.0 1 Firefox 26.0 bjacob:~/crash-stats$ zcat 20140301-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 66 Firefox 27.0.1 10 Firefox 28.0 2 Firefox 27.0 bjacob:~/crash-stats$ zcat 20140315-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 69 Firefox 27.0.1 12 Firefox 28.0 2 Firefox 29.0 1 Firefox 27.0 bjacob:~/crash-stats$ zcat 20140401-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 41 Firefox 28.0 18 Firefox 27.0.1 10 Firefox 29.0 2 Firefox 1 Firefox 27.0 bjacob:~/crash-stats$ zcat 20140405-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 65 Firefox 28.0 10 Firefox 29.0 5 Firefox 27.0.1 2 Firefox 30.0 1 Firefox 27.0 1 Firefox 26.0 bjacob:~/crash-stats$ zcat 20140410-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 505 Firefox 28.0 101 Firefox 29.0 27 Firefox 27.0.1 12 Firefox 27.0 9 Firefox 30.0 1 Firefox 31.0 1 Firefox 25.0.1 1 Firefox 24.0 1 Firefox 21.0 1 Firefox bjacob:~/crash-stats$ zcat 20140415-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep 'AdapterVendorID: 0x8086' | sed 's/.*\(Firefox.[0-9.]*\).*/\1/g' | sort | uniq -c | sort -rn 718 Firefox 28.0 113 Firefox 29.0 21 Firefox 27.0.1 12 Firefox 30.0 5 Firefox 27.0 2 Firefox 1 Firefox 31.0 1 Firefox 12 1 Firefox!
Summary: Decrease in number of users getting layers on OS X → Decrease in number of users getting OpenGL layers on OS X (Intel Graphics, OSX 10.6 and 10.7, Firefox 27.0+)
Hey, I updated again my stats, and we're most of the way back up now! http://people.mozilla.org/~bjacob/gfx_features_stats/#mac Still not quite where we used to be. Let's keep this bug open for at least a few weeks and monitor this graph (which I have to update manually, so ping me).
And on http://people.mozilla.org/~bjacob/gfx_features_stats/#os-market-share this seems correlated with a OSX crashiness spike...
If someone feels like taking a look, here are 20 random crash links from affected users on April 11: bjacob:~/crash-stats$ zcat 20140411-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | cut -d ' ' -f 3 | head -n20 http://crash-stats.mozilla.com/report/index/0526cae5-7ede-4236-8813-6f6262140411 http://crash-stats.mozilla.com/report/index/5134b935-43d5-4371-a22f-d30da2140411 http://crash-stats.mozilla.com/report/index/f8c1cef7-f512-4e45-9deb-b12d42140411 http://crash-stats.mozilla.com/report/index/1a624d36-02f4-4069-aa9d-955662140411 http://crash-stats.mozilla.com/report/index/bfda4008-abe1-48c9-a1c7-3bb2d2140411 http://crash-stats.mozilla.com/report/index/bbe944d8-6490-473e-848a-97ecf2140411 http://crash-stats.mozilla.com/report/index/8dc7f56a-78e8-4393-b92b-b56892140411 http://crash-stats.mozilla.com/report/index/c6282504-22d2-4021-ba96-77f1e2140411 http://crash-stats.mozilla.com/report/index/d97d12b0-8f91-4f1d-bf6d-ad20a2140411 http://crash-stats.mozilla.com/report/index/b571e169-958d-40c5-a785-ae6162140411 http://crash-stats.mozilla.com/report/index/0eb52689-3e19-4617-a2e1-ed1ee2140411 http://crash-stats.mozilla.com/report/index/837263b7-b617-46dc-b1e7-0b11a2140411 http://crash-stats.mozilla.com/report/index/9e88677f-98c3-446d-a5ee-7e2ab2140411 http://crash-stats.mozilla.com/report/index/2d8b866e-ceac-4d8a-87f7-26b092140411 http://crash-stats.mozilla.com/report/index/80462ee7-8b39-4e57-8e41-5859c2140411 http://crash-stats.mozilla.com/report/index/681209be-2195-4b8d-ba0d-13d4c2140411 http://crash-stats.mozilla.com/report/index/5fb20eca-4220-4af6-8752-91d302140411 http://crash-stats.mozilla.com/report/index/be1a06da-f91b-4799-bc7a-3baab2140411 http://crash-stats.mozilla.com/report/index/8ce06851-3126-4fd4-9035-77b212140411 http://crash-stats.mozilla.com/report/index/46324d0b-7bf0-4dce-9d8f-08a902140411
Over 80% of these are flash plugin crashes; here's another run filtering only non-plugin crashes: bjacob:~/crash-stats$ zcat 20140411-pub-crashdata.csv.gz | grep 'Mac OS X' | grep 'Layers-' | grep -v plugin | cut -d ' ' -f 3 | head -n20 http://crash-stats.mozilla.com/report/index/5134b935-43d5-4371-a22f-d30da2140411 http://crash-stats.mozilla.com/report/index/8ce06851-3126-4fd4-9035-77b212140411 http://crash-stats.mozilla.com/report/index/46324d0b-7bf0-4dce-9d8f-08a902140411 http://crash-stats.mozilla.com/report/index/89075571-ad97-42f3-8e5b-fc0dd2140411 http://crash-stats.mozilla.com/report/index/c0830413-10d4-4a77-8c35-4518d2140411 http://crash-stats.mozilla.com/report/index/894aeb50-051d-47df-a2ae-de3772140411 http://crash-stats.mozilla.com/report/index/e10e008c-8873-4dd9-bda4-956062140411 http://crash-stats.mozilla.com/report/index/59ec294b-dd4f-47c8-921e-0345a2140411 http://crash-stats.mozilla.com/report/index/327e388f-14e7-4401-9ee8-3f06a2140411 http://crash-stats.mozilla.com/report/index/92c2923a-67ad-49c5-9c44-5762c2140411 http://crash-stats.mozilla.com/report/index/94352fdd-e720-4500-8276-fedd52140411 http://crash-stats.mozilla.com/report/index/637daac9-f88c-487e-a9ff-c99ed2140411 http://crash-stats.mozilla.com/report/index/9268adfb-3274-419e-a9e4-ab3ae2140411 http://crash-stats.mozilla.com/report/index/b610959e-79bd-495b-bcbd-ddc302140411 http://crash-stats.mozilla.com/report/index/05408d72-3bd4-40ad-9167-63fc42140411 http://crash-stats.mozilla.com/report/index/cbf4faa9-2732-4a19-a211-809482140411 http://crash-stats.mozilla.com/report/index/7f93fe00-8045-44b8-b26b-224072140411 http://crash-stats.mozilla.com/report/index/4a3504af-2fb5-4957-910c-1b4e22140411 http://crash-stats.mozilla.com/report/index/ec7b67aa-a825-450e-9547-c8d612140411 http://crash-stats.mozilla.com/report/index/0b20156f-7b7f-48ef-bd21-25d702140411
Depends on: 1002846
Updated http://people.mozilla.org/~bjacob/gfx_features_stats/ . Mac GL layers rates are still not back up to 100%. They're sort of plateauing around 95%.
Blocks: 995145
It's good that we're tracking this and that we may get some telemetry. There may be a number of different explanations for this, given that we're picking up the data from crash reports and that the crash reports spiked at the same time.
The timing and versions may not quite match, but can't help but wonder if bug 1004134 (thanks for a CC :gal), is related to this (10.9, intel graphics)
Because people have also noticed (and seem to be anxiously tracking) the corresponding decrease of the WebGL success rate on the public_webgl list, I looked into that. It's also concentrated on OSX 10.6 and 10.7 only, and is already back to normal on 10.7, so only 10.6 remains affected.
Note: generated by for DATE in 20140101 20140115 20140201 20140215 20140301 20140315 20140401 20140415 20140420 20140425 20140428 20140501 20140503; do echo "<tr>"; echo "<td>$DATE</td>"; for VERSION in 10.6 10.7 10.8 10.9; do echo "<td> $(echo "100 * `zcat $DATE-pub-crashdata.csv.gz | grep Mac.OS.X.$VERSION | grep 'WebGL+' | wc -l` / `zcat $DATE-pub-crashdata.csv.gz | grep Mac.OS.X.$VERSION | grep 'WebGL[?!]' | wc -l`" | bc) % </td>"; done; echo "</tr>"; done
This seems to have been caused by bug 923194. Benoit will create a fix.
Many thanks Jeff :) (The conciseness of comment 16 might not make this clear, but Jeff just did a careful bisection on a Mac Mini that reproduces.) To be clear, what happened is that in addition to doing what it says it does, bug 923194 also made us actually refuse to initialize GL layers on too-low GL versions. The affected Macs have GL 1.4 so they are rejected. They should not be rejected, and they used to work fine with GL layers, because they support all the GL extensions that we need. Basically, we need to do a proper "GL feature" check like we already do for various things. Patch coming.
Attachment #8417519 - Flags: review?(jmuizelaar)
Blocks: 923194
Filed bug 1006050 to do the right thing i.e. properly verify that the GL supports the features that we need, on initialization. (I'm not planning to work on it at this point). For now, the patch here is minimal to make it easy to uplift to beta.
Depends on: 1006050
Attachment #8417519 - Flags: review?(jmuizelaar) → review+
Comment on attachment 8417519 [details] [diff] [review] dont-require-gl-version [Approval Request Comment] Bug caused by (feature/regressing bug #): bug 923194 / Firefox 27 User impact if declined: About 20% of Mac OS 10.6 users (5% of total Mac users) get no hardware acceleration (so they have a slower and far less tested code path) and also no WebGL. Testing completed (on m-c, etc.): m-i Risk to taking this patch (and alternatives if risky): Should not be risky, since on release builds it's just reverting bug 923194. Basically, I think it's a no-brainer for aurora; regarding beta, well this is clearly a nontrivial switch for beta, but my perspective here is that it is even more scary to me (as a gfx dev) to have 5% of release channel users on a code path (non-GL layers) that we didn't even know was still functional on Mac. So I'd like this to be fixed asap, which is why I would still like this to be taken on beta. Hopefully it's early enough in the cycle for that. String or IDL/UUID changes made by this patch: none
Attachment #8417519 - Flags: approval-mozilla-beta?
Attachment #8417519 - Flags: approval-mozilla-aurora?
https://hg.mozilla.org/mozilla-central/rev/ee3f79376eef
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla32
Comment on attachment 8417519 [details] [diff] [review] dont-require-gl-version Early in the beta process, we can accept it.
Attachment #8417519 - Flags: approval-mozilla-beta?
Attachment #8417519 - Flags: approval-mozilla-beta+
Attachment #8417519 - Flags: approval-mozilla-aurora?
Attachment #8417519 - Flags: approval-mozilla-aurora+
(In reply to Ryan VanderMeulen [:RyanVM UTC-4] from comment #25) > https://hg.mozilla.org/releases/mozilla-b2g30_v1_4/rev/f15b4b99adc4 Hah, OK. I should have unflagged: there is no way that OpenGL ES 2.0 platforms (i.e. all of mobile) could be affected. Contrary to desktop GL, in the ES world, 2.0 is not just 1.x plus extensions.
Looks like the downward trend has been reversed. Marking this bug verified fixed.
Status: RESOLVED → VERIFIED
status-firefox30: fixed → verified
status-firefox31: fixed → verified
status-firefox32: fixed → verified
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: